Process management with supervisor in Raspberry Pi - Deep Dive

Overview - Process management with supervisor

What is it?

Process management with Supervisor means using a tool called Supervisor to start, stop, and monitor programs automatically on your Raspberry Pi. It helps keep your programs running all the time, even if they crash or the Raspberry Pi restarts. Supervisor watches your programs and can restart them if they stop working. This makes managing background tasks easier and more reliable.

Why it matters

Without Supervisor, you would have to manually start your programs every time your Raspberry Pi boots or if they crash. This can be frustrating and unreliable, especially for important tasks like servers or sensors. Supervisor solves this by automating the process, saving time and preventing downtime. It makes your Raspberry Pi projects more professional and dependable.

Where it fits

Before learning Supervisor, you should know basic Linux commands and how to run programs on Raspberry Pi. After mastering Supervisor, you can explore more advanced system management tools like systemd or Docker for managing complex applications.

Mental Model

Core Idea

Supervisor is like a helpful manager who watches your programs and restarts them automatically if they stop, so you don't have to.

Think of it like...

Imagine you have a plant that needs watering every day. Instead of remembering yourself, you hire a gardener who checks the plant regularly and waters it whenever needed. Supervisor is that gardener for your programs.

┌───────────────┐       ┌───────────────┐
│   Supervisor  │──────▶│  Program A    │
│ (Process     │       │ (Your script)  │
│  Manager)    │       └───────────────┘
│               │
│  ┌─────────┐  │       ┌───────────────┐
│  │Monitor  │  │──────▶│  Program B    │
│  │Restart  │  │       │ (Another task)│
└───────────────┘       └───────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding processes on Raspberry Pi

Concept: Learn what a process is and how programs run on Raspberry Pi.

A process is a program that is running on your Raspberry Pi. When you open a terminal and run a command, it starts a process. You can see running processes using the 'ps' command. Each process has an ID number called PID. Processes can run in the foreground (you see output) or background (hidden).

Result

You can identify and list running programs on your Raspberry Pi.

Knowing what a process is helps you understand what Supervisor manages behind the scenes.

2

Foundation

Running programs in the background

Concept: Learn how to start programs so they keep running after you close the terminal.

You can run a program in the background by adding '&' at the end of the command, like 'python3 myscript.py &'. This lets the program keep running while you use the terminal for other commands. However, if the Raspberry Pi restarts or the program crashes, it stops running.

Result

Programs can run without blocking your terminal, but are not automatically restarted.

Running programs in the background is a simple way to keep tasks running, but it lacks reliability and automation.

3

Intermediate

Installing and configuring Supervisor

🤔Before reading on: do you think Supervisor needs complex setup or just a simple config file? Commit to your answer.

Concept: Learn how to install Supervisor and write a basic configuration to manage a program.

Install Supervisor using 'sudo apt install supervisor'. Supervisor uses config files in '/etc/supervisor/conf.d/' to know which programs to manage. A simple config includes the program's command, autostart option, and restart policy. For example: [program:myapp] command=python3 /home/pi/myscript.py autostart=true autorestart=true stderr_logfile=/var/log/myapp.err.log stdout_logfile=/var/log/myapp.out.log After adding the config, run 'sudo supervisorctl reread' and 'sudo supervisorctl update' to apply changes.

Result

Supervisor starts managing your program, keeping it running and restarting if it crashes.

Understanding Supervisor's config files unlocks the power to automate and monitor your programs easily.

4

Intermediate

Using supervisorctl to control processes

🤔Before reading on: do you think supervisorctl commands affect all programs or only specified ones? Commit to your answer.

Concept: Learn how to start, stop, restart, and check status of programs using supervisorctl command.

The 'supervisorctl' command lets you control programs managed by Supervisor. Examples: - 'sudo supervisorctl status' shows all managed programs and their states. - 'sudo supervisorctl start myapp' starts the program. - 'sudo supervisorctl stop myapp' stops it. - 'sudo supervisorctl restart myapp' restarts it. This lets you manage your programs without rebooting or manually killing processes.

Result

You can control your programs easily from the command line anytime.

Knowing how to use supervisorctl gives you quick control over your programs, improving your workflow.

5

Intermediate

Logging and troubleshooting with Supervisor

🤔Before reading on: do you think Supervisor logs are stored in system logs or separate files? Commit to your answer.

Concept: Learn where Supervisor stores logs and how to use them to fix problems.

Supervisor saves program output and errors in log files specified in the config (e.g., stdout_logfile and stderr_logfile). These logs help you see what your program did and why it might have stopped. You can also check Supervisor's own logs in '/var/log/supervisor/'. If a program fails to start, logs are the first place to look.

Result

You can diagnose and fix issues with your programs using logs.

Understanding logging is key to maintaining reliable programs and quickly solving problems.

6

Advanced

Managing multiple programs with Supervisor groups

🤔Before reading on: do you think Supervisor can manage multiple programs as a single group? Commit to your answer.

Concept: Learn how to organize several programs into groups for easier control.

Supervisor lets you define groups of programs in the config. This means you can start, stop, or restart many related programs together. For example: [group:mygroup] programs=myapp1,myapp2,myapp3 You can then run 'sudo supervisorctl start mygroup:' to start all programs in that group. This is useful for projects with multiple components.

Result

You can manage complex projects with many programs more easily.

Grouping programs simplifies managing related tasks and reduces manual commands.

7

Expert

Supervisor internals and process lifecycle

🤔Before reading on: do you think Supervisor directly replaces init system or works alongside it? Commit to your answer.

Concept: Understand how Supervisor interacts with the operating system and manages process states internally.

Supervisor runs as a daemon process started by the init system (like systemd). It launches and monitors child processes for each program. It uses UNIX signals to stop or restart processes gracefully. Supervisor keeps track of process states in memory and updates status on commands. It does not replace the init system but works alongside it to manage user programs reliably.

Result

You gain deep understanding of how Supervisor ensures your programs stay running.

Knowing Supervisor's internal workings helps you troubleshoot complex issues and optimize configurations.

Under the Hood

Supervisor runs as a background service (daemon) on your Raspberry Pi. It reads configuration files to know which programs to manage. For each program, Supervisor starts a child process and keeps track of it. If the child process stops unexpectedly, Supervisor detects this and restarts it automatically. It uses system calls and signals to control processes and writes logs to files for monitoring. Supervisor communicates with users via the 'supervisorctl' command, which sends commands to the daemon.

Why designed this way?

Supervisor was designed to provide a simple, language-agnostic way to manage long-running programs without replacing the system's init system. It focuses on user-level process management with easy configuration and control. Alternatives like systemd are more complex and system-wide, while Supervisor is lightweight and flexible for developers. This design balances ease of use with powerful features.

┌───────────────┐
│   Init System │
│  (systemd)    │
└──────┬────────┘
       │ starts
       ▼
┌───────────────┐
│  Supervisor   │
│   Daemon      │
├───────────────┤
│ Reads configs │
│ Starts child  │
│ processes     │
│ Monitors them │
└──────┬────────┘
       │
       │ manages
       ▼
┌───────────────┐   ┌───────────────┐
│  Program A    │   │  Program B    │
│ (child proc)  │   │ (child proc)  │
└───────────────┘   └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does Supervisor replace the Raspberry Pi's init system? Commit to yes or no.

Common Belief: Supervisor replaces the system's init system and manages all system services.

Reality: Supervisor runs alongside the init system and only manages user-defined programs, not core system services.

Why it matters: Thinking Supervisor replaces init can cause confusion and improper system setup, risking system stability.

Quick: If a program crashes, does Supervisor always restart it immediately? Commit to yes or no.

Common Belief: Supervisor instantly restarts any crashed program without delay or limits.

Reality: Supervisor can be configured with restart limits and delays to avoid rapid restart loops that can overload the system.

Why it matters: Without restart limits, a buggy program can cause Supervisor to repeatedly restart it, consuming resources and causing instability.

Quick: Are Supervisor's log files stored in the system journal? Commit to yes or no.

Common Belief: Supervisor logs are part of the system's journal and cannot be customized.

Reality: Supervisor writes logs to separate files specified in its configuration, allowing flexible log management.

Why it matters: Assuming logs are in the system journal can lead to missing important program output and harder troubleshooting.

Quick: Can Supervisor manage programs written in any programming language? Commit to yes or no.

Common Belief: Supervisor only works with Python programs because it is written in Python.

Reality: Supervisor can manage any program or script regardless of language, as long as it can be started from the command line.

Why it matters: Limiting Supervisor to Python reduces its usefulness and prevents leveraging its full power for diverse projects.

Expert Zone

1

Supervisor's event notification system allows hooking custom scripts on process state changes, enabling advanced automation.

2

The 'priority' setting in Supervisor config controls the order programs start and stop, critical for dependent services.

3

Supervisor's XML-RPC interface enables remote control and integration with other management tools or dashboards.

When NOT to use

Supervisor is not ideal for managing system-level services or containers; systemd or Docker are better suited there. For very simple tasks, cron jobs or simple shell scripts may suffice. Also, Supervisor does not handle resource limits or security sandboxing, so use it alongside other tools for production environments.

Production Patterns

In production Raspberry Pi projects, Supervisor is used to keep sensor data collectors, web servers, or automation scripts running continuously. It is often combined with log rotation tools and monitoring systems to ensure reliability. Grouping related programs and using event listeners for alerts are common patterns.

Connections

Systemd service management

Alternative and complementary system-level process manager

Understanding Supervisor helps grasp systemd's more complex service management, as both manage processes but at different system levels.

Watchdog timers in embedded systems

Similar concept of monitoring and automatic recovery

Both Supervisor and hardware watchdogs aim to keep systems running by detecting failures and triggering restarts, showing a shared principle of fault tolerance.

Project management in teams

Both involve monitoring progress and restarting stalled tasks

Just like Supervisor watches and restarts programs, project managers monitor team tasks and intervene when work stalls, illustrating management principles across domains.

Common Pitfalls

#1 Not reloading Supervisor after changing config files

Wrong approach:

Editing '/etc/supervisor/conf.d/myapp.conf' but not running 'sudo supervisorctl reread' and 'sudo supervisorctl update'

Correct approach:

After editing config, run 'sudo supervisorctl reread' and 'sudo supervisorctl update' to apply changes

Root cause:

Assuming Supervisor automatically detects config changes without manual reload commands

#2 Running programs without specifying full paths

Wrong approach:

[program:myapp]
command=python3 myscript.py
autostart=true
autorestart=true

Correct approach:

[program:myapp]
command=/usr/bin/python3 /home/pi/myscript.py
autostart=true
autorestart=true

Root cause:

Not providing absolute paths causes Supervisor to fail finding the executable or script

#3 Ignoring log files leading to undiagnosed failures

Wrong approach:

Not setting 'stdout_logfile' and 'stderr_logfile' in config or not checking logs

Correct approach:

Always specify log files and review them regularly to catch errors early

Root cause:

Underestimating the importance of logs for troubleshooting and maintenance

Key Takeaways

Supervisor is a tool that automatically starts, stops, and restarts programs on your Raspberry Pi to keep them running reliably.

It works alongside the system's init system and manages user programs through simple configuration files.

Using supervisorctl commands, you can easily control your programs without rebooting or manual intervention.

Proper logging and configuration reloads are essential to maintain and troubleshoot your supervised programs.

Understanding Supervisor's internals and limitations helps you use it effectively in real-world Raspberry Pi projects.