Long lived Python/PHP daemon-script-wrapper with Supervisor instead Cron schedules

All novice Linux users and seasoned sysadmins know that Cron is a great simple solution to the problem of regularly calling the same application, it is present in almost every Linux system, has simple syntax and a number of logging and warning functions that make it a great addition to system administrator tools.

However, the Cron solution has some limitations.

1. The minimum time is limited to 1 minute – this is the shortest repeating interval that you can configure.
2. Minimal support to avoid concurrency.
3. It almost completely ignores the state of the command being called, that is, if the script fails, Cron will just keep trying (as it should), and you, as an administrator, should check and diagnose the problem.
4. But what if you need to implement a robust “long-lived” application in Python or PHP?
   Not everything can fit into convenient 1-minute time intervals for program code execution. In addition, there is an unnecessary overhead for loading the required libraries and opening file and socket descriptors each time the program is called for execution.

What if your application code needs to maintain a connection to a remote server and respond to events on an ad-hoc basis? Probably someone will say that these are already full-fledged demon programs. But the line between a daemon and a long-lived application in scripting languages ​​is not very clear.

Services

Server system administrators are familiar with the concept of a “service” in the operating system, or more specifically, daemon programs that run without interacting with user terminal I / O. MacOS (and NetBSD) has LaunchD, and Linux currently has a SystemD-based init system. All of these OS-level solutions allow administrators to write a small amount of configuration that instructs the operating system to launch applications at a specific point in the OS life cycle (at boot, at logon, etc.) and ensure that they continue to run.

Thus, all of the above systems take care of a number of key aspects of application performance. They all control the process (s) by a process ID, they offer logging and output control, and they usually have options to control restarts and the number of attempts to start in case of an error.

Supervisor

All of the above is fine, but writing service specifications can be tricky. It is required that you understand the various levels of execution and complexity of the OS-scale service management system and have a good Russian academic background in computer science.

Fortunately, however, there is a simpler intermediate solution that gives us all the benefits (and more) without the tedious configuration of Supervisor.
Its architecture includes three main parts:

1. supervisord is a daemon that keeps all software processes “running”.
2. supervisorctl is a command line client for the daemon where you can load, start, stop, restart, and delete new supervisor jobs.
3. web and XMLRPC interfaces to do most of what you can do from supervisorctl.

The last components are not interesting for us now.

Let’s assume that you have a Centos 7/8 server ready to go, otherwise you can deploy it to OpenStack Cloud 3hcloud, an Openstack Big Object Data Provider, or wherever you have a virtual machine or hardware server.

Installing Supervisor

You can notice, that supervisor implemented as a python application if will check list of files in package:

Obviously, any python application always have problems of compatibility, relability and performance. But in such context of managment this have not big meaning.Even if it will be python 2.7 in Centos 7.

Now you need check, that file of configuration contains clause ‘[include]’ for files of configurations of programm.

If everything is ok, we need activate and start supervisor daemon in OS.

Connection to service nad tests for a Python’s case

Supervisorctl is a simple CLI interface with decent built in help. Let’s start by confirming the version we have running. Type version and hit enter:

These are the commands that are available within Supervisor. Don’t worry too much about these at the moment as we are going to define a Program first so enter exit to return to the normal command line.

Our first Supervisor’s Program (endless process) on python

Supervisor jobs or processes are called programs and are defined using simple syntax, either in the main supervisor configuration file or through separate files. We saw an include directive in supervisor.conf, which allows us to configure various programs as separate files.

First we’re going to need a Python application to run. This can be any long lived Python script but for the sake of this post I’ve created a simple application that will generate a random integer between 1 and 10 and sleep for that number of seconds. Except if it generates a 10 in which case it will crash:

Let’s create this file in some suitable directory on the server and run:

Now we need to write a Supervisor Program to run this very useful script. Programs are written in ini format and start with the directive program and the name of your application.

Now need to write a Supervisor Program to run this script. These programs are written in ini format and start with the directive program and the name of application. Names should be simple!
Firstly we’ll need to type them into supervisorctl and secondly any excess punctuation can break the ini format. This is the simplest form of Supervisor Program we can create. Let’s make a new file in our /etc/supervisor.d/ directory so we can load it:

Now we jump back into supervisorctl terminal. We need to run a couple of commands in Supervisor. First we need to reread to load the new config file. Then we need to add supervisor_python_daemon to add and start it. Finally we’ll check the status of the job:

Our app works, but are we getting the added benefit of all this work? Well, let’s see.

If you’ve run the app outside of Supervisor (or read the source code), you’ll see that we have a number of script outputs. We have a print statement on initialization, log messages, and an application crash trace. If you run the script in the terminal, you expect all of this output to be reflected in your shell. But where is it in Supervisor?

One of the most convenient features offered by the standard configuration is the ability to redirect Standard Out and Standard Error to log files. Supervisor does this by default and creates the main supervisord.log file.

Keep watching and you should see the app crash time and time again, and every time, Supervisor brings it back to life. It’s a Pythonic Reincarnation Miracle!

Supervisor has many options for both the main application and the definition of programs. /Etc/supervisord.conf needs to be reviewed and edited as the file is well commented and the default configuration is likely to require some work.

To define a program, you should definitely read the documentation in the Program: X Section Values, which details all the parameters available to you. A couple of things to try:

1. Make a change to your application and use supervisorctl to reload and restart it
2. Limit your application to 3 runs before giving up (because we all know the definition of insanity)
3. Try spinning up multiple copies of your application
4. Redirect your application output to a different set of log files

Also please don’t forget that, having an application that comes back to life is no replacement for writing solid code in the first place. You should always monitor your logs and, where possible, fix issues that cause your applications to crash. Remember, exceptions should be exceptional.

If you have any questions or feedback, please feel free to reach out on email services@openstack.by.