Setting up systemd services

I often need to set up a VM with a server to run some ML models. While I could package things in Docker and deploy them like that, whenever possible, I like to do things more directly, using systemd services.

To create a systemd service, create a file called /etc/systemd/system/my_service.service. This will create a service that will be started by the root user, but it can be run under another user.

There are three necessary components of a systemd service: unit, service and install.

[Unit]
Description=Service to run my fancy API
After=network.target
StartLimitIntervalSec=100
StartLimitBurst=5

[Service]

Restart=always
RestartSec=10
User=ubuntu
Group=www-data
WorkingDirectory=/home/ubuntu/app
Environment="PATH=/home/ubuntu/.venv/bin"
Environment="API_KEY=123456"
ExecStart=/home/ubuntu/app/.venv/bin/uvicorn main:app --my_params
StandardOutput=append:/var/log/app/stdout.log
StandardError=append:/var/log/app/std_error.log

[Install]
WantedBy=multi-user.target

First, we have to describe the service. Then we tell it when to start: after the network is up and running. We tell it to stop restarting if it crashes 5 times in 100 seconds.

Then we describe the actual service. We tell it to always restart (keeping the above mentioned limits), waiting 10 seconds between restart. We can specify a user, group, working directory and environment variables in which to run the program.

With ExecStart we can specify the program to run. If it's a simple FastAPI server, I usually write out the full path here. But, if it's something more complicated, where I might have to play around with the command more often, I create a bash file and start the service using that: ExecStart=bash /home/ubuntu/start_service.sh.

Then we redirect the standard output and the error stream to files. The last bit I have no idea what it is, but it's needed.

To start the service you have to first run sudo systemctl daemon-reload. You will have to run this every time you change the service definition file. Then you can start the service with sudo systemctl start my_service (the name of the file, without .service).

To enable starting it automatically on startup, you can run sudo systemctl enable my_service.

To see the status of the service: sudo systemctl status my_service. If it's crashing, you can look at journalctl -xe or you can check the log files you configured.

I recommend always writing out full paths, especially for the Python interpreter, to avoid any ambiguities as to what will run your program or what exactly will be loaded.

And now you know how to create a simple systemd service!