An application started from a terminal belongs to that terminal. Close the session and it stops. Reboot the machine and it is gone, with nothing to explain where it went.
Defining it as a service hands it to the system instead.
A minimal unit
Create /etc/systemd/system/myapp.service:
[Unit] Description=My Application After=network.target [Service] Type=simple User=myapp WorkingDirectory=/var/www/myapp ExecStart=/usr/bin/node /var/www/myapp/server.js Restart=always RestartSec=5 [Install] WantedBy=multi-user.target
Then:
systemctl daemon-reload systemctl enable --now myapp systemctl status myapp
enable makes it start at boot; --now also starts it immediately. Forgetting enable is why a service that has been working perfectly disappears after the first reboot.
The absolute path
ExecStart must be a full path. A unit runs with almost no environment, so node, python or php alone will not be found even though they work in your shell.
which node
"Command not found" from a unit that runs fine by hand is always this. Version managers make it worse, because the interpreter your shell uses is not the one at the system path, use the real path, or the failure returns whenever the shell configuration changes.
Restart policy
Restart=always brings the process back whenever it exits, for any reason.
Restart=on-failure restarts only on a non-zero exit, which is right for something that may legitimately finish.
RestartSec=5 waits between attempts. Without it, an application that fails instantly is restarted in a tight loop, which fills the journal and consumes the processor.
systemd also gives up after several rapid failures, which is deliberate. A service that will not start should stop trying rather than mask the fault.
Do not run it as root
User= is the most valuable line in the file.
A service running as root means any compromise of that application is a compromise of the machine. Create a user with no login shell, give it access to the directories it needs, and nothing else:
useradd -r -s /usr/sbin/nologin myapp chown -R myapp:myapp /var/www/myapp
The exception is a service that must bind to a port below 1024, and the answer there is a capability or a reverse proxy rather than root. Setting up nginx as a reverse proxy sets out the usual arrangement.
Logs
journalctl -u myapp -n 50 journalctl -u myapp -f journalctl -u myapp --since "1 hour ago"
Anything the application writes to its output goes here, which is why a unit is also the fastest way to stop losing an application's own error messages.
The journal has a size limit and rotates, which is usually what you want. Managing logs and log rotation deals with adjusting it.
Environment variables
Environment=NODE_ENV=production EnvironmentFile=/etc/myapp.env
Prefer the file for anything secret, and give it restrictive permissions, values written directly into the unit are readable by anyone who can read the unit, which is everyone.
chmod 600 /etc/myapp.env chown myapp:myapp /etc/myapp.env
Dependencies
After= controls ordering but does not require anything. If the service genuinely cannot function without the database, state it:
After=mysql.service Requires=mysql.service
Without Requires, the service starts anyway, fails to connect, and restarts in a loop until the database happens to be ready. It usually works, and it produces a confusing boot.
After every edit
systemctl daemon-reload systemctl restart myapp
systemd reads unit files into memory, so editing the file alone changes nothing. A change that "did not take effect" is nearly always a missing daemon-reload.
Scheduling jobs with cron and systemd timers covers the periodic case, which uses the same unit format.
When it will not start
The status output usually contains the answer, and the journal always does:
systemctl status myapp -l --no-pager journalctl -u myapp -n 50 --no-pager
Three failures cover most cases, and each has a distinctive message.
Exit code 203 means the command could not be executed. A wrong path, or a file without the execute bit. This is the absolute-path problem in its most common form.
Exit code 200 or 217 means the user or group in the unit does not exist. Creating the service user is a step that is easy to skip when copying a unit from elsewhere.
Permission denied on a directory means the service user cannot read or write where it needs to. Confirm ownership rather than loosening permissions. Understanding file permissions and ownership explains the difference.
Give a failure somewhere to go
A service that fails repeatedly gives up and stays stopped, which is correct behaviour and silent by default.
OnFailure=notify@%n.service
That runs a unit of your own when this one fails, which can send a message. Without something like it, the first indication is a customer, and the machine has been fine the whole time; nothing about the server is wrong, one service simply is not running.
The alternative is external monitoring that checks the thing the service provides in place of the service itself, which is better because it also catches a process that is running and not working. Managing VPS resources and monitoring performance sets out arranging it.
Restarting without dropping requests
For anything serving traffic, a plain restart drops whatever was in progress.
ExecReload=/bin/kill -HUP $MAINPID
systemctl reload myapp
Where the application supports it, reload replaces the configuration without stopping. Where it does not, the honest arrangement is a proxy in front that can hold requests briefly. Setting up nginx as a reverse proxy picks it up from there.
Bounding what the service can consume
MemoryMax=2G CPUQuota=50% TasksMax=256
These matter most for anything processing input from outside. A video converter, an import job, an image processor. A single request should not be able to exhaust the machine's memory, because the system then starts terminating processes and does not necessarily choose the one at fault.
Combined with a dedicated user and a restricted directory, that is most of what containment means on an ordinary server. There is more on the case where the input is genuinely hostile in ffmpeg-hosting/how-to-process-untrusted-video-uploads-safely.html">processing untrusted video uploads safely.
Start it after the things it needs, not just after the network
Ordering by network availability is the usual first attempt and it is not sufficient, because a network that exists is not a database that is ready.
[Unit] After=network-online.target postgresql.service Wants=network-online.target Requires=postgresql.service
The difference between wanting and requiring matters. A wanted unit that fails leaves yours running. A required one that fails stops yours too, which is correct when the service genuinely cannot work without it.
Even with correct ordering, a dependency that is running is not necessarily accepting connections. Applications that retry their first connection start reliably; ones that exit on the first failure need a restart policy that gives them another attempt rather than a longer wait.
Tell the system when the service is genuinely ready
By default the system considers the service started the moment the process exists, which is usually earlier than the moment it can answer.
That matters when something else is ordered after it, because the next service starts against an application that is still loading.
Type=notify # ya da: ExecStartPost=/bin/sh -c 'until curl -sf http://127.0.0.1:8080/health; do sleep 1; done' TimeoutStartSec=60
A readiness check turns starting into a state that means something. Without one, a boot sequence that looks correct produces a service that fails for the first thirty seconds after every restart, and only the visitors in that window see it.
Read the failure rather than the status
A service that will not start reports the same short message for a dozen different causes, and the detail is one command away.
systemctl status myapp --no-pager -l journalctl -u myapp -n 50 --no-pager systemd-analyze verify /etc/systemd/system/myapp.service
The verification command finds mistakes in the unit file itself, including options that were silently ignored because they were misspelled or placed in the wrong section, which is a common and invisible fault.
Exit codes are worth recognising too. A permission problem, a missing binary and an application that started and chose to exit all look the same from the status line and are plainly different in the journal. Managing logs and log rotation goes into reading it efficiently.