Fix systemd service keeps restarting in a loop

Check service status and recent activity

Start by examining the current service state and recent restart activity.

sudo systemctl status your-service-name
sudo systemctl list-units --state=failed

Examine restart history with journalctl

View detailed logs to understand the restart pattern and identify error messages.

sudo journalctl -u your-service-name --since "1 hour ago"
sudo journalctl -u your-service-name -f

Check for restart rate limiting

Systemd has built-in rate limiting that can temporarily stop restart attempts after too many failures.

sudo systemctl show your-service-name --property=StartLimitBurst,StartLimitIntervalSec
sudo journalctl -u your-service-name | grep "start request repeated too quickly"

Fix exit code issues

Services that exit with non-zero codes trigger restarts. Check your application logs and fix the underlying issue causing the exit.

sudo journalctl -u your-service-name | grep "Main process exited"
sudo journalctl -u your-service-name | grep "code=exited, status="

Common exit codes and their meanings:

• Status 1: General application error
• Status 2: Misuse of shell commands
• Status 125: Container runtime error
• Status 126: Command not executable
• Status 127: Command not found

Fix permission errors

Services often fail due to incorrect file permissions or ownership issues.

sudo journalctl -u your-service-name | grep -i "permission denied"
sudo ls -la /path/to/your/application
sudo ls -la /var/log/your-service/

Fix ownership and permissions correctly:

sudo chown -R service-user:service-group /path/to/your/application
sudo chmod 755 /path/to/your/application
sudo chmod 644 /path/to/your/application/config.conf

Fix resource limit issues

Services may fail due to memory, file descriptor, or process limits. Check for resource-related errors and adjust limits.

sudo journalctl -u your-service-name | grep -i "out of memory\|too many open files\|resource temporarily unavailable"
sudo systemctl show your-service-name --property=LimitNOFILE,LimitNPROC,MemoryHigh

You can learn more about managing memory limits in our systemd memory limits tutorial.

Fix configuration file errors

Invalid configuration files cause applications to exit immediately. Validate your configuration syntax.

sudo journalctl -u your-service-name | grep -i "config\|syntax\|invalid"
For nginx
sudo nginx -t
For apache
sudo apache2ctl configtest

Test the application manually

Run your application manually to see error messages that might not appear in systemd logs.

sudo -u service-user /path/to/your/application --config /etc/your-service/config.conf
Or test with full environment
sudo systemd-run --uid=service-user --gid=service-group --setenv=PATH=/usr/local/bin:/usr/bin:/bin /path/to/your/application

Check environment variables

Services may need specific environment variables that are not available in the systemd context.

sudo systemctl show-environment
sudo systemctl cat your-service-name | grep Environment

Add missing environment variables to your service file:

[Unit]
Description=Your Service
After=network.target

[Service]
Type=simple
User=service-user
Group=service-group
Environment=PATH=/usr/local/bin:/usr/bin:/bin
Environment=NODE_ENV=production
ExecStart=/path/to/your/application
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Check service dependencies

Services may fail if required dependencies are not available or not started in the correct order.

sudo systemctl list-dependencies your-service-name
sudo journalctl -u your-service-name | grep -i "connection refused\|no such file\|not found"

Add proper dependencies to your service file:

[Unit]
Description=Your Service
After=network.target postgresql.service redis.service
Requires=postgresql.service
Wants=redis.service

Set appropriate restart policies

Configure when and how systemd should restart your service based on failure conditions.

[Service]
Restart only on failure (not on successful exit)
Restart=on-failure
Wait 10 seconds between restart attempts
RestartSec=10
Limit restart attempts to prevent infinite loops
StartLimitBurst=3
StartLimitIntervalSec=60

Restart policy options:

• no: Never restart (default)
• on-failure: Restart only on failure (recommended)
• on-abnormal: Restart on abnormal exit, signals, timeouts
• always: Always restart (use carefully)

Configure service timeouts

Set appropriate timeouts to prevent services from hanging during startup or shutdown.

[Service]
Time to wait for startup
TimeoutStartSec=30
Time to wait for graceful shutdown
TimeoutStopSec=15
Send SIGKILL if graceful shutdown fails
KillMode=mixed

Add health checks and watchdog

Configure systemd to monitor service health and restart if the service becomes unresponsive.

[Service]
Enable watchdog (requires application support)
WatchdogSec=30
Add a simple health check script
ExecStartPost=/usr/local/bin/service-health-check.sh
ExecReload=/bin/kill -HUP $MAINPID

Create a simple health check script:

#!/bin/bash
sleep 5
if curl -f http://localhost:8080/health > /dev/null 2>&1; then
    exit 0
else
    echo "Health check failed"
    exit 1
fi

sudo chmod 755 /usr/local/bin/service-health-check.sh

Reload and restart the service

Apply your configuration changes and test the service behavior.

sudo systemctl daemon-reload
sudo systemctl restart your-service-name
sudo systemctl enable your-service-name

Implement proper logging

Configure comprehensive logging to help diagnose future issues quickly.

[Service]
StandardOutput=journal
StandardError=journal
SyslogIdentifier=your-service

For more logging setup guidance, see our centralized logging tutorial.

Set up monitoring and alerting

Monitor service status and set up alerts for restart loops before they become critical issues.

# Check if service has restarted recently
sudo systemctl show your-service-name --property=ExecMainStartTimestamp,ActiveState
Monitor restart frequency
watch -n 5 'sudo systemctl status your-service-name | head -20'

Document service requirements

Create documentation about service dependencies, resource requirements, and known issues.

[Unit]
Description=Your Service - handles web requests on port 8080
Documentation=file:///usr/share/doc/your-service/README.md
Documentation=https://your-service-docs.example.com