Linux and networking essentials

What you need to know

• Processes and signals: every program is a process with a user, a parent and a state; SIGTERM asks it to stop, SIGKILL forces it.
• Files and permissions: owner, group and mode bits decide who can read, write or execute; the principle of least privilege starts here.
• Services: systemd starts, restarts and logs long-running programs; journalctl reads their logs.
• The network stack: an IP address identifies a host, a port identifies a service, TCP gives reliable streams, UDP gives datagrams.
• DNS turns names into addresses, and most mysterious outages are DNS, certificates or time.
• TLS wraps a connection so it is encrypted and the server is authenticated by a certificate chain.

A debugging routine

dig +short api.example.com                      # does the name resolve, and to what?
curl -sv --max-time 5 https://api.example.com/health   # connect, TLS and HTTP in one go
ss -ltnp | grep :8080                           # is anything listening on this host?
systemctl status myapp --no-pager               # is the service running?
journalctl -u myapp --since "10 min ago" -n 100 # what did it say?
openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null 2>/dev/null | openssl x509 -noout -dates

Permissions and a service unit

[Unit]
Description=My app
After=network-online.target

[Service]
User=myapp
Group=myapp
ExecStart=/opt/myapp/bin/myapp
Restart=on-failure
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/myapp

[Install]
WantedBy=multi-user.target

Habits

• Learn to read before you change: logs, status and metrics first, edits second.
• Never run a service as root if it does not need to.
• Keep clocks synchronized (NTP): expired tokens and certificates are often a clock problem.
• Write down the commands that worked. A runbook starts as a shell history.