Linux server incident response checklist for small teams

How to use this checklist

Order matters more than speed. Evidence disappears in a fixed order (memory and connections first, then logs, then disk), and some steps cannot be undone. The steps use the vocabulary of NIST SP 800-61 Rev. 3, which maps incident response onto the NIST Cybersecurity Framework 2.0: contain (RS.MI-01), eradicate (RS.MI-02), recover (RC.RP), communicate (RS.CO) and improve (ID.IM).

Two people are enough if you split the work: one acts on the server, the other keeps the log and talks to the provider, customers and anyone else. If the server holds customer or regulated data, or you are out of your depth at any step, bring in a professional incident responder early rather than late.

Signs a Linux server has been compromisedSecurity glossary and guides

1. Before anything else

• Do not reboot, shut down or reinstall. A reboot erases running processes, connections and anything held only in memory, and it can trigger the attacker's persistence.
• Do not delete files, kill processes, change passwords on the server or clear logs yet. Each destroys evidence or warns the attacker.
• Start an incident log off the server, in a shared document or a notebook: UTC time, who did what, the exact command, what it showed. NIST asks for exactly this record (RS.AN-06).
• Decide who is in charge. One person makes the calls; the other records them.
• Write down why you think this is an incident: the alert, abuse notice or odd process, with times.

2. Capture volatile data first

Before you change anything, capture the state that a reboot or a containment step would erase. This block only reads, and writes its results to two files in /root. Run it as root (sudo -i first).

# Run as root. Reads system state and writes it to two files in /root.
umask 077
OUT=/root/ir-$(hostname)-$(date -u +%Y%m%dT%H%M%SZ)
{
  echo "== collected $(date -u +%Y-%m-%dT%H:%M:%SZ) on $(hostname)"
  echo "== uptime"; uptime
  echo "== logged in now"; who -a; w
  echo "== recent logins"; last -F -n 50 || wtmpdb last
  echo "== processes"; ps -ww -eo pid,ppid,user,lstart,etime,pcpu,pmem,args --forest
  echo "== process executables"; ls -l /proc/[0-9]*/exe
  echo "== sockets"; ss -tunap
  echo "== addresses and routes"; ip addr; ip route
  echo "== mounts and disk use"; mount; df -h
  echo "== kernel modules"; lsmod
  echo "== ld.so.preload"; cat /etc/ld.so.preload
  echo "== running services"; systemctl list-units --type=service --state=running --no-pager
  echo "== timers"; systemctl list-timers --all --no-pager
  echo "== crontabs"; for u in $(getent passwd | cut -d: -f1); do crontab -l -u "$u" 2>/dev/null | sed "s/^/[$u] /"; done
} > "$OUT.txt" 2>&1
# The journal for the current boot (Rocky Linux and AlmaLinux keep it in memory by default)
journalctl -b --no-pager -o short-iso > "$OUT-journal.txt" 2>&1
sha256sum "$OUT.txt" "$OUT-journal.txt"

Copy both files to your workstation (they are owned by root, so copy them somewhere your login user can read first, or use your provider's console), check the SHA-256 there against what the server printed, and write both in your log. The files are themselves a change to the server: record their names and times so nobody later mistakes them for the attacker's.

Errors in the file are expected: kernel threads have no executable, and on Debian 13 last is replaced by wtmpdb, which the block falls back to. Then save the details of anything that already looks wrong, using the process and network guides.

Investigate a suspicious processCheck ports and connections

3. Contain

• Choose between isolation and restriction. Full isolation stops damage and data loss but alerts the attacker and takes the service down; restricting outbound traffic to the few destinations the application needs is often the better middle ground for a production service.
• Make the change at your provider's firewall or security group, not on the server, where an attacker with root can undo it. Keep your own SSH access, from one fixed address.
• Snapshot the disk through your cloud provider, after the volatile capture and before any cleanup. Label it with the incident and the UTC time, and restrict who can reach it. It is your evidence and your fallback.
• If a credential is being abused right now (a cloud key making calls, a database user exporting data), disable it at the provider or service side.
• If your server is attacking others, sending spam or mining, tell your provider what you have done, especially in reply to an abuse notice.

4. Scope: what did they reach?

NIST frames this as estimating and validating the incident's magnitude (RS.AN-08). Work through each area and write down what you find and what you could not check.

• Accounts and keys: who logged in, from where and when; new accounts; sudo changes; every SSH key that can log in.
• Persistence: cron, systemd services and timers, startup files, ld.so.preload, kernel modules.
• Payloads: suspicious processes and the files behind them, miners, web shells, recently changed files in /etc and web roots.
• Network: listeners, outbound connections, and your provider's bandwidth graphs for signs of data leaving.
• Data: which databases, files and secrets the compromised account could read. Assume it read everything it had access to.
• Other servers: anything sharing SSH keys, passwords, API tokens, a deploy user or configuration management with this one. Check those next with the same steps.
• Entry point: a web application bug, a guessed or reused SSH credential, or an exposed service. Finding it is what stops a repeat.

Unauthorized SSH keysCron and systemd persistenceCryptominersWeb shellsChanged files

5. Rotate credentials from a clean machine

• Work from a machine you trust, never the compromised server; anything typed or generated there may be captured.
• Generate new SSH key pairs on your own machine and replace the old public keys on every server that trusted them. If this server held private keys for other systems, remove their public halves there.
• Rotate cloud provider keys and API tokens found in environment files, application configuration or shell history.
• Change database passwords and application secrets the server held, and invalidate sessions if session signing keys were stored there.
• Replace TLS private keys and reissue certificates if root was compromised.
• Time it well: rotate after containment and before the rebuilt server goes live, so the attacker cannot simply read the new secrets from the old server.

6. Eradicate, or (usually) rebuild

If the attacker had root, or you cannot show they did not, rebuild. Binaries, logs, the package database and even the kernel may have been changed, so a root-compromised server cannot be shown to be clean. Cleaning in place is defensible only when root was never reached, the payload and its persistence are fully understood, and rebuilding is genuinely impossible.

• Build a new server from a known-good image or your provisioning code, fully patched, with new credentials.
• Close the entry point before it goes on the network: patch the application, close the exposed service, turn off SSH password logins.
• Restore data, not system files, from a backup taken before the compromise began. If you do not know when that was, go further back.
• Check the backup before you use it for the web shells, keys, cron entries and accounts you found (RC.RP-03 asks for exactly this).
• Keep the old server or its snapshot isolated until you are sure you no longer need it as evidence.

Linux server hardening checklistSSH hardening checklist

7. Recover and watch

• Bring the rebuilt server back in stages and watch it closely for the first days: logins, new keys, cron and systemd changes, outbound connections.
• Expect another attempt with whatever was taken: stolen credentials, the same vulnerability, a known address.
• Confirm that monitoring and backups run on the new server, then record the decision that recovery is complete (RC.RP-06).

8. Notify the people who need to know

• Customers whose data or service was affected, in plain language: what happened, what data, what you have done, what they should do.
• Your hosting provider, especially if your server attacked others or you received an abuse notice.
• Regulators, where the law applies to you. In the UAE, the Personal Data Protection Law (Federal Decree-Law 45 of 2021) includes an obligation to notify personal data breaches, and free-zone and sector rules can add their own. This is not legal advice: confirm your obligations and deadlines with a lawyer.
• Your insurer, if you hold cyber cover, before you commit to outside help.
• Use your incident log for every message, and do not state numbers or dates you have not established.

UAE PDPL and server security

9. Learn from it

• Within a week or two, write a short account: the timeline, the entry point, what worked, what did not, and how long each step took.
• Turn each lesson into one concrete change with an owner: a patch routine, key-only SSH, logs kept off the server, tested restores, an alert you were missing.
• Keep this checklist next to your provider's firewall, snapshot and support details, so the next incident starts faster.

Where SecAI fits, and where it does not

SecAI is software, not an incident response service or a staffed security operations centre. It does not investigate for you, clean servers, or prove a server is clean.

What it gives an investigator is a record held off the server, because the agent sends it to SecAI as it reports every 60 seconds: alerts with their evidence, including the text before and after for changed cron entries, sudoers, authorized_keys and systemd units; interactive SSH sessions with user, source, method, key fingerprint, and open and close times; and every address it blocked, listed with its evidence. Incident and security event history is kept for your plan's history period. The one action it takes without a person is blocking an address seen attacking (for SSH, three sightings within thirty minutes), only on a server set to Automatic, through the firewall the server already runs, for 6 hours the first time and longer for an address that keeps coming back (7 days at most), and never your own or a trusted address. That is not containment of a compromise, and everything else it proposes waits for approval.

Automated incident responseAuth log analyzer