If you are reading this because something feels wrong with a server right now, start with one instruction: do not reboot it, and do not start deleting things. Both destroy evidence you will want, and neither fixes a compromise. Work through the checks below in order first.
Most compromised servers show themselves in mundane ways, a bill higher than expected, a provider abuse notice, a site that has become slow. The dramatic signals are rare.
Key Takeaways
- Do not reboot and do not clean until you have collected evidence. Memory-resident artefacts and process state are gone after a restart.
- Work outside-in: network behaviour first, then processes, then persistence, then accounts, then logs.
- Assume your tools may be lying if root was obtained.
psandlscan be replaced. - Absence of evidence in local logs is not evidence of absence, because local logs are editable by root.
- Finding the entry point matters more than removing the payload. Removal without root cause means it returns.
Table of Contents
- Before you touch anything
- Step 1: network behaviour
- Step 2: processes
- Step 3: persistence
- Step 4: accounts and access
- Step 5: logs, and why they may be fiction
- What to do next
Before you touch anything
Three things, in this order.
Do not reboot. A reboot clears running processes, open connections, and anything resident only in memory. It also frequently triggers whatever persistence the attacker installed, which is the opposite of helpful.
Decide whether to isolate. Taking the server off the network stops ongoing damage and exfiltration, and also alerts the attacker while ending your ability to observe. For a business-critical service, restricting outbound traffic to known destinations is often a better middle option than full isolation.
Start writing things down. Timestamps, commands run, output. If this turns out to involve client data you may have a notification obligation with a deadline, and reconstructing a timeline from memory afterwards is not adequate. Save command output to a file on a different machine where possible.
Step 1: network behaviour
Start here rather than with processes, because network activity is the hardest thing for an attacker to hide. A payload must communicate to be useful.
ss -tupn state established
ss -tulpn
What matters in that output:
- Established outbound connections to addresses with no business relationship to your application
- A listener on a port you did not configure, particularly on
0.0.0.0rather than localhost - Any long-lived connection owned by a process that has no reason to make one
- Connections from your web server user to external hosts
Then check what the server has been sending. A sudden sustained increase in outbound traffic is the signature of both mining and data exfiltration:
cat /proc/net/dev
ip -s link show
If your provider gives you bandwidth graphs, look at those too, they are outside the server and therefore trustworthy in a way that on-host figures are not.
Persistent outbound connections to one destination that never varies is the classic mining pattern, covered in how to detect a cryptominer on a Linux server.
Step 2: processes
ps auxf
ps -eo pid,ppid,user,%cpu,%mem,etime,cmd --sort=-%cpu | head -20
Read for:
- Processes whose name imitates a system component but sits in the wrong place in the tree
- Anything running from
/tmp,/dev/shm,/var/tmp, or a home directory - A process owned by
www-dataor another service account that is a shell or an interpreter - Names with brackets faking kernel threads
Verify the real path, since the displayed name is attacker-controlled:
ls -l /proc/<PID>/exe
cat /proc/<PID>/cmdline | tr '\0' ' '; echo
ls -l /proc/<PID>/cwd
An exe link pointing to a deleted file is significant on its own, the binary was unlinked after launch specifically to frustrate this step.
Now the important caveat. If the attacker obtained root, ps may have been replaced or a library preloaded to hide specific processes. Check:
cat /etc/ld.so.preload 2>/dev/null
If that file exists and you did not create it, stop trusting everything on this machine. A library listed there is injected into every dynamically linked process, and your investigation tools are among them. Cross-check process listings against /proc directly:
ls /proc | grep -E '^[0-9]+$' | wc -l
ps -e --no-headers | wc -l
A meaningful mismatch between those two counts means something is hiding processes from ps.
Step 3: persistence
The payload you found is replaceable. The persistence mechanism is what brings it back.
# cron, all users and system-wide
for u in $(cut -f1 -d: /etc/passwd); do crontab -l -u "$u" 2>/dev/null | sed "s/^/[$u] /"; done
ls -la /etc/cron.d/ /etc/cron.*/ /var/spool/cron/crontabs/ 2>/dev/null
# systemd, including recently modified units
systemctl list-timers --all --no-pager
find /etc/systemd/system /usr/lib/systemd/system -newermt "-14 days" -type f 2>/dev/null
ls -la ~/.config/systemd/user/ 2>/dev/null
# shell startup and preload
grep -rn "curl\|wget\|base64\|nc \|/tmp/" /etc/profile /etc/profile.d/ /root/.bashrc /home/*/.bashrc 2>/dev/null
cat /etc/ld.so.preload 2>/dev/null
ls -la /etc/ld.so.conf.d/
Recently modified files across the whole filesystem are worth a look, bounded to avoid drowning:
find /etc /usr/local /opt -newermt "-14 days" -type f 2>/dev/null | head -50
These locations are static on a healthy server, which is exactly why they make good monitoring targets, see what file integrity monitoring actually catches on Linux.
Step 4: accounts and access
awk -F: '$3 == 0 {print}' /etc/passwd # UID 0 accounts other than root
tail -20 /etc/passwd # recently added users
getent group sudo wheel adm 2>/dev/null
ls -la /etc/sudoers.d/
last -20
lastlog | grep -v "Never logged in" | tail -20
And SSH keys, the most durable persistence there is, because it survives password rotation entirely:
for d in /root /home/*; do
[ -f "$d/.ssh/authorized_keys" ] && echo "== $d" && cat "$d/.ssh/authorized_keys"
done
Any key you cannot account for means the attacker retains access regardless of what else you fix. The full treatment of this territory is in detecting privilege escalation on a Linux server.
Step 5: logs, and why they may be fiction
Check them, and hold the results loosely:
grep "Accepted" /var/log/auth.log | tail -30
grep -c "Failed password" /var/log/auth.log
journalctl --since "7 days ago" | grep -iE "error|fail|denied" | tail -40
For a web-facing server, the access log usually contains the entry point. Look for requests to paths that should not exist:
grep -E "\.php|\.env|/admin|/wp-|/\.git" /var/log/nginx/access.log | tail -50
awk '$9 ~ /^(200|500)$/' /var/log/nginx/access.log | grep -E "\.php" | tail -30
A 200 response to a request for a PHP file in an upload directory is close to conclusive, that is a webshell being used, which is the subject of how to detect a webshell on a Linux server.
The caveat that matters: root can edit or truncate any of these files. Gaps in a log are themselves a finding. Check whether the file's modification time is consistent with its last entry, and whether log rotation history is intact. This is the entire argument for shipping logs off the host as they are generated, telemetry that has already left the server cannot be edited retroactively.
What to do next
Identify the entry point. This is the step people skip and the only one that prevents recurrence. Realistically it is a web application vulnerability, guessed or reused SSH credentials, or an unauthenticated service exposed publicly, Redis, Docker's API, Elasticsearch, a database. Do not stop at removing the payload.
Rotate every credential the server held. SSH keys, API tokens, database passwords, cloud provider keys, anything in an environment file. Assume all of it was read, because there is usually no way to prove otherwise.
Prefer rebuilding to cleaning. Once code has run as root, proving the machine is clean is genuinely harder than reprovisioning it. Rebuild from a known-good image, restore data selectively after inspection, and apply hardening before it goes back on the network. Cleaning in place is defensible only when rebuilding is truly impossible.
Check the rest of the fleet. Whatever got in probably has the same opportunity elsewhere, reused credentials, the same unpatched package, the same exposed service. Run the checks above across every server, not just this one.
Then harden. A practical Linux server hardening checklist for small teams in priority order, and how to track CVEs for the packages installed on your Linux servers so the next known vulnerability does not sit unpatched.
SecAI is built to make this triage unnecessary by catching the sequence earlier: file integrity monitoring on the persistence paths above, per-server process and listener baselining so a new binary or port is flagged rather than discovered weeks later, egress anomaly detection for the outbound patterns in step one, webshell detection in web-writable directories, and telemetry shipped off-host as it is generated so a root-level attacker cannot retroactively edit the record. Agent tamper-resistance means silence is treated as a finding, alongside the rest of the detection stack.
Once the immediate incident is handled, the comprehensive Linux server security software guide covers what should have caught it.
Frequently Asked Questions
Should I reboot a server I think is compromised? No. It destroys process and memory evidence and typically re-triggers whatever persistence was installed. Collect first.
How can I be certain a server is clean after cleaning it? You cannot, fully. That is the honest answer and the reason rebuilding from a known-good image is the recommended path when it is available.
How long do attackers usually stay before being noticed? On small business infrastructure, often a long time, because nothing is watching. Most discoveries come from a side effect, a bill, an abuse notice, degraded performance, rather than from detection.
Do I need to notify anyone? If personal data may have been accessed, likely yes, and there are deadlines. This is why documenting timestamps from the beginning matters. Take legal advice on your specific obligations rather than guessing.
What single check gives the most information fastest?
Established outbound connections with their owning process, via ss -tupn state established. Payloads must communicate, and that is the hardest thing for them to hide.
If you would rather catch the next one while it is happening, this is what continuous monitoring on a Linux server actually watches, and what it does automatically when something matches.