How to investigate a suspicious process on a Linux server

Before you start: read, record, do not stop it

Stopping a process destroys most of what this guide reads: its memory, its connections, its environment and, if its binary was deleted, the only remaining copy. Save the output of every command somewhere other than the server, with the UTC time you ran it.

If an attacker has root, ps, ls and even /proc can be made to lie. The checks below still find most real problems, and the hidden-process section covers the cross-checks. If you find signs of root compromise, stop investigating in place and plan a rebuild.

The full triage sequenceIncident response checklist

Find the process worth looking at

# Top 20 by CPU, with start time and elapsed time
ps -eo pid,ppid,user,lstart,etime,pcpu,pmem,cmd --sort=-pcpu | head -20
# The same, sorted by memory
ps -eo pid,ppid,user,lstart,etime,pcpu,pmem,cmd --sort=-pmem | head -20
# The whole tree, so parents and children appear together
ps -ef --forest

lstart is the wall-clock start time and etime how long it has run; a start time that lines up with a login, a deploy or an odd log entry is worth writing down. pcpu is averaged over the process's whole life, so something that has only just started burning CPU looks low here; top shows current usage.

What stands out: a name you cannot place, a service account (www-data, apache, nginx, a database user) running something that is not its service, a shell or interpreter with an inline script, or a name that imitates a system component.

Read the real binary, directory and command line

The name in ps is whatever the process says it is. The kernel's view lives under /proc/<PID>; replace <PID> with the process ID.

# The executable the kernel actually loaded
sudo ls -l /proc/<PID>/exe
# The directory it is running in
sudo ls -l /proc/<PID>/cwd
# The full command line (arguments are separated by NUL bytes)
sudo cat /proc/<PID>/cmdline | tr "\0" " "; echo
# Its parent, owner and start time in one line
ps -o pid,ppid,user,lstart,cmd -p <PID>

Normally exe points at a packaged path such as /usr/sbin/nginx. Red flags: exe or cwd in /tmp, /var/tmp, /dev/shm or a hidden (dot) directory; exe ending in " (deleted)" for a path no package owns; a command line that does not match the binary, such as a process calling itself sshd whose exe points elsewhere.

dpkg -S <path> (Debian, Ubuntu) or rpm -qf <path> (Rocky Linux, AlmaLinux) shows whether a package owns a file. On some merged-/usr systems dpkg records /bin or /sbin rather than /usr/bin or /usr/sbin, so try both. An unowned binary is not proof of malice (hand-installed software, runtimes under /opt), but it moves the process up your list.

Sweep every process for temp-directory and deleted binaries

# Every process whose binary lives in a world-writable temp directory
sudo ls -l /proc/[0-9]*/exe 2>/dev/null | grep -E " -> /(tmp|var/tmp|dev/shm|run/shm)/"
# Every process whose binary has been deleted from disk
sudo ls -l /proc/[0-9]*/exe 2>/dev/null | grep "(deleted)"
# Services still running replaced binaries or libraries after upgrades (Debian, Ubuntu)
sudo needrestart -r l
# The same on Rocky Linux and AlmaLinux (from dnf-plugins-core)
sudo dnf needs-restarting

Anything from the first command deserves a close look: legitimate software almost never runs from a world-writable temp directory, and attackers use them because any user can write there.

The second is noisier. After an upgrade, services that were not restarted keep running the old, now-deleted binary, so /usr/sbin/sshd (deleted) is common and normal. needrestart ships with Ubuntu Server and may need installing on Debian. A deleted binary at a packaged path that needrestart also lists is an upgrade leftover; one at a path no package has ever owned is not.

Work out how it started

Whatever launched the process is what brings it back. Ask systemd which unit owns it and walk up the parent chain.

# Which systemd unit or session the process belongs to
systemctl status <PID> --no-pager
# Its control group, which also shows container membership
cat /proc/<PID>/cgroup
# Walk up the parent chain to PID 1
pid=<PID>; while [ -n "$pid" ] && [ "$pid" -gt 1 ]; do ps -o pid=,ppid=,user=,lstart=,args= -p "$pid"; pid=$(ps -o ppid= -p "$pid" | tr -d " "); done
# The tree around it (pstree is in psmisc, which may need installing)
pstree -aps <PID>

A named unit such as nginx.service means systemd started it; systemctl cat <unit> shows how. A session-N.scope means a login session, so match it to an SSH login. cron.service or crond.service, or a session-cN.scope on the RHEL family (where cron opens a session per job), points to a scheduled job. A docker-<id>.scope or containerd path means a container: its paths are as the container sees them, and docker top <container> maps it back.

On web servers the pattern that matters most is a shell or download tool whose parent is the web server or PHP: sh, bash, python, perl, curl or wget under nginx, apache2, httpd or php-fpm, running as www-data, apache or nginx. Web applications rarely start shells; this is what a web shell or a remote code execution bug looks like in the process table.

How to detect a web shellCron and systemd persistence

See what it has open and who it talks to

# Open files, including sockets and deleted files it still holds
sudo ls -l /proc/<PID>/fd
# Its connections and listeners (the trailing comma stops 123 matching 1234)
sudo ss -tunap | grep "pid=<PID>,"
# Optional: the same through lsof, which may need installing
sudo lsof -p <PID>

socket:[12345] entries in the fd listing are sockets; ss shows which are TCP or UDP and where they go. A long-lived connection from an unexpected process to an address with no business relationship to your application is the strongest single signal in this guide.

Reading ports and connectionsThe mining-pool pattern

Check its environment and hash the binary

# Environment at start (may contain passwords and tokens: read it, do not paste it anywhere)
sudo cat /proc/<PID>/environ | tr "\0" "\n"
# SHA-256 of the running binary; this works even if the file was deleted
sudo sha256sum /proc/<PID>/exe

Look for LD_PRELOAD (a library forced into the process), an odd PATH, HISTFILE set to /dev/null, or proxy variables you did not set. Application environments routinely hold database passwords and API keys, so never paste this output into a ticket, a chat or an AI tool; note only the variable names that matter.

Look the hash up later in a malware hash database from your own workstation. A match identifies known malware; no match proves nothing, because attackers recompile. If the binary is deleted, the running process holds the only copy: copying /proc/<PID>/exe off the server preserves it, and that copy is live malware, never to be run.

Names that imitate kernel threads and services

Attackers name processes after things you expect to see, such as [kworker/0:2], kswapd0 or sshd; a program can overwrite its own arguments to show any name (MITRE ATT&CK T1036.011). A genuine kernel thread has kthreadd (PID 2) as its parent and no executable.

# Parent of one process: a real kernel thread shows PPID 2
ps -o pid,ppid,user,cmd -p <PID>
# A real kernel thread has no executable, so this fails with "No such file or directory"
sudo ls -l /proc/<PID>/exe
# Bracketed names whose parent is not kthreadd (expect only PID 2 itself and zombies)
ps -eo pid,ppid,user,cmd | awk '$4 ~ /^\[/ && $2 != 2'

A bracketed name with another parent and a readable exe is a userland program in disguise, a strong red flag on its own. Zombies also show in brackets, followed by <defunct>; they have already exited and are a bug in their parent, not an attack. The same test applies to anything called sshd or systemd-journald: if exe is not the packaged binary, it is not what it says. On OpenSSH 9.8 and later, per-connection processes are sshd-session (and sshd-auth from 10.0), from the same package.

If processes may be hidden

A rootkit can hide processes with a library in /etc/ld.so.preload, a replaced ps, or code in the kernel. A cheap cross-check compares the PIDs /proc lists with the ones ps reports.

# A preload file should not exist on most servers
cat /etc/ld.so.preload 2>/dev/null
# PIDs that /proc lists and ps does not (run it twice; short-lived PIDs come and go)
diff <(ls /proc | grep -E "^[0-9]+$" | sort -n) <(ps -e -o pid= | tr -d " " | sort -n)

Lines starting with < are PIDs in /proc that ps did not show. One that appears on every run is being hidden. If /etc/ld.so.preload exists and you did not create it, assume every tool on the server is compromised. Neither check sees a kernel rootkit, because then /proc lies too: snapshot the disk through your provider and plan a rebuild.

Common false positives

• Kernel threads such as [kworker/0:1] and [ksoftirqd/0]: parent 2, no readable exe.
• Upgrade leftovers: deleted binaries at packaged paths until the service restarts.
• Monitoring, backup and cloud agents: root, long-lived, outbound connections, child processes. SecAI's agent is /usr/local/bin/security-agent, run by the secai-agent service, and it starts short-lived system tools as children. Confirm each agent against what you installed.
• Control panels: cPanel, Plesk and DirectAdmin run many daemons and jobs as root and as each site's user, from /usr/local/cpanel, /usr/local/psa or /opt/psa, and /usr/local/directadmin.
• Containers: Docker workloads appear in the host's process list with their own paths; read /proc/<PID>/cgroup first.
• Busy legitimate services: a database or PHP worker at high CPU during a backup or a traffic spike looks like a miner in ps. The binary, the parent and the connections tell them apart; the CPU figure does not.

What to record, what not to do yet, when to escalate

• Record PID, PPID, user, start time, exe path and whether it is deleted, cwd, command line, unit or cgroup, parent chain, sockets, SHA-256, and the UTC time of each observation.
• Do not kill the process, delete its file or reboot yet. All three destroy evidence, and killing a process whose persistence you have not found usually just restarts it.
• Do not run the binary, or upload it or its scripts to a public service if they could contain your credentials or data.
• If it is causing harm now, cut its network access at your provider's firewall or security group: that stops the damage and keeps the evidence.
• Escalate or plan a rebuild if it ran as root, you find an ld.so.preload entry, an impostor kernel thread or hidden PIDs, you find persistence, the server holds customer or regulated data, or you cannot explain how it started.

What to do next, in order

What SecAI reports about processes

SecAI reports three situations from this guide as alerts with evidence. A process whose executable resolves into /tmp, /var/tmp, /dev/shm or /run/shm is reported with its PID, path and command line, deleted or not; a deleted binary anywhere else is deliberately not flagged, because package upgrades cause that constantly. CPU, memory or outbound data volume above the server's own baseline for five consecutive reports is reported, and closes itself after five calm ones. A hidden process, where the process table and the kernel disagree, is reported too.

None of these proposes an action; a person investigates with the steps above. (On a server opted in to survival mode's Protect setting, a process group exhausting the server may be throttled or frozen, never killed.) SecAI is software, not an incident response service. To check another server's exposure before installing anything, the free server assessment is a read-only first look.

Linux threat detectionDetection coverageFree server assessment