Before you start
Every command here lists or reads. Do not delete or disable what you find yet: removing an entry before you have recorded it loses evidence, can warn whoever put it there, and may trigger a fallback they planted. Copy each finding off the server with the time you saw it.
Look at everything, not just the first suspicious entry. Attackers often install two or three mechanisms so that removing one changes nothing. If root was compromised, files and tools can be altered to hide entries, so what you see is a lower bound.
Per-user crontabs
Each account can have its own crontab, stored in /var/spool/cron/crontabs/ on Debian and Ubuntu and in /var/spool/cron/ on Rocky Linux and AlmaLinux. Only root can read them all.
# Every account's crontab, prefixed with the account name
for u in $(getent passwd | cut -d: -f1); do sudo crontab -l -u "$u" 2>/dev/null | sed "s/^/[$u] /"; done
# The spool files with owners and times (Debian, Ubuntu)
sudo ls -la /var/spool/cron/crontabs/
# The same on Rocky Linux and AlmaLinux
sudo ls -la /var/spool/cron/The spool listing catches crontabs for names the loop missed, such as an account that has since been deleted, and gives each file's modification time. Service accounts such as www-data, apache, nginx, redis or postgres rarely need a crontab; one that has one needs an explanation.
System cron, anacron and at
# The system crontab and drop-in crontabs (these have a user field)
sudo cat /etc/crontab
sudo ls -la /etc/cron.d/
sudo cat /etc/cron.d/*
# Scripts run hourly, daily, weekly and monthly
sudo ls -la /etc/cron.hourly/ /etc/cron.daily/ /etc/cron.weekly/ /etc/cron.monthly/
# anacron (installed by default on the RHEL family)
sudo cat /etc/anacrontab
# at jobs, if at is installed: list them, then print one by number
sudo atq
sudo at -c <job-number>Lines in /etc/crontab and /etc/cron.d have a sixth field before the command: the account it runs as. The periodic directories hold scripts, so read any you do not recognise in full. On Debian and Ubuntu, run-parts skips file names containing a dot, so backup.sh in /etc/cron.daily never runs there; a new extensionless file is the one to look at.
at -c prints the job's saved environment before its command. Read the last lines for the command, and do not paste the rest anywhere, since it can hold secrets.
Systemd services and timers
A systemd unit restarts its process when it dies and starts it at every boot, which makes it the more common choice today (MITRE ATT&CK T1543.002 for services, T1053.006 for timers).
# Units enabled to start at boot
systemctl list-unit-files --state=enabled --no-pager
# Every timer, including inactive ones, with last and next run
systemctl list-timers --all --no-pager
# Local units and their enablement links (packages install to /usr/lib/systemd)
sudo ls -la /etc/systemd/system/ /etc/systemd/system/*.wants/
# Drop-ins and overrides that change packaged units
systemd-delta --type=extended,overridden,redirected --no-pager
# Unit files whose content (mtime) or metadata (ctime) changed in the last 14 days
sudo find /etc/systemd /usr/lib/systemd /run/systemd/system -type f \( -mtime -14 -o -ctime -14 \) -ls 2>/dev/nullCompare each enabled name with what you installed, and for each timer read what it activates (the ACTIVATES column) and how often. In /etc/systemd/system, enablement entries are usually symlinks into /usr/lib/systemd/system; a regular unit file there was written by a person, an installer or a configuration tool, and must be read.
systemd-delta lists drop-in .conf files that extend packaged units. A drop-in that adds ExecStartPre= or replaces ExecStart= on a service you trust, such as ssh or nginx, is a quiet way to persist. Upgrades refresh many files under /usr/lib/systemd, so recent times there are often legitimate; compare them with /var/log/apt/history.log or dnf history.
Read a unit before judging it
# The unit file and every drop-in, with each file's path as a comment
systemctl cat <unit>
# Its state, main process and recent log lines
systemctl status <unit> --no-pager
# Does a package own the unit file? Use the path systemctl cat printed on its first line
dpkg -S <unit-file-path>
# The same on Rocky Linux and AlmaLinux
rpm -qf <unit-file-path>Read ExecStart=, ExecStartPre= and ExecStartPost= (what runs), User= (as whom), Restart= (whether it comes back when killed) and WantedBy= (when it starts). A Description= copied from a real service means nothing; the ExecStart path is what counts. Packages install their units under /usr/lib/systemd/system (or /lib/systemd/system), so a packaged service's file is found there and dpkg -S or rpm -qf names its package. A unit file that sits in /etc/systemd/system itself was written by an administrator, an installer script or an attacker, and the package query answers "no path found" or "not owned by any package". An unowned unit is not necessarily bad, but every one of them needs an explanation.
Other places code runs at boot or login
# User units and per-user startup files, for every home directory
for h in $(getent passwd | cut -d: -f6 | sort -u); do sudo ls -la "$h/.config/systemd/user" "$h/.bashrc" "$h/.profile" "$h/.bash_profile" "$h/.bash_logout" 2>/dev/null; done
# Accounts whose user units start at boot without a login
sudo ls -la /var/lib/systemd/linger/
# System-wide startup files
sudo ls -la /etc/profile.d/ /etc/update-motd.d/
# The old boot script (on the RHEL family it exists but runs only if executable)
sudo ls -la /etc/rc.local /etc/rc.d/rc.local
# Libraries forced into every process; should not exist on most servers
ls -la /etc/ld.so.preload
# udev rules that run programs when devices appear
sudo grep -rn "RUN" /etc/udev/rules.d/A file in /var/lib/systemd/linger/ means that user's own systemd units start at boot with nobody logged in, which turns a user unit in ~/.config/systemd/user into boot persistence for an unprivileged account. Shell startup files run only when someone logs in, but they run as whoever logs in, root included. On Ubuntu and Debian the scripts in /etc/update-motd.d run as root at each login to build the message of the day.
On Debian and Ubuntu /etc/rc.local does not exist unless someone created it. /etc/ld.so.preload should normally not exist at all: a library listed there is loaded into every dynamically linked program, including the tools you are investigating with (MITRE ATT&CK T1574.006). If it exists and you did not create it, treat the whole server as compromised.
What malicious entries usually look like
- A download piped straight into a shell: curl or wget fetching a script from a bare address or a paste site and passing it to sh or bash, with output sent to /dev/null.
- An encoded blob: a long base64 string decoded with base64 -d and piped into a shell, so the command cannot be read in the file.
- A binary in /tmp, /var/tmp, /dev/shm or a hidden directory, including directories named only with dots or spaces.
- @reboot lines, and every-minute schedules (* * * * *) that restart something the moment it dies.
- A unit with Restart=always and an ExecStart outside /usr, or a name one letter away from a real service.
- Entries that kill other processes by name (often rival miners or security agents) or rewrite cron, authorized_keys or ld.so.preload themselves.
# Quick sweep of cron and unit files for the usual patterns
sudo grep -rnE "curl|wget|base64|/dev/shm|/tmp/|/var/tmp/|@reboot|nohup" /etc/crontab /etc/cron.d /etc/cron.hourly /etc/cron.daily /etc/cron.weekly /etc/cron.monthly /var/spool/cron /etc/systemd/system
# The same sweep over shell startup files, run as root so every home directory is readable
sudo sh -c 'grep -nE "curl|wget|base64|/dev/shm|/tmp/" /etc/profile /etc/bash.bashrc /etc/bashrc /etc/profile.d/* /root/.bashrc /root/.profile /home/*/.bashrc /home/*/.profile 2>/dev/null'Expect some legitimate hits (next section). Every hit you cannot explain goes in your notes with its path, owner and stat times.
Separate package files from additions
# Which package owns a file (Debian, Ubuntu)
dpkg -S <path>
# Which package owns a file (Rocky Linux, AlmaLinux)
rpm -qf <path>
# Verify every installed file against the package database (prints only differences)
sudo dpkg --verify
sudo rpm -Vadpkg --verify prints lines such as ??5?????? c /etc/cron.daily/apt-compat, and rpm -Va lines such as S.5....T. c /etc/crontab. In both, 5 means the content changed and c marks a configuration file. Configuration files that administrators edit show up and are usually fine; a changed file not marked c, such as a binary in /usr/bin or a unit in /usr/lib/systemd/system, is not. rpm -Va reads every installed file and takes a while. On some merged-/usr systems dpkg records files under /bin or /sbin, so try both paths with dpkg -S.
Both tools trust the package database on the server, which an attacker with root can also change, so a clean result is evidence, not proof.
Common false positives
- Package jobs and timers: apt-daily, apt-daily-upgrade and unattended-upgrades on Debian and Ubuntu; dnf-makecache on the RHEL family; logrotate, man-db, fstrim, e2scrub_all and systemd-tmpfiles-clean. All are package-owned and verify.
- Certificate renewal: certbot.timer or /etc/cron.d/certbot from the distribution package, or snap.certbot.renew.timer from the snap.
- Control panels: cPanel, Plesk and DirectAdmin add many root crontab and /etc/cron.d entries that call their own scripts under /usr/local/cpanel, /usr/local/psa or /opt/psa, and /usr/local/directadmin. Customer crontabs on panel servers routinely call curl or wget, for example to trigger wp-cron.php.
- Backup, monitoring and cloud agents: restic, borg and JetBackup jobs, exporters, qemu-guest-agent and similar. SecAI's own agent installs /etc/systemd/system/secai-agent.service with a drop-in, and /etc/logrotate.d/secai-agent.
- Docker and containerd units, and units that Ansible, cloud-init or other configuration management writes on every run.
A false positive is something you can trace to a package, a tool you installed or a person who can explain it. Anything you cannot trace stays on the list.
What to record, and when to rebuild
- For each finding: path, full text of the entry, owner and mode, mtime and ctime from stat, whether a package owns it, and what it runs, with the SHA-256 of any script or binary it calls.
- Do not delete or disable entries yet, and do not run the scripts they reference to see what they do.
- Find how each was written: the owning account, the change time, and the logins or web requests around it. Removing persistence without closing the entry point means it returns.
- If you find persistence that runs as root, an ld.so.preload entry, or several mechanisms for one payload, stop cleaning in place and plan a rebuild from a known-good image.
Incident response checklistInvestigate the process it startsCheck SSH keys too
What SecAI watches here, and what it does not
SecAI's file integrity monitoring reports changes to part of this list as alerts with the text before and after: cron entries in /etc/crontab, /etc/cron.d and per-user crontabs, and systemd services and timers in /etc/systemd/system. It also alerts when /etc/ld.so.preload appears or changes. It records what exists when the agent first starts rather than reporting it, so entries that predate installation are not alerted on.
The rest of this guide (the cron.daily-style directories, anacron, at, user units in home directories, shell startup files, update-motd.d, rc.local and udev rules) is not on its monitored path list, so the manual sweep is how you check those. None of these alerts removes anything.
File integrity monitoringThe monitored pathsFree server assessment