Why keys are the persistence that outlives everything
A key in authorized_keys lets its holder log in without a password, so rotating passwords changes nothing for them. The file records no owner, no date and no reason. Attackers who get any foothold often add one (MITRE ATT&CK T1098.004), because it keeps working after the original hole is patched.
Every command below only reads. Record what you find, off the server, before you remove anything.
Ask sshd where it looks for keys
Keys are not always in ~/.ssh/authorized_keys. AuthorizedKeysFile can point elsewhere, AuthorizedKeysCommand can fetch keys from a program, and Match blocks can change both per user. Read the effective configuration, not the file you assume is in use.
# Effective settings, after includes and defaults
sudo sshd -T | grep -iE "^(authorizedkeysfile|authorizedkeyscommand|authorizedprincipalsfile|trustedusercakeys|pubkeyauthentication|permitrootlogin) "
# Match blocks and key settings in the configuration files
sudo grep -rniE "^\s*(match|authorizedkeys|trustedusercakeys)" /etc/ssh/sshd_config /etc/ssh/sshd_config.d/authorizedkeysfile .ssh/authorized_keys .ssh/authorized_keys2 is the OpenSSH default, and both files count. Debian and Ubuntu keep that default; Rocky Linux and AlmaLinux usually name only .ssh/authorized_keys. Relative paths are from each user's home directory, and %u or %h tokens mean per-user files elsewhere, such as /etc/ssh/authorized_keys/%u.
If authorizedkeyscommand is anything but none, a program supplies keys at login, so a key can work without being in any file; find out what it queries. trustedusercakeys means any key signed by that certificate authority is accepted for the principals its certificate names, which is a far larger grant than one key.
List every key, for every account
Check every account, not just the people you know: root, service accounts such as www-data, git, deploy, postgres or redis, and accounts created by control panels. This loop reads each home directory from getent passwd and fingerprints every key in both standard files.
for user in $(getent passwd | cut -d: -f1); do
home=$(getent passwd "$user" | cut -d: -f6)
for f in "$home/.ssh/authorized_keys" "$home/.ssh/authorized_keys2"; do
if sudo test -f "$f"; then
echo "== $user $f"
sudo ssh-keygen -lf "$f"
fi
done
done
# Second pass: any authorized_keys file anywhere, including unexpected places
sudo find / \( -path /proc -o -path /sys -o -path /run \) -prune -o -name "authorized_keys*" -type f -print 2>/dev/nullssh-keygen -lf prints one line per key: size, SHA256 fingerprint, comment and type, for example 256 SHA256:Xk3... alice@laptop (ED25519). It handles lines that begin with options. Lines that are not valid keys print nothing, so also read each file with sudo cat.
Files that only find reports sit where sshd does not look unless AuthorizedKeysFile points there; note them anyway. Files under /var/lib/docker belong to container images. A key file under /var/www, /var/lib/redis or another service home deserves a close look.
Read the options and distrust the comment
An authorized_keys line can start with options that change what the key can do. They are easy to miss when you scan a file by eye.
- command="...": the key can only run that command. Backup tools, Git hosting and rsync jobs use this legitimately. An unfamiliar command, especially one that downloads or decodes something, is not legitimate.
- from="...": the key works only from the listed addresses or host names. An address you do not recognize tells you where the holder connects from.
- restrict, no-pty, no-port-forwarding: limits. A key with restrict and a forced command is typical of automation.
- cert-authority: the line is a certificate authority, not a key, and any key it signs is accepted for this account.
- environment="...": sets variables at login, only if PermitUserEnvironment is on. expiry-time="...": the key stops working after that date.
The comment at the end of the line (alice@laptop, deploy-2024) is free text chosen by whoever added the key, and sshd ignores it. An attacker can label a key with your colleague's name. Decide legitimacy from the fingerprint and the logins, never the comment.
Match fingerprints to real logins
sshd logs the fingerprint of the key behind every successful key login. That line ties a key in a file to actual use: which account, from where, and when.
# Ubuntu, and Debian with rsyslog: current and rotated logs
sudo zgrep -h "Accepted publickey" /var/log/auth.log* 2>/dev/null
# Rocky Linux and AlmaLinux
sudo grep -h "Accepted publickey" /var/log/secure* 2>/dev/null
# Debian 12 and later without rsyslog, or any systemd server: the journal
sudo journalctl --no-pager | grep "Accepted publickey"A line reads: Accepted publickey for deploy from 203.0.113.10 port 51122 ssh2: ED25519 SHA256:Xk3.... The SHA256 value is the same string ssh-keygen -lf prints, so every key can be matched to every login. On OpenSSH 9.8 and later these lines come from sshd-session rather than sshd, which the journal search above still finds. The journal only goes back as far as its size limits allow.
Build a table of fingerprint, account, source addresses, and first and last use. Look for unfamiliar source addresses, logins to root or service accounts, logins at hours nobody works, and keys that never appear in the logs (unused, newly added, or used while logs were missing). Root can edit logs, so a missing login proves nothing. The auth log analyzer can group successful logins by account, source, method and fingerprint for you.
Check when each file changed, and how far to trust it
# Full timestamps for one key file (repeat for each file found above)
sudo stat /root/.ssh/authorized_keys
# Compact: modification time, change time, owner, mode
sudo stat -c "mtime=%y ctime=%z owner=%U:%G mode=%a %n" /root/.ssh/authorized_keysmtime is when the content last changed; ctime is when content or metadata last changed. touch can set mtime to any date, but that sets ctime to the moment it ran, so an old mtime with a recent ctime suggests a faked date. Root can defeat both, so use them to order events, not to prove them.
Note the owner too. sshd rejects key files other users can write (StrictModes, on by default), so a working key file is owned by its account or root. A .ssh directory owned by a service account, such as www-data in /var/www/.ssh, tells you which account wrote it.
Keys on accounts that should never log in
Service accounts normally have /usr/sbin/nologin or /bin/false as their shell and no keys. A key on one is a finding even though it cannot open a shell: nologin only stops the interactive session. The key still authenticates, and unless forwarding is disabled for that key or account, it can open tunnels through your server into your private network.
# Accounts with a real login shell (only these should normally hold keys)
getent passwd | awk -F: '$7 !~ /(nologin|false)$/ {print $1, $3, $6, $7}'Compare this list with the people and automation that should have access. While you are in each .ssh directory (sudo ls -la), also look for an rc file: sshd runs ~/.ssh/rc, and /etc/ssh/sshrc for everyone, at every login, which makes them quiet persistence points.
How the key got there
Removing a key without knowing how it arrived invites the same key back. The usual write paths:
- A web application bug lets the attacker run commands as www-data, apache or nginx, which can write to that account's own home (/var/www on Debian and Ubuntu). Check the web access log around the file's change time.
- An exposed service running as its own user, such as Redis without authentication, has been abused to write a key into that account's home, or into root's if it runs as root.
- A guessed or stolen password: the attacker logs in once and adds a key so the password no longer matters. Look for a password login shortly before the change time.
- An existing person or tool: a former contractor, a provisioning script, cloud-init at first boot. These are often legitimate, and must still be explained.
Common false positives
- Keys added at provisioning by cloud-init or your provider. Check the provider console for the key pair the server was created with.
- Panel-managed keys: cPanel and Plesk let site owners add keys through the panel, so keys on those accounts may be a customer's.
- Automation: backup and deploy keys with command= or from= restrictions, and keys pushed by Ansible, CI or other configuration management. A key that reappears after removal may come from your own tooling.
Removing keys without locking yourself out
Change nothing until the files, fingerprints and log lines are recorded off the server. Then:
- Open a second root session and keep it open until a fresh login works; a typo or a permission change can lock out every key on the account.
- Remove only the keys you have identified, from every file sshd reads for that account, and check any AuthorizedKeysCommand source.
- Removing a key does not end sessions already open with it. Check who and ss for live SSH sessions from the addresses you found, and cut them at the provider firewall if needed.
- Rotate any key whose private half was stored on this server, since root could read it.
- If the key was on root, or you cannot explain how it was added, treat the server as compromised; that usually ends in a rebuild.
What SecAI reports about SSH keys
SecAI reports a new or changed authorized_keys entry as an alert carrying the key's fingerprint and comment. It separately alerts when a privilege change, such as a new account, sudo grant or key, appears with no interactive administrator session behind it, which is what a key written by a compromised process looks like. It also records interactive SSH sessions with user, source, method, key fingerprint, and open and close times.
These are alerts: SecAI proposes no action for them and does not delete files, so removing a key stays your decision. Keys already present when the agent was installed form its baseline and are not reported as new, so this one-time audit is still worth running. To review the rest of your sshd settings, paste sudo sshd -T output into the SSH config checker, which runs in your browser.