Any Linux server with port 22 open to the internet is being probed continuously. Most of it is automated and untargeted. Some of it is not. This is the complete sequence: work out what you are looking at, remove the thing being attacked, add the layers that deal with the noise, and check that every one of them is doing what you think.
Key Takeaways
- Two patterns matter and they need different responses: concentrated attempts from one source, and distributed attempts spread thin across many.
- Blocking is a response. Disabling password authentication removes the problem, because there is nothing left to guess.
- Per-address thresholds catch the concentrated pattern and miss the distributed one, because no single source crosses the limit. That is true of fail2ban, of a firewall rate limit, and of SecAI.
- Every layer fails silently. A jail that matches nothing and a quiet week look identical, so verify each layer instead of assuming it.
- Blocked does not mean safe. After any heavy attempt, confirm that nothing succeeded and check for persistence.
- Any automatic blocking needs an allowlist for your own addresses, or you will eventually lock yourself out during an incident.
Table of Contents
- Diagnosis: what it looks like
- Key-based authentication: remove the attack surface
- OpenSSH configuration
- Firewalling and the provider firewall
- Rate limiting
- fail2ban
- CrowdSec
- Logging
- Verification: is each layer working
- What to check after a heavy attempt
- Ongoing monitoring
- How SecAI handles SSH attacks
Diagnosis: what it looks like
First find the log. Where SSH writes depends on the distribution:
| Distribution | Where SSH authentication events are |
|---|---|
| Ubuntu | /var/log/auth.log, and the journal (journalctl -u ssh) |
| Debian 12 and later | The journal only (journalctl -u ssh), unless you installed rsyslog |
| Rocky Linux, AlmaLinux | /var/log/secure, and the journal (journalctl -u sshd) |
Then look at the failures. On a server with a log file:
sudo grep "Failed password" /var/log/auth.log | tail -50 # Ubuntu
sudo grep "Failed password" /var/log/secure | tail -50 # Rocky Linux, AlmaLinux
On a journal-only server, the same lines come from the journal:
sudo journalctl -u ssh --since "24 hours ago" | grep "Failed password" | tail -50
Count attempts by source. The pipeline is the same whichever command feeds it:
sudo grep "Failed password" /var/log/auth.log \
| awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -20
Two patterns matter. Concentrated: one address, hundreds of attempts, usually against root or admin, noisy and easy to block. Distributed: many addresses, a handful of attempts each, often against a valid username. That one is quieter, more deliberate, and slips past every per-address threshold.
Counting by username instead of by source
The distributed case becomes visible when you pivot the same data:
sudo grep "Failed password" /var/log/auth.log \
| grep -oP "for (invalid user )?\K\S+" | sort | uniq -c | sort -rn | head -20
Untargeted botnets try root, admin, test, ubuntu, oracle, a dictionary of defaults. If the top entry is a real username that exists on your system, particularly one that is not obvious from the outside, someone has done reconnaissance. That changes the situation from background noise to something worth attention.
Reading the volume honestly
A public server seeing thousands of failed attempts per day is normal and not itself an incident. What matters is the shape: a sudden change in volume, a shift to a valid username, attempts arriving from a range that also appears in your web logs, or attempts that stop abruptly, which occasionally means one succeeded.
Key-based authentication: remove the attack surface
Blocking is a response. Prevention is better: disable password authentication and the brute force problem disappears, because there is no password to guess. Attempts still appear in logs and none can succeed.
Do it in this order, because it is the one change here that can lock you out:
- Generate a key on your own computer and install it:
ssh-keygen -t ed25519, thenssh-copy-id you@server. - Prove that the key works in a new session, with the password fallback disabled:
ssh -o PasswordAuthentication=no you@server. - Only then turn passwords off, keeping your existing session open while you do.
The commands for step 3, the verify step and the rollback are in the hardening guide under turn off SSH password login, and the rest of the SSH lockdown is in the SSH hardening checklist.
Worth being explicit about what this does and does not solve. It ends guessing as an attack path completely. It does nothing about a key that has been stolen from a developer laptop, or a deploy credential leaked in a repository. Those authenticate correctly and produce a normal Accepted publickey line, which is why the later sections still matter after you have made this change.
OpenSSH configuration
OpenSSH has its own limits, and they apply before any other tool sees the connection. Read what your server is really using, after every included file:
sudo sshd -T | grep -i -E '^(passwordauthentication|kbdinteractiveauthentication|permitrootlogin|maxauthtries|logingracetime|maxstartups|persourcepenalties)'
The ones that matter against brute force:
PasswordAuthentication noandKbdInteractiveAuthentication no. With both off, only keys are accepted.PermitRootLogin prohibit-passwordorno. Root is the account every botnet tries first.MaxAuthTriesis how many authentication attempts one connection gets. The default is 6. Lowering it to 3 is common, with one caveat: every key your SSH agent offers counts as an attempt, so an administrator with several keys loaded can be refused before the right one is tried. Naming the key with-iand settingIdentitiesOnly yeson the client avoids that.LoginGraceTimeis how long an unauthenticated connection may stay open. The default is 120 seconds; 30 is plenty.MaxStartupslimits concurrent unauthenticated connections, and starts refusing a share of new ones as the number grows. The default is10:30:100. It protects the daemon under load, and note what it cannot do: it does not distinguish you from the attacker.PerSourcePenalties, in OpenSSH 9.8 and later, makes sshd itself refuse connections for a while from an address that keeps failing authentication. It is on by default where it exists, and it counts by address, so people who share one office or VPN address share its penalties. Check your version withssh -V. Debian 13 (OpenSSH 10.0), Ubuntu 26.04 (10.2), and Rocky Linux and AlmaLinux 10 and 9.8 or later (9.9) have it; Debian 12 (9.2), Ubuntu 22.04 (8.9) and Ubuntu 24.04 (9.6) do not.
Put your changes in a file under /etc/ssh/sshd_config.d/ whose name sorts first, because sshd uses the first value it reads for each setting and cloud images often ship a drop-in of their own. Run sudo sshd -t before every reload; it prints nothing when the configuration is valid. If the setting you edited does not show up in sshd -T, paste the output into the free SSH config checker: it runs entirely in your browser and is the quickest way to find the file that is overriding you.
Does moving SSH off port 22 help? It removes most automated noise, which makes your logs readable. A targeted attacker will find the new port in a scan. Treat it as noise reduction rather than a control, and on Rocky Linux and AlmaLinux remember that SELinux has to be told about the new port first.
Firewalling and the provider firewall
The most effective firewall rule against SSH brute force is the one that never lets the attacker reach SSH. If the people who administer the server have fixed addresses, or come in through a VPN, allow SSH only from those:
sudo ufw allow from 203.0.113.10 to any port 22 proto tcp # Debian, Ubuntu
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.10" port port="22" protocol="tcp" accept'
sudo firewall-cmd --permanent --remove-service=ssh
sudo firewall-cmd --reload # Rocky Linux, AlmaLinux
Add your own rule first, confirm you can still connect in a new session, and only then remove the general one. Better still, do the same thing in the provider's firewall or security group: it sits outside the server, so it still works if the host firewall is flushed by a reload, and a mistake in it can be fixed from the provider's panel without needing SSH at all.
If your administrators do not have fixed addresses, leave SSH open and rely on the other layers. A restriction that people work around by opening SSH to the world "for a moment" is worse than none.
Rate limiting
On Debian and Ubuntu, ufw has a rate limit built in:
sudo ufw limit 22/tcp
It refuses connections from an address that has opened six or more in the last thirty seconds. That is a blunt instrument: it slows a concentrated attack, does nothing about a distributed one, and can briefly catch an administrator who opens many sessions quickly, for example with automation tooling. It is a reasonable default on a server you log in to by hand.
Avoid global connection limits on the SSH port, the kind that accept only so many new connections per minute from everybody combined. During an attack the attacker uses the whole allowance, and the person locked out is you.
fail2ban
fail2ban reads the authentication log, counts failures per address, and adds a firewall rule for addresses that cross a threshold. It handles the concentrated case well. A reasonable SSH jail, in /etc/fail2ban/jail.local:
[sshd]
enabled = true
maxretry = 4
findtime = 10m
bantime = 1h
ignoreip = 127.0.0.1/8 ::1 203.0.113.10
Put your own addresses in ignoreip before you need them. On a journal-only server the jail has to read the journal. On Debian 12 that means adding backend = systemd to the jail and installing python3-systemd; without them fail2ban looks for /var/log/auth.log, does not find it, and the SSH jail fails to start. The fail2ban package in Debian 13 sets the systemd backend by default. The longer explanation is in why fail2ban reports active and bans nothing.
For distributed attempts, per-address thresholds do not trigger at all, because no single address reaches maxretry. If you are weighing fail2ban against a shared-reputation approach that can ban an address the first time it shows up, I compared the two in fail2ban vs CrowdSec. The wider set of decisions around ban scope and duration is in how automated IP blocking works and how it goes wrong.
CrowdSec
CrowdSec separates detection from enforcement. The agent reads your logs and decides; a separate "bouncer" applies the decision in the firewall. It also subscribes to a community blocklist, so an address that attacked other servers can be refused on yours before it has done anything. That is the one mechanism in this article that helps against the distributed pattern, because it does not depend on the address misbehaving on your server first.
Install it from CrowdSec's own instructions, which set up their package repository for your distribution, and install a firewall bouncer with it: crowdsec-firewall-bouncer-nftables or crowdsec-firewall-bouncer-iptables, whichever matches the server. A CrowdSec agent with no bouncer detects and blocks nothing, which is the most common mistake.
Logging
Every layer above depends on the authentication log being there and being kept:
- Make the journal persistent, so that a reboot does not erase the evidence. The hardening guide has the three commands.
- Keep enough history to investigate something you notice a month late.
- Send a copy off the server. Logs on the machine are only as trustworthy as the machine, and an intruder with root can edit them.
Verification: is each layer working
A filter that has silently stopped working looks identical to a quiet week. Check each layer you enabled:
sudo sshd -T | grep -i -E '^(passwordauthentication|permitrootlogin)' # what sshd really uses
sudo ufw status verbose # or: sudo firewall-cmd --list-all
sudo fail2ban-client status sshd # currently failed, currently banned
sudo cscli metrics # CrowdSec: are log lines being read and parsed?
sudo cscli bouncers list # CrowdSec: is anything enforcing the decisions?
sudo cscli decisions list # CrowdSec: what is blocked right now
Zero bans on an internet-facing server usually means the filter no longer matches your log, not that nobody is trying. And test from outside: from another machine, ssh -o PubkeyAuthentication=no you@server must end in "Permission denied" without ever asking for a password.
What to check after a heavy attempt
Blocked does not mean safe. Verify nothing got through:
sudo grep "Accepted" /var/log/auth.log | tail -20 # or /var/log/secure
sudo journalctl -u ssh --since "7 days ago" | grep "Accepted" | tail -20 # journal-only servers
Every line should be a person and a source you recognise. Then check for persistence: new entries in ~/.ssh/authorized_keys, new cron jobs, new systemd services, changed sshd_config. If an attacker did get in, they often leave a webshell in a web-writable directory as a second way back - worth checking for at the same time.
Concretely, the fastest sweep:
for d in /root /home/*; do
[ -f "$d/.ssh/authorized_keys" ] && echo "== $d" && cat "$d/.ssh/authorized_keys"
done
awk -F: '$3 == 0 {print}' /etc/passwd
ls -la /etc/sudoers.d/
find /etc/systemd/system -newermt "-7 days" -type f 2>/dev/null
An added key is the one that matters most, because it survives the password rotation most people perform after an incident. The full treatment is in detecting privilege escalation on a Linux server, and if any of this comes back unexpected, how to tell if your Linux server has been compromised is the wider triage.
Ongoing monitoring
Everything above is configuration, and configuration drifts. A drop-in file re-enables passwords, a jail stops matching after an upgrade, somebody opens SSH to the world during an incident and never closes it. What needs watching continuously is short:
- the effective SSH configuration, not the file you last edited;
- successful logins, especially one that follows failures or comes from a source never seen before;
- new authorized keys, accounts and sudo rules;
- whether the blocking layer is still blocking.
Doing that by hand across more than a couple of servers does not survive a busy month, which is the argument in why manual server monitoring fails at 3am.
How SecAI handles SSH attacks
This is what SecAI does with the same signals, stated as the product actually behaves.
Detection. The agent reads SSH authentication events from /var/log/auth.log or /var/log/secure, or from sshd's journal where there is no log file, and reports them with their source. The platform counts sightings per address.
Blocking. An address seen attacking three times within thirty minutes becomes a block. On a server set to Automatic it is applied at the agent's next check-in, usually within a minute or two; on a server set to approval it waits for you. The block goes through the tool the server already runs, fail2ban first, then CrowdSec, CSF, firewalld or iptables, and it is scoped: an SSH attacker is dropped on the SSH port, not cut off from everything. An automatic block expires after 6 hours the first time, and each further block of the same address on the same server within 30 days lasts twice as long, up to 7 days. Where fail2ban, CrowdSec, firewalld or CSF carries the block, that tool may end it sooner on its own timer.
Never locking you out. Your own address and any address you mark as trusted are never blocked, and the "Let me in" link in an alert lifts a block that caught you without needing approval.
What it does not do. Blocking is per address and per server. A distributed attack that stays under the threshold on every address is not blocked by SecAI any more than by fail2ban; it is shown as one incident with all its sources, so you can see the campaign and decide, and the prevention for it is still key-only login. Blocks are not copied between your servers or shared between customers.
After the login. The part brute-force tooling cannot see is what SecAI watches most closely: a successful login that follows failures or comes from a new source, which registered administrator key opened a session, a root login, a new authorized key, and a privilege change that no administrator session explains.
Before you enable anything that blocks automatically, confirm two things: that your own ranges are allowlisted, and that you know how to reach your provider's web console if it goes wrong anyway. Where this fits among the other controls worth running is covered in the guide to Linux server security software and in server-level monitoring for a VPS.
Frequently Asked Questions
Is it normal to see thousands of failed SSH attempts per day? On a public server, yes. It is background noise from untargeted scanning. What matters is a change in the pattern rather than the raw count.
What is the single most effective fix?
Key-only authentication. With PasswordAuthentication no, guessing cannot succeed however many attempts arrive. Everything else in this article is about noise, load and the cases keys do not cover.
Does moving SSH off port 22 stop brute force? It removes most automated noise, which makes your logs readable. A targeted attacker will find the new port in a scan. Treat it as noise reduction rather than a control.
Should I ban permanently after repeated attempts? Escalating temporary bans work better. Addresses get reassigned, and a permanent ruleset grows into something nobody dares to prune.
How do I catch distributed attempts that never cross a threshold? You mostly cannot block them with a per-address threshold, which is why key-only login matters. To see them, count failures by targeted username instead of by source, and consider a shared blocklist such as CrowdSec's, which can refuse an address before it has done anything to you.
Why does fail2ban show zero bans?
Usually because it is not reading the log SSH writes to. Since Debian 12 there is no /var/log/auth.log by default; on Debian 12 the sshd jail needs backend = systemd and the python3-systemd package, and Debian 13's package already sets that backend.
An attempt succeeded. What now? Treat the server as compromised. Check for added keys, new users, sudoers changes, and scheduled tasks, and work through the triage in how to tell if your Linux server has been compromised before deciding whether to clean or rebuild.
Sources
- OpenSSH 9.8 release notes: PerSourcePenalties, on by default
- sshd_config(5): first value wins, MaxAuthTries, LoginGraceTime, MaxStartups
- Debian 12 release notes: rsyslog no longer installed by default
- Debian bug 1037437: fail2ban's SSH jail and the missing auth.log
- Red Hat Enterprise Linux 9.8 release notes: OpenSSH 9.9
- ufw manual: the limit rule
- CrowdSec documentation
- MITRE ATT&CK T1110: Brute Force