How to Read Dovecot Login Failed Logs & Detect Email Brute

Introduction

One afternoon, I was monitoring the email queue on a production server when two log lines caught my eye — they looked trivial at first glance, but once I looked closer, they were enough to raise alarm bells:

2026-07-20 16:38:05 dovecot_login authenticator failed for H=(aGrT3z2j) [10.0.0.1]:61890: 535 Incorrect authentication data (set_id=info@client-a.com)
2026-07-20 16:38:06 dovecot_login authenticator failed for H=(jAvmHBGX) [10.0.0.2]:55446: 535 Incorrect authentication data (set_id=info@client-b.com)

Two failed email login attempts within 1 second, from two different IPs, targeting two different email addresses. But there was one suspicious pattern: the HELO strings were random garbage — aGrT3z2j and jAvmHBGX. Humans usually send something recognizable like a domain name (mail.client.com), not random strings like these.

That was the moment I realized our email server was being brute-forced — and these log lines were just the tip of the iceberg.

Symptoms — What Was Happening?

These logs appeared in /var/log/exim/mainlog (or /var/log/maillog on some distros). Heres what they said:

2026-07-20 16:38:05 dovecot_login authenticator failed for H=(aGrT3z2j) [10.0.0.1]:61890: 535 Incorrect authentication data (set_id=info@client-a.com)
2026-07-20 16:38:06 dovecot_login authenticator failed for H=(jAvmHBGX) [10.0.0.2]:55446: 535 Incorrect authentication data (set_id=info@client-b.com)

The symptoms:

• Failed email login attempts (SMTP authentication)
• Error code 535 — “Incorrect authentication data” (wrong password)
• Random HELO strings — a strong indicator of bots
• Two different IPs within 1 second — signs of a distributed attack
• Targeted emails: info@client-a.com and info@client-b.com — generic addresses that are common brute force targets

Root Cause — What Went Wrong?

Nothing was “wrong” on the server side — the server was doing its job correctly. Authentication failed because the password was wrong. But the problem is the brute force attack in progress.

Why Is This Brute Force and Not Just a Forgotten Password?

Several signs give it away:

1. Random HELO strings — aGrT3z2j and jAvmHBGX are random. Humans logging in via Outlook/Thunderbird typically send a HELO like mail.client.com or a computer name (DESKTOP-ABC123). Random strings are a hallmark of bots/scripts
2. Two different IPs in 1 second — way too fast for the same user. This is a distributed attack
3. Targeted emails: info@ — generic email addresses like info@, admin@, support@ are prime brute force targets because they often have weak passwords
4. Error code 535 — “Incorrect authentication data” means the password attempted is wrong. Bots try thousands of password combinations

How to Read This Log in Detail

Lets break down every part of this log line by line:

2026-07-20 16:38:05 dovecot_login authenticator failed for H=(aGrT3z2j) [10.0.0.1]:61890: 535 Incorrect authentication data (set_id=info@client-a.com)

Part	Value	Meaning
2026-07-20 16:38:05	Timestamp	When it happened (July 20, 2026, 4:38:05 PM)
dovecot_login	Authenticator	Authentication type used — Dovecot SASL (Simple Authentication and Security Layer). This means Exim uses Dovecot to verify username/password
authenticator failed	Status	Authentication failed — password didnt match
H=(aGrT3z2j)	HELO hostname	The hostname sent by the client during SMTP handshake. Random string = bot. Humans use recognizable domain/computer names
[10.0.0.1]	Source IP	The IP address trying to log in. This private IP suggests the attack is coming from inside the network or via VPN/tunnel
:61890	Source port	Port used by the client. High ports (>1024) are normal for clients
535	SMTP error code	SMTP error code. 535 = “Authentication credentials invalid” (wrong password or non-existent user)
Incorrect authentication data	Error message	Error message: authentication data is incorrect (wrong password)
set_id=info@client-a.com	Target email	The email address being attempted. info@ is a common brute force target

Why Doesnt the Server Block Automatically?

Exim/Dovecot dont block IPs after failed attempts by default. They just log it and keep going. Without additional configuration (Fail2Ban, rate limiting, or firewall rules), brute force can go on for days without being detected — unless youre diligent about reading logs.

Step-by-Step Solution — Securing the Email Server

Step 1: Check the Brute Force Activity

First, we need to know how bad the attack is. Run the following command to see failed login attempts over the last hour:

# Count failed login attempts per IP in the last hour
grep "dovecot_login authenticator failed" /var/log/exim/mainlog | 
  grep "$(date -d 1 hour ago +%d/%b/%Y:%H:)" | 
  grep -oE [[0-9]+.[0-9]+.[0-9]+.[0-9]+] | 
  sort | uniq -c | sort -rn | head -n 10

Example output:

  15423 10.0.0.1
  12847 10.0.0.2
   9834 10.0.0.3
   8721 10.0.0.4
   7654 10.0.0.5

If the numbers are in the thousands per IP, its definitely brute force. Now check which emails are being targeted most:

# Check most-targeted email addresses
grep "dovecot_login authenticator failed" /var/log/exim/mainlog | 
  grep "$(date -d 1 hour ago +%d/%b/%Y:%H:)" | 
  grep -oE set_id=[^)]+ | 
  sort | uniq -c | sort -rn | head -n 10

Example output:

  8432 set_id=info@client-a.com
  6721 set_id=admin@client-a.com
  5432 set_id=support@client-a.com
  4321 set_id=info@client-b.com
  3210 set_id=admin@client-b.com

info@, admin@, support@ emails are always primary targets because these addresses are widely known and often have default/weak passwords.

Step 2: Install & Configure Fail2Ban

Fail2Ban is the most effective solution for blocking brute force IPs. Install it if not already present:

# Install Fail2Ban (CentOS/RHEL)
yum install -y fail2ban

# Install Fail2Ban (Ubuntu/Debian)
apt install -y fail2ban

Create a custom config for email brute force at /etc/fail2ban/jail.local:

[DEFAULT]
bantime = 3600
findtime = 600
maxretry = 5

[dovecot]
enabled = true
port = smtp,465,submission,imap,pop3,pop3s
filter = dovecot
logpath = /var/log/exim/mainlog
maxretry = 3
bantime = 86400

[exim]
enabled = true
port = smtp,465,submission
filter = exim
logpath = /var/log/exim/mainlog
maxretry = 5
bantime = 3600

Config explanation:

• maxretry = 3 — after 3 failed attempts, the IP gets blocked
• bantime = 86400 — blocked for 24 hours (86,400 seconds)
• findtime = 600 — 10-minute window to count attempts

Start Fail2Ban:

# Start and enable Fail2Ban
systemctl enable --now fail2ban

# Check status
fail2ban-client status dovecot
fail2ban-client status exim

Step 3: Rate Limiting in Dovecot

Besides Fail2Ban, you can add rate limiting directly in Dovecot. Edit /etc/dovecot/conf.d/10-auth.conf:

# Add at the end of the file
auth_mechanisms = plain login

# Rate limiting — max 5 attempts per minute per IP
service auth {
  process_min_avail = 1
  process_limit = 10
}

# Disable plaintext auth without SSL (additional security)
disable_plaintext_auth = yes

Restart Dovecot:

systemctl restart dovecot

Step 4: Block Brute Force IPs via Firewall

If you already know which IPs are brute forcing, you can block them directly via firewall:

# Check which IPs are brute forcing the most
grep "dovecot_login authenticator failed" /var/log/exim/mainlog | 
  grep -oE [[0-9]+.[0-9]+.[0-9]+.[0-9]+] | 
  sort | uniq -c | sort -rn | head -n 5

# Block IPs with iptables (replace with actual IPs)
iptables -I INPUT -s 10.0.0.1 -j DROP
iptables -I INPUT -s 10.0.0.2 -j DROP

# Save rules to persist after reboot
service iptables save

Step 5: Verify

After applying the configuration, check if the brute force is still happening:

# Monitor logs in real time
tail -f /var/log/exim/mainlog | grep "dovecot_login authenticator failed"

If Fail2Ban is active, brute force IPs should be blocked and the log should show [dovecot] Ban 10.0.0.x.

Step 6: Create a Brute Force Monitoring Script

Create a simple script to alert when brute force is detected:

#!/bin/bash
# /usr/local/bin/check-email-brute-force.sh

LOG="/var/log/exim/mainlog"
THRESHOLD=100
HOURS=1

COUNT=$(grep "dovecot_login authenticator failed" "$LOG" | 
  grep "$(date -d "${HOURS} hour ago" +%d/%b/%Y:%H:)" | wc -l)

if [ "$COUNT" -gt "$THRESHOLD" ]; then
    echo "WARNING: $COUNT failed email logins in last $HOURS hour(s)" | 
      mail -s "Email Brute Force Alert" admin@your-server.com
fi

Run this script via cron every 30 minutes:

# Edit crontab
crontab -e

# Add this line
*/30 * * * * /usr/local/bin/check-email-brute-force.sh

Pro Tips & Warnings

• Use strong passwords for email accounts — info@ emails often have default passwords from the control panel. Change them immediately to passwords at least 16 characters long
• Disable unused email accounts — if theres an info@ email thats not being used, disable or delete it. No email is better than an email with a weak password
• Use SPF, DKIM, and DMARC — these protect your domain from spoofing, not brute force. But theyre still important for overall email security
• Monitor logs regularly — dont wait for an alert. Check logs at least once a day
• Use RFC 5737 IPs for testing — if you want to test whether Fail2Ban is working, use IPs from the RFC 5737 range (192.0.2.x, 198.51.100.x, 203.0.113.x) so you dont block production IPs
• Back up config before editing — always backup /etc/dovecot and /etc/exim.conf before making any changes

FAQ

Q: Does a 535 error always mean brute force?

Not always. A 535 error can occur because a user forgot their password, theres a typo in the email address, or the client is misconfigured. But if youre seeing thousands in an hour from many different IPs with random HELO strings — its definitely brute force. Always check the context before drawing conclusions.

Q: How do you tell the difference between brute force and a forgotten password?

Look at the pattern: (1) A forgotten password usually comes from 1-2 IPs with the same address, (2) the HELO string is clear (computer name/domain), (3) the timing of attempts is irregular. Brute force: many different IPs, random HELO, uniform attempts in a short timeframe.

Q: Can Fail2Ban block the wrong IP (false positive)?

Yes, if maxretry is set too low. For example, a user who genuinely forgot their password and tries 5 times could get blocked. To avoid this, set maxretry = 5 or higher, and use a sufficiently large findtime (e.g. 600 seconds = 10 minutes). Also, make sure internal IPs (e.g. office IP) arent blocked — add them to ignoreip in the Fail2Ban config.

Q: Why are random HELO strings an indicator of bots?

Because normal email clients (Outlook, Thunderbird, Apple Mail) automatically send a HELO that matches the configured computer or domain name. Brute force bots/scripts usually send random HELO strings because they dont care about HELO — all they care about is trying as many passwords as possible. Random HELO strings like aGrT3z2j or jAvmHBGX would never be generated by a normal email client.

Conclusion

The dovecot_login authenticator failed log is one of the first signs that your email server is being brute-forced. By reading logs in detail — understanding what every part means, from timestamp and HELO string to source IP, error code, and target email — you can spot an attack early and take preventive action.

The most effective solution is a combination of three layers: Fail2Ban to automatically block IPs, rate limiting in Dovecot to restrict attempts per IP, and strong passwords for all email accounts — especially generic addresses like info@ and admin@. These three layers work together to make your email server much harder to brute force.

The biggest lesson from this incident: never dismiss “small” logs. Two lines of log that look trivial can be the beginning of a massive attack threatening your entire email infrastructure.

Author: NOC Engineer — written based on experience detecting and mitigating email brute force on production servers.