How to Detect Bad Bots in Nginx Logs

Bots announce a lot about themselves, mostly in the user-agent field and in their behavior over time. The user agent lives at the end of each combined-format line, so if your logs are short a couple of fields, check the log format guide first. You need the user agent to do most of this.

Good bots, bad bots, and the gray area

Not every bot is a problem. Search engine crawlers bring you traffic. SEO tools and uptime monitors are doing a job you may want done. The ones to worry about fall into three buckets: content scrapers that copy your pages, vulnerability scanners hunting for exploits, and crawlers that hit you so hard they cost you money in bandwidth and CPU.

Signal 1: the user-agent string

The laziest bots do not bother faking a browser. These user agents are almost always automated traffic:

• python-requests/2.31.0 and python-urllib, scripts built with the standard Python libraries
• curl/8.4.0 and Wget/1.21, command-line fetchers
• Go-http-client/1.1, the default for anything written in Go that did not set a UA
• Java/1.8.0, okhttp, axios, and other raw HTTP library strings
• An empty user agent, logged as "-", which real browsers never send
• Scanner names like masscan, zgrab, Nmap, or nikto

Then there are the named crawlers. SEO tools like AhrefsBot, SemrushBot, MJ12bot, and DotBot identify themselves honestly, which is good, but they can crawl aggressively. If they are eating your bandwidth and you get nothing from them, robots.txt or an nginx block is fair.

A quick way to see who is visiting, sorted by volume:

awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -30

That pulls the user-agent field (the sixth quote-delimited chunk in combined format) and ranks it. The names at the top that are not browsers are your candidates.

Signal 2: request rate and pattern

Behavior gives away bots that fake a browser UA. A person clicks a few pages a minute and loads the CSS, JS, and images that go with each one. A scraper hammers hundreds of unique URLs in order and never touches a stylesheet.

Rank the noisiest IPs first:

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20

Then look at what the top IP actually did. Tells that point at automation:

• Dozens of requests in the same second, faster than a human can click
• Walking through sequential URLs or an ID range, like /product/1, /product/2, and so on
• Only HTML, no static assets, because the scraper does not care how the page looks
• No referer on internal pages a human would have clicked through to
• Requests for paths that do not exist on your site: /wp-login.php, /.env, /phpmyadmin, /.git/config

That last one is the clearest signal of all. If you do not run WordPress and something is requesting /wp-admin forty times, it is a scanner working through a list. Those requests pair with 404s, so cross-referencing with your status codes narrows them fast.

Signal 3: fake Googlebot

This one catches people out. A request can claim to be Googlebot in its user agent and be nothing of the sort. Spammers fake it because some sites let Googlebot through checks they apply to everyone else. The user-agent string is not proof of anything.

Google publishes the way to verify: do a reverse DNS lookup on the IP and confirm it resolves to a googlebot.com or google.com hostname, then do a forward lookup on that hostname and check it points back to the same IP. If either step fails, the UA is lying.

# reverse lookup on the IP from the log
host 66.249.66.1
# should return something ending in .googlebot.com

# then confirm the hostname resolves back to that IP
host crawl-66-249-66-1.googlebot.com

Bingbot uses the same reverse-then-forward DNS method with search.msn.comhostnames. Any crawler worth trusting documents a way to verify it. If it does not, treat the name as decoration.

Signal 4: the browser string that is too clean or too old

Some bots set a real-looking UA but get the details wrong. A user agent claiming Chrome 49 on Windows XP in 2026 is not a real person, that combination has not existed in years. Identical user-agent strings arriving from dozens of scattered IPs at the same rate is another giveaway, because real browsers vary in tiny ways and real users do not move in lockstep.

Turning signals into a verdict

No single signal is proof. A high request rate could be a busy real user on a shared office IP. A curl UA could be your own health check. The call gets reliable when the signals stack: a library user agent, hitting nonexistent admin paths, at machine speed, from an IP that fails reverse DNS. That is a bot, and an unfriendly one.

Doing this by hand across a large log is slow. The analyzer does the grouping, rate analysis, and known-bot matching for you in one pass, and flags the suspicious patterns so you can see them instead of grepping for them.