Bot Traffic Guide: What It Is, Risks, and Safe Uses

Bot traffic is any request to your site made by software rather than a person at a screen. That single definition covers things as different as Googlebot indexing a product page, an uptime monitor pinging your homepage every sixty seconds, a scraper lifting your prices, and a credential-stuffing script working through a leaked password list. Lumping them together is the mistake. This guide is about telling them apart, because the right response to each one is completely different.

Key takeaways

• Bot traffic is not a single phenomenon. It splits into useful automation, hostile automation, and synthetic traffic you generated yourself, and each tends to have a different owner inside an organisation.
• Google Analytics 4 automatically excludes traffic from known bots and spiders using the IAB/MRC International Spiders and Bots List. That filter is real, but it works from a known-bot list, which is precisely the wrong tool for catching automation that is trying not to be known.
• User-agent strings and robots.txt are declarations, not controls. Both work fine on well-behaved bots and mean nothing to the rest.
• Identity should be verified at the network level (reverse DNS or published IP ranges), not taken from a header any client can set.
• Once automation targets logins, checkout, forms or pricing, it has stopped being a reporting nuisance and become a security matter with a different escalation path.

What actually counts as bot traffic?

The useful boundary is the operator's intent, not the technology. A headless browser is not inherently malicious; your own QA suite probably runs one. Three questions settle most cases: who is running this, what are they taking or doing, and did they announce themselves? A crawler that identifies itself and respects your crawl directives is a guest with a business card. A set of residential IPs cycling through your product URLs at three requests per second behind a stock Chrome user-agent is something else, even though both technically just fetch pages.

The three categories worth separating

Useful automation

These bots exist because you benefit from being visible or observable. Search engine crawlers such as Googlebot and Bingbot build the index your organic traffic depends on. Uptime monitors fetch pages on a schedule so you learn about an outage before your customers do. Price comparison and affiliate crawlers may be part of a distribution channel you actively want. Link preview fetchers render the card someone sees when your URL is shared in a chat.

A newer entry in this group is the AI crawler. OpenAI documents GPTBot, Anthropic documents ClaudeBot, and Common Crawl has long operated CCBot. Each publishes documentation describing the agent and how to allow or disallow it. Whether you want them is a business decision about how your content is used, not a technical defence question. Notably, blocking them is a genuine choice with a cost on both sides, which is not true of the next category.

Hostile automation

Here the operator's gain is your loss. Content and price scrapers copy your catalogue, often at a request rate that costs you real server capacity. Credential-stuffing scripts replay username and password pairs from unrelated breaches against your login endpoint, which is why a spike in failed logins from many IPs deserves attention within hours rather than at the next reporting cycle. Click fraud consumes paid advertising budget on clicks no human made. Form and comment spam pollutes your CRM and, worse, your conversion data. Inventory hoarding bots add limited stock to carts they never intend to buy.

The defining behaviour is evasion. This traffic rotates IP addresses, mimics common browser fingerprints, paces itself below obvious thresholds, and ignores anything you write in robots.txt, because complying would only reduce what it collects.

Synthetic test traffic

The third category is automation you commissioned: load tests, analytics validation runs, uptime checks you configured, staged sessions used to confirm that a conversion event fires with the right parameters. It is not hostile, but it is not human either, and it will distort any report that includes it. The distinguishing feature is that you control it, which means you can tag it, schedule it, and exclude it cleanly. Whether you actually do is the difference between a reversible experiment and a permanently muddled analytics property.

Category	Identifies itself?	Respects robots.txt?	Who should own the response
Search and AI crawlers	Yes, and verifiably	Generally yes	SEO / content strategy
Monitoring and preview fetchers	Usually	Often exempt by design	Engineering / operations
Scrapers and credential stuffing	No	No	Security
Click fraud and form spam	No	Not applicable	Paid media / security
Your own synthetic traffic	Only if you tag it	Under your control	Analytics owner

What GA4 filters automatically, and what slips through

Google Analytics 4 excludes traffic from known bots and spiders by default. Google's own documentation describes this as being based on the IAB/MRC International Spiders and Bots List combined with Google's internal identification, and it is applied automatically rather than as a setting you toggle. For the common case, declared crawlers hitting a page with your measurement code, it works well and quietly.

The limitation follows directly from the mechanism. A list-based filter catches automation that appears on a list. It is not designed to identify software that presents itself as a normal Chrome session from a residential IP address, and you should not expect it to. Three practical gaps are worth naming:

• Undeclared automation. Anything deliberately mimicking a browser will generally be measured as a session, with a plausible device, geography and referrer.
• Traffic that never runs your JavaScript. Many bots fetch HTML and stop. GA4 never sees them at all, so your analytics understates total load. This is the opposite failure mode from the one above, and it is why analytics alone cannot tell you how much automation your server is handling.
• Your own synthetic sessions. Nothing in the automatic filter knows that the load test you commissioned was not a customer. That exclusion is yours to build.

The general lesson: GA4's bot filter is a hygiene feature for reporting, not a detection system. Treat it as removing obvious noise, not as an answer to the question "how much of this traffic was human?"

How to tell them apart in server logs

Server logs record every request that arrived, including the ones that never executed a line of JavaScript. That makes them the more complete record, and the place where classification actually happens.

The signals that carry weight, roughly in order of reliability:

1. Verified identity. Google publishes a verification procedure for its crawlers: a reverse DNS lookup on the requesting IP, then a forward lookup on the resulting hostname to confirm it maps back to the same address. Google also publishes its crawler IP ranges as JSON, and OpenAI publishes ranges for GPTBot. Bing offers a comparable verification tool. This is the only class of signal a client cannot simply assert.
2. Request pattern. Perfectly regular intervals, sequential URL enumeration, and requests for HTML with no accompanying CSS, image or font fetches all suggest a non-browser client.
3. Endpoint concentration. Heavy traffic to /login, a search endpoint, or a price-bearing template, with little interest in anything else, describes a purpose.
4. Network origin. Datacentre address space is normal for declared crawlers and unusual for consumer visits. Residential proxy traffic inverts this and is one reason origin alone is not conclusive.
5. User-agent string. Useful for grouping, worthless as proof. It goes last for a reason.

One further point specific to classification: the group of clients that appear in your access log but never in analytics is, almost by definition, non-browser automation. That group is invisible to anyone working from analytics alone.

Why robots.txt and user-agent are weak signals

robots.txt is a request for voluntary cooperation. The Robots Exclusion Protocol was formalised as RFC 9309 in 2022, and the standard itself describes compliance as voluntary on the part of the crawler. Major search engines honour it; a scraper has no reason to. There is a further trap: publishing a Disallow rule for a sensitive path advertises that path to anyone reading the file, which is public by definition. Use it to manage crawl behaviour and crawl budget, never as access control.

The user-agent header has the same structural weakness. It is a string the client chooses. Anyone can send Googlebot in that field, and anyone can send a current Chrome string instead. That is why the verification step above exists at all. In practice, treat the user-agent as a label for organising traffic you have already classified by other means.

How much of the web is automated?

Honest answer: nobody can give you a figure you should quote with confidence. Published estimates come mainly from security vendors, most visibly Imperva's annual Bad Bot Report and similar telemetry from other providers. These are useful directionally, but they are derived from traffic passing through each vendor's own customer base, which skews toward sites large enough to buy bot mitigation and toward the attack surfaces those sites present. Different vendors, different methodologies, different numbers.

What is safe to say qualitatively is that the automated share is large enough to matter on essentially any public site, and that it varies enormously by site type. A login-heavy consumer service, a ticketing platform and a static documentation site do not face the same mix at all. The number that should drive your decisions is the one from your own logs, not an industry average.

When bot traffic becomes a security problem

The line is behavioural, not volumetric. Bot traffic is an analytics problem while it merely inflates counts. It becomes a security problem the moment it interacts with something that has consequences.

Escalate rather than filter when you see any of the following: a rise in failed login attempts spread across many source addresses, form submissions producing conversion events with no plausible downstream activity, checkout or cart actions that never complete, systematic enumeration of user IDs or order numbers, or request volume causing genuine capacity strain. Each of these has a cost that continues whether or not you exclude the traffic from a dashboard.

The practical consequence of getting this boundary wrong is that a credential-stuffing campaign gets handled as a reporting annoyance, filtered out of a GA4 view, and left running against the login endpoint. The filter made the symptom invisible without touching the cause.

Frequently asked questions

Is all bot traffic bad?

No, and treating it that way causes real damage. Blocking search engine crawlers removes you from the index. Blocking link preview fetchers breaks how your URLs appear when shared. The goal is classification and appropriate response, not elimination.

Does GA4 remove all bot traffic?

It removes traffic from known bots and spiders on the IAB/MRC list automatically. It does not remove automation that presents itself as an ordinary browser, and it never sees clients that do not execute your measurement code. Those are two different blind spots pointing in opposite directions.

Can I identify Googlebot from the user-agent string alone?

No. Google's documentation explicitly recommends verifying by reverse DNS lookup followed by a forward lookup, or by matching against Google's published crawler IP ranges. The user-agent is trivially spoofable and should never be the basis for granting access.

Should I block AI crawlers such as GPTBot or ClaudeBot?

That is a content licensing decision rather than a security one. Both are documented and both can be disallowed in robots.txt. Weigh the value of appearing in AI-generated answers against your position on how your content is used for training. There is no technically correct answer, only a commercial one.

What is the first thing to check on an unexplained traffic spike?

Server logs, not analytics. Look at which endpoints were hit, at what interval, from which networks, and whether the requests fetched page assets or only HTML. Those four observations usually identify the category within minutes, and the category determines who should handle it.