Skip to main content

Bots and AI crawlers

Bots and AI/LLM crawlers — GPTBot, ClaudeBot, PerplexityBot, Bytespider and the rest — are detected and excluded by default. Nothing needs configuring for that; it is the shipped posture, because a crawler in your visit numbers makes every other number wrong.

'bots' => [
    'track' => false,               // record bots at all?
    'detect_ai_crawlers' => true,   // the maintained AI/LLM crawler token list
    'detect_generic' => true,       // generic crawler signals
    'allow' => [],                  // UA tokens always treated as human
    'deny' => [],                   // UA tokens always treated as bots
    'detector' => null,             // your own callable
],

Set bots.track to true to record them instead of excluding them. Detection still runs — it is what the report needs to distinguish them — only the gate stops denying.

The layers, in order​

Detection is a cheap case-insensitive substring match against the User-Agent, in a fixed order. The first layer that decides wins:

  1. An empty User-Agent is a bot. No real browser omits it.
  2. bots.allow — a matching token is treated as human, overriding every layer below. This is the escape hatch for a false positive.
  3. bots.deny — a matching token is treated as a bot.
  4. The AI-crawler list — a curated, auto-updated list of AI and LLM crawler tokens. Switch it off with detect_ai_crawlers.
  5. Generic crawler signals — bot, crawler, spider, slurp, +http, the common HTTP client identifiers (curl/, wget, python-requests, …), the big search engines, the SEO crawlers, and the social link-preview fetchers that carry none of the other signals. Switch it off with detect_generic.
  6. Your own detector — consulted last, and only reached when nothing above decided.

allow sitting above everything is what makes this safe to tune. If a legitimate client is caught by a generic signal — a monitoring agent whose name contains bot, say — add its token to allow rather than turning a whole layer off.

The AI-crawler list stays current​

The AI-crawler landscape moves fast, so the list is maintained automatically rather than hand-curated on release cadence. A scheduled workflow regenerates it from the canonical ai.robots.txt catalog — and from Cloudflare Radar when a token is configured — and opens a pull request for review.

The review step is deliberate: a token that is a substring of a real browser's User-Agent would silently exclude real visitors, so no list update ships unreviewed.

Exhaustive coverage, optionally​

Substring matching against a curated list is fast and covers what matters. For exhaustive, always-current coverage of every category — search, social, SEO/marketing, monitoring, and so on — opt into the same catalog Matomo itself uses server-side:

composer require matomo/device-detector
// config/matomo-analytics.php
'bots' => [
    'detector' => \MatomoAnalytics\Bots\DeviceDetectorBotDetector::class,
],

The dependency is a suggestion, not a requirement, so the package stays light for everyone who does not need it. The wrapper is consulted as the last detection layer, so your allow list still wins over it.

Writing your own detector​

bots.detector takes any callable that receives the User-Agent string and returns true for a bot:

namespace App\Analytics;

final class ExcludeOurLoadTester
{
    public function __invoke(string $userAgent): bool
    {
        return str_contains($userAgent, 'acme-loadtest');
    }
}

Only true counts as a detection; anything else leaves the earlier layers' verdict intact. Use an invokable class-string so config:cache keeps working — see configuration.

Three kinds of AI traffic​

The list above is one of three, and they are genuinely different things. Matomo distinguishes AI crawlers (training and indexing bots, covered here), AI assistants as an acquisition channel, and AI chatbots — the on-demand fetchers that read a page for a user. See AI assistants for the other two.