Skip to content
Nichr
Data & analyticsNew techTop 2% of today's analysed ideas

Analytics dashboard for AI crawler traffic

Build an AI-crawler analytics dashboard for website owners that logs server-side requests from Claude, ChatGPT, Perplexity and other LLM bots (invisible to JS-based tools like Google Analytics), showing which pages get crawled, how often, and how much referral traffic each AI assistant sends back. Target content sites, SaaS marketing teams and SEO agencies who want to understand and optimize for AI-driven discovery.

via Reddit
Original post

What to build

A server-log-based analytics dashboard that website owners install (via a log-shipping agent or CDN/reverse-proxy integration) to track AI crawler visits from Claude, ChatGPT, Perplexity and other LLM bots, showing per-page crawl frequency and any AI-referred traffic that JS-based tools like Google Analytics never capture.

AI crawlers fetch pages server-side and never execute JavaScript, so they're a fast-growing traffic source that's completely invisible to standard analytics stacks, creating an opening for a dedicated AI-crawler visibility tool.

Demand

Site owners, SEO agencies, and SaaS marketing teams optimizing for AI-driven discovery (being cited by ChatGPT/Claude/Perplexity) want to know which pages get crawled and how often, but currently have zero tooling built for this beyond raw server logs.

  • r/SideProject (Reddit)Community post

    Reddit post 'Claude crawled my site 571 times this month. My analytics never showed it.' — explains AI crawlers don't run JavaScript, so Google Analytics misses them entirely.

  • Site owner reportAnecdote

    Website owner's server logs showed 571 requests from Claude's crawler in one month while their JS-based analytics platform recorded zero corresponding visits.

  • Industry observationNews

    General pattern that LLM crawlers (Claude, ChatGPT, Perplexity) fetch pages via server-side HTTP requests that bypass JS-based analytics tags.

Stack

  • Cloudflare Workers / Logpush
  • Nginx or CDN access logs
  • ClickHouse or Postgres for log storage
  • User-agent/IP bot-verification (Anthropic, OpenAI, Perplexity published crawler lists)
  • Next.js dashboard
  • Stripe for billing

Solo + AI difficulty

The hard part is reliably parsing and verifying bot traffic (IP/reverse-DNS or signed user-agent verification to avoid spoofed bots) plus building a lightweight log-ingestion path that doesn't require heavy setup; the dashboard and charts themselves are straightforward. A scrappy MVP (log upload or Cloudflare Worker snippet + basic dashboard) is buildable in 1-2 weeks solo with AI coding help; robust multi-CDN support and referral-tracking add another few weeks.

Entry threshold
Requires parsing server/CDN logs or a lightweight edge middleware to detect bot user-agents and IP ranges, plus a simple dashboard; buildable solo in 1-2 weeks using existing log-parsing libraries and bot-identification lists, no licences needed.
Window
6-12 months

Where to find first users

  • r/SideProject and r/SEO
  • Indie Hackers
  • Product Hunt launch
  • Outreach to SEO agencies and newsletter writers covering 'AI search optimization' (AEO/GEO)

Competitors

  • Cloudflare Radar (AI bot traffic reporting)

Counter-signals & risks

  • Server log analysis and bot-detection tools (e.g., Cloudflare Radar, existing log analytics platforms) already provide some visibility into AI crawler traffic, reducing the novelty of a dedicated dashboard product.

  • AI crawler behavior and user-agent identifiers may change or be deliberately obfuscated over time, making consistent tracking and product reliability difficult to maintain.

  • Website owners may have limited willingness to pay for a niche analytics tool if the perceived business value of AI-driven referral traffic remains unclear or unmonetized.

Original title: Claude crawled my site 571 times this month. My analytics never showed it.

Related signals

  • Data & analytics#17

    Agentic data-exploration tool for solo analysts

    Build a narrow agentic BI wrapper that connects to a single database type (e.g. Postgres or Snowflake), lets a user ask a business question in plain language, and autonomously runs a multi-step query exploration loop (drill into cohorts, compare periods, test hypotheses) before returning a written finding plus supporting charts. Target solo founders, small e-commerce teams, or startup ops folks who currently pay for Tableau or Metabase but lack an analyst to actually interrogate the data.

    Demand
    5/10
    est.
    Buildability
    6/10
    est.
    Competition
    6/10
    est.
    via Hacker News — Show HN
    33 competitors
  • Data & analytics#18

    Dynamic pricing tools for small restaurants and shops

    Build an affordable AI-driven dynamic pricing add-on for independent restaurants and small retail chains that cannot afford McDonald's-scale AI systems. The tool would ingest POS sales data, local weather, foot traffic and competitor pricing to suggest daypart-based menu or product price adjustments, sold as a lightweight app on top of Toast, Square or Clover POS platforms.

    Demand
    4/10
    measured
    Buildability
    6/10
    est.
    Competition
    10/10
    est.
    via Hacker News
    0No named competitors
Analytics dashboard for AI crawler traffic — Nichr