Skip to content
Nichr
Data & analyticsNew techTop 2% of today's analysed ideas

Analytics dashboard for AI crawler traffic

Build an AI-crawler analytics dashboard for website owners that logs server-side requests from Claude, ChatGPT, Perplexity and other LLM bots (invisible to JS-based tools like Google Analytics), showing which pages get crawled, how often, and how much referral traffic each AI assistant sends back. Target content sites, SaaS marketing teams and SEO agencies who want to understand and optimize for AI-driven discovery.

via Reddit
Original post

What to build

A server-log-based analytics dashboard that website owners install (via a log-shipping agent or CDN/reverse-proxy integration) to track AI crawler visits from Claude, ChatGPT, Perplexity and other LLM bots, showing per-page crawl frequency and any AI-referred traffic that JS-based tools like Google Analytics never capture.

AI crawlers fetch pages server-side and never execute JavaScript, so they're a fast-growing traffic source that's completely invisible to standard analytics stacks, creating an opening for a dedicated AI-crawler visibility tool.

Demand

Site owners, SEO agencies, and SaaS marketing teams optimizing for AI-driven discovery (being cited by ChatGPT/Claude/Perplexity) want to know which pages get crawled and how often, but currently have zero tooling built for this beyond raw server logs.

  • r/SideProject (Reddit)Community post

    Reddit post 'Claude crawled my site 571 times this month. My analytics never showed it.' — explains AI crawlers don't run JavaScript, so Google Analytics misses them entirely.

  • Site owner reportAnecdote

    Website owner's server logs showed 571 requests from Claude's crawler in one month while their JS-based analytics platform recorded zero corresponding visits.

  • Industry observationNews

    General pattern that LLM crawlers (Claude, ChatGPT, Perplexity) fetch pages via server-side HTTP requests that bypass JS-based analytics tags.

Stack

  • Cloudflare Workers / Logpush
  • Nginx or CDN access logs
  • ClickHouse or Postgres for log storage
  • User-agent/IP bot-verification (Anthropic, OpenAI, Perplexity published crawler lists)
  • Next.js dashboard
  • Stripe for billing

Solo + AI difficulty

The hard part is reliably parsing and verifying bot traffic (IP/reverse-DNS or signed user-agent verification to avoid spoofed bots) plus building a lightweight log-ingestion path that doesn't require heavy setup; the dashboard and charts themselves are straightforward. A scrappy MVP (log upload or Cloudflare Worker snippet + basic dashboard) is buildable in 1-2 weeks solo with AI coding help; robust multi-CDN support and referral-tracking add another few weeks.

Entry threshold
Requires parsing server/CDN logs or a lightweight edge middleware to detect bot user-agents and IP ranges, plus a simple dashboard; buildable solo in 1-2 weeks using existing log-parsing libraries and bot-identification lists, no licences needed.
Window
6-12 months

Where to find first users

  • r/SideProject and r/SEO
  • Indie Hackers
  • Product Hunt launch
  • Outreach to SEO agencies and newsletter writers covering 'AI search optimization' (AEO/GEO)

Competitors

  • Cloudflare Radar (AI bot traffic reporting)

Counter-signals & risks

  • Server log analysis and bot-detection tools (e.g., Cloudflare Radar, existing log analytics platforms) already provide some visibility into AI crawler traffic, reducing the novelty of a dedicated dashboard product.

  • AI crawler behavior and user-agent identifiers may change or be deliberately obfuscated over time, making consistent tracking and product reliability difficult to maintain.

  • Website owners may have limited willingness to pay for a niche analytics tool if the perceived business value of AI-driven referral traffic remains unclear or unmonetized.

Original title: Claude crawled my site 571 times this month. My analytics never showed it.

Related signals

  • Data & analytics#6

    Real-time risk alerts for Hormuz shipping chokepoint

    Build a real-time Strait of Hormuz transit-risk monitor for oil traders, freight forwarders, and marine insurers: combine AIS vessel-tracking feeds, war-risk insurance premium data, and news/attack reports into a risk score with Telegram/Slack/email alerts, sold as a subscription to brokers and small trading desks who can't afford enterprise maritime-intelligence platforms.

    Demand
    5/10
    measured
    Buildability
    7/10
    est.
    Competition
    5/10
    est.
    via Hacker News
    44 competitors
  • Data & analytics#10

    Curated AI safety news and research tracker

    Build a curated newsletter or dashboard that monitors AI safety news, research papers, and policy developments, aggregating sources like arXiv, lab blogs, and regulatory filings into a daily or weekly digest for AI safety researchers, policy teams, and compliance officers at AI companies. Monetize via a paid tier with search, alerts, and API access for teams tracking specific labs or topics.

    Demand
    9/10
    measured
    Buildability
    8/10
    est.
    Competition
    5/10
    est.
    via GitHub
    44 competitors
  • Data & analytics#10

    Packaged local-first data ontology tool for SMBs

    Build a packaged, local-first data unification tool for small businesses and solo consultants: connect spreadsheets, CRMs, and SaaS exports into one queryable ontology on the user's own machine using HelixDB and DuckDB under the hood, with a simple setup wizard and pre-built connectors. Target ops teams and agencies who want a single source of truth without sending data to the cloud or hiring a data engineer.

    Demand
    9/10
    measured
    Buildability
    6/10
    est.
    Competition
    6/10
    est.
    via GitHub
    33 competitors
Analytics dashboard for AI crawler traffic — Nichr