Archive-as-a-service for vanishing government web pages

What to build

A scheduled snapshot-and-diff tool that archives EPA (and other federal agency) web pages, datasets, and PDFs on a recurring basis, stores timestamped versions in a searchable repository, and alerts users when content is removed or edited — built for journalists, environmental nonprofits, researchers, and FOIA/litigation teams who need to prove what the agency previously published.

Reporting that the EPA is letting its public web archive degrade creates a narrow but real niche: a self-serve, searchable 'proof of what the government said' tool for people who currently rely on manual screenshots, the Wayback Machine, or ad hoc nonprofit tracking projects.

Demand

Journalists, environmental advocacy groups, and lawyers who cite federal agency pages need durable, timestamped, citable proof when that content disappears — demand spikes whenever a removal story breaks, as this one did with strong HN engagement.

• Hacker News / Mother JonesNews

  'The EPA Is Quietly Killing Its Web Archive' reached 259 points and 49 comments on Hacker News — unusually high engagement for a regulatory/civic-tech story, indicating a technically literate audience actively discussing the problem.

• EDGI (Environmental Data & Governance Initiative)Precedent

  A nonprofit (originally formed in 2017) has run manual, volunteer-driven monitoring of EPA website changes for years — evidence that sustained demand for this tracking exists, but no polished, self-serve commercial product currently fills it.

Stack

• Playwright (scheduled page snapshots + rendering)
• wget/httrack or a custom crawler for bulk page + PDF capture
• S3-compatible storage (Backblaze B2 or Cloudflare R2) for raw snapshots
• Postgres + full-text/pgvector search for the searchable archive
• A diffing library (htmldiff / diff-match-patch) for change detection
• Cron-based worker (GitHub Actions or a small VPS cron) for scheduling

Solo + AI difficulty

The crawling, diffing, and storage pipeline is straightforward to build solo with AI help in 2-4 weeks; the hard parts are handling JS-heavy government pages reliably at scale, avoiding being rate-limited or blocked, and building a search/alerting UI good enough that journalists trust it as a citable source rather than falling back to the Wayback Machine.

Entry threshold

Low capital barrier; a solo builder can combine scheduled scraping, diffing, and cloud storage (or wrap existing tools like the Wayback Machine API and Common Crawl) into a working product within one to two weeks, though sourcing and legal reliability of scraped government data needs care.

Window

3-6 months

Where to find first users

• Pitch directly to journalists and outlets already covering this story (Mother Jones, Grist, InsideClimate News)
• r/journalism and r/datahoarder
• Environmental nonprofit and FOIA-focused mailing lists / Slack communities
• Hacker News 'Show HN' launch given existing thread engagement on this exact topic

Competitors

• Internet Archive Wayback Machine
• End of Term Web Archive
• EDGI (Environmental Data & Governance Initiative)
• Visualping

Counter-signals & risks

• The reduction in archive content could be due to routine IT infrastructure changes, budget constraints, or technical migration rather than an intentional effort to suppress information.

• Existing third-party archival services (e.g., Internet Archive's Wayback Machine, End of Term Web Archive) may already capture much of the at-risk content, reducing the unique value of a new tool.

• Government web archive changes may be contested or reversed through FOIA requests, congressional oversight, or litigation, limiting the long-term market need for a dedicated monitoring product.