Independent verification tool for AI coding benchmark scores
Build an independent AI coding benchmark verification tool that lets developers and teams re-run, audit, and stress-test published benchmark scores (like DeepSWE or SWE-bench leaderboard entries) against their own codebases or custom task suites. The product targets developers, AI researchers, and CTOs who need to know whether a model's claimed score translates to real-world coding performance before committing to it. Monetise via a SaaS subscription with a free tier for public re-runs and paid tiers for private repo testing and detailed reports.
What to build
A web-based AI coding benchmark verification SaaS that lets developers, AI researchers, and CTOs re-run and audit published benchmark scores (SWE-bench, DeepSWE) against their own private repos or custom task suites, then get a plain-English report comparing claimed scores to real-world performance on their actual codebase.
Published AI coding benchmark leaderboards are increasingly distrusted — developers suspect benchmark gaming, data contamination, and overfitting mean a model's headline score tells them little about real-world usefulness. There is a clear gap for an independent, reproducible verification layer between 'leaderboard number' and 'should I pay for this model'.
Demand
Developers and engineering leads evaluating which coding AI to adopt need ground-truth verification before committing budget or workflow changes; the skepticism is peaking right now as DeepSWE and SWE-bench scores proliferate and differentiation between models narrows on paper.
- MaxxBench on Hacker NewsData
MaxxBench post titled 'Nice DeepSWE score. But is it true?' received 211 points and 64 comments on Hacker News, signalling high developer engagement and widespread skepticism about published benchmark validity.
- SWE-bench LeaderboardData
SWE-bench is the most widely cited coding benchmark for LLMs; multiple researchers have publicly raised concerns about data contamination, task selection bias, and overfitting, which directly erodes developer trust in leaderboard scores and creates demand for independent re-run tooling.
- AI Research Community DiscourseNews
Recurring public debates about benchmark gaming and training-set leakage across AI forums and social channels indicate a sustained, growing appetite for third-party verification tools among practitioners.
Stack
- Next.js (frontend + API routes)
- Supabase (auth, database, storage for task suites and reports)
- Docker / Modal for sandboxed benchmark task execution
- OpenAI API or Anthropic API (model-under-test integration)
- GitHub API (private repo ingestion for custom task generation)
- Stripe (subscription billing for paid tiers)
Solo + AI difficulty
The core difficulty is compute isolation — safely running model-generated code patches in sandboxed environments without runaway costs or security exposure; this is the hardest part and will take 3-5 weeks to get right. The dashboard, reporting layer, and SaaS plumbing are straightforward with modern scaffolding and AI assistance. Realistic time-to-MVP with a constrained public task set (no private repo support yet): 6-8 weeks solo.
- Entry threshold
- Buildable in a few weeks using existing benchmark harnesses (SWE-bench, HumanEval) wrapped in a clean UI plus LLM API calls; no special licences needed. Main effort is scaffolding the eval runner and building a credible leaderboard with reproducible methodology.
- Window
- 3-9 months
Where to find first users
- Hacker News Show HN launch — the existing HN thread proves the audience is there and already discussing the problem
- r/MachineLearning and r/LocalLLaMA — post benchmark audit results as content to drive tool signups
- Direct outreach to AI-adjacent CTOs and engineering leads on LinkedIn with a free public re-run report as the hook
- AI Twitter/X — share live re-run results comparing claimed vs. actual model scores to generate organic sharing
Competitors
Counter-signals & risks
Established AI labs (OpenAI, Anthropic, Google DeepMind) may release their own transparent, reproducible evaluation frameworks, reducing developer demand for a third-party verification product.
Re-running benchmark suites requires significant compute resources per task execution, making a free tier economically unsustainable and potentially blocking adoption among individual developers who most need it.
Building a generalised verification tool that works across diverse benchmark formats, model APIs, and private codebases at scale is technically very hard for one developer, risking a tool that only works well on a narrow set of public benchmarks.
Original title: MaxxBench – "Nice DeepSWE score. But is it true?"
Related signals
- Developer tools#4
MCP Connector Library for AI Coding Agents
Build a ready-made MCP (Model Context Protocol) server or toolkit that connects Claude, Cursor, and Codex to popular third-party APIs and services, enabling AI coding agents to automate integrations without manual glue code. Package it as a hosted service or CLI tool targeting solo developers and small teams who use AI-first IDEs. Monetize via subscription access to a growing library of pre-built MCP connectors.
Demand7/10est.Buildability7/10est.Competition7/10est.via Product Hunt22 competitors - Developer tools#7
Plan State Persistence Layer for Claude Code Re-Runs
Build a lightweight plan persistence and idempotency layer for Claude Code that tracks executed plan steps, stores state between runs, and prevents duplicate actions on re-runs. Target solo developers and small teams using Claude Code for multi-step agentic workflows who lose progress when sessions restart or plans re-execute from scratch.
Demand6/10est.Buildability8/10est.Competition8/10est.via Product Hunt11 competitor - Developer tools#8
Developer Context Save and Restore Tool as Micro-SaaS
Build a lightweight SaaS or CLI-plus-web-app called something like Park that lets developers save and restore full work context — open files, terminal state, branch, notes, clipboard snippets — across multiple repos and machines. Sell it as a subscription to solo developers and small teams who constantly context-switch and lose their train of thought. Integrate with VS Code and JetBrains as a plugin for frictionless adoption.
Demand1/10measuredBuildability7/10est.Competition6/10est.via Hacker News33 competitors