Turnkey custom coding-agent benchmarks for small teams

What to build

A hosted SaaS that connects to a small team's GitHub repo, auto-generates regression benchmarks from real past PRs and bugs in that codebase, then runs multiple coding agents (Claude, Copilot, Cursor, etc.) against them and shows a dashboard of which agent actually produces working, mergeable code on their specific stack.

Self-bench's Show HN proves there's appetite for benchmarking coding agents against real software instead of toy problems, but it's a self-hosted dev tool — a turnkey hosted version lowers the setup bar for small teams who just want a quick answer on which agent works on their repo.

Demand

Solo builders and small startups now juggle several coding agents and have no fast way to know which one actually performs on their own codebase and build environment, rather than on a generic public leaderboard.

• Hacker NewsNews

  Show HN: Self-bench – benchmark coding agents on real-world software — 4 points, 0 comments, introducing an open-source tool for benchmarking agents against real repos instead of synthetic problems

Stack

• GitHub REST/Webhooks API
• Docker or E2B/Daytona sandboxes for isolated test runs
• Claude API, OpenAI API, GitHub Copilot API
• Postgres (via Supabase)
• Temporal or BullMQ for job orchestration
• Next.js dashboard

Solo + AI difficulty

The hard part is reliably sandboxing arbitrary repos (install steps, flaky or non-deterministic tests, varied build tools) well enough to score a PR as pass/fail — this is real infra work, not prompt engineering. The GitHub ingestion, agent API calls, and dashboard UI are comparatively easy with AI assistance. Realistic MVP for one language/framework (e.g. Node or Python) is 4-6 weeks solo.

Entry threshold

Core pieces (PR-to-eval generation, sandboxed harness execution, model API billing passthrough) are moderate to build solo with AI assistance over a few weeks, but reliable sandboxing infrastructure and cost-control for running many models adds real engineering overhead beyond a weekend project.

Window

6-12 months

Where to find first users

• Show HN launch
• Indie Hackers community
• r/ExperiencedDevs and r/programming
• Product Hunt launch

Competitors

• self-bench
• SWE-bench
• Braintrust
• LangSmith

Counter-signals & risks

• Established eval/observability platforms and model labs could add repo-specific regression benchmarking as a feature, eroding differentiation for a standalone SaaS.

• Auto-generating meaningful PR-based regression benchmarks from arbitrary repos is technically hard (flaky tests, non-deterministic agent outputs, repo-specific build/test environments), limiting turnkey feasibility and raising support burden.

• Price-sensitive indie developers and small startups may prefer free, self-hosted tools like the original self-bench project over a paid hosted service.