Turnkey custom coding-agent benchmarks for small teams
Build a lightweight, hosted eval-generation service for small dev teams: given a GitHub repo, auto-generate PR-based regression benchmarks for coding agents and output a simple dashboard comparing model/agent performance on the team's own codebase, undercutting self-hosted or enterprise-grade tools with a turnkey SaaS for indie developers and small startups who want to know which coding agent actually works on their stack without setting up sandboxes themselves.
What to build
A hosted SaaS that connects to a small team's GitHub repo, auto-generates regression benchmarks from real past PRs and bugs in that codebase, then runs multiple coding agents (Claude, Copilot, Cursor, etc.) against them and shows a dashboard of which agent actually produces working, mergeable code on their specific stack.
Self-bench's Show HN proves there's appetite for benchmarking coding agents against real software instead of toy problems, but it's a self-hosted dev tool — a turnkey hosted version lowers the setup bar for small teams who just want a quick answer on which agent works on their repo.
Demand
Solo builders and small startups now juggle several coding agents and have no fast way to know which one actually performs on their own codebase and build environment, rather than on a generic public leaderboard.
- Hacker NewsNews
Show HN: Self-bench – benchmark coding agents on real-world software — 4 points, 0 comments, introducing an open-source tool for benchmarking agents against real repos instead of synthetic problems
Stack
- GitHub REST/Webhooks API
- Docker or E2B/Daytona sandboxes for isolated test runs
- Claude API, OpenAI API, GitHub Copilot API
- Postgres (via Supabase)
- Temporal or BullMQ for job orchestration
- Next.js dashboard
Solo + AI difficulty
The hard part is reliably sandboxing arbitrary repos (install steps, flaky or non-deterministic tests, varied build tools) well enough to score a PR as pass/fail — this is real infra work, not prompt engineering. The GitHub ingestion, agent API calls, and dashboard UI are comparatively easy with AI assistance. Realistic MVP for one language/framework (e.g. Node or Python) is 4-6 weeks solo.
- Entry threshold
- Core pieces (PR-to-eval generation, sandboxed harness execution, model API billing passthrough) are moderate to build solo with AI assistance over a few weeks, but reliable sandboxing infrastructure and cost-control for running many models adds real engineering overhead beyond a weekend project.
- Window
- 6-12 months
Where to find first users
- Show HN launch
- Indie Hackers community
- r/ExperiencedDevs and r/programming
- Product Hunt launch
Competitors
Counter-signals & risks
Established eval/observability platforms and model labs could add repo-specific regression benchmarking as a feature, eroding differentiation for a standalone SaaS.
Auto-generating meaningful PR-based regression benchmarks from arbitrary repos is technically hard (flaky tests, non-deterministic agent outputs, repo-specific build/test environments), limiting turnkey feasibility and raising support burden.
Price-sensitive indie developers and small startups may prefer free, self-hosted tools like the original self-bench project over a paid hosted service.
Original title: Show HN: Self-bench – benchmark coding agents on real-world software
Related signals
- Developer tools#4
MCP Connector Library for AI Coding Agents
Build a ready-made MCP (Model Context Protocol) server or toolkit that connects Claude, Cursor, and Codex to popular third-party APIs and services, enabling AI coding agents to automate integrations without manual glue code. Package it as a hosted service or CLI tool targeting solo developers and small teams who use AI-first IDEs. Monetize via subscription access to a growing library of pre-built MCP connectors.
Demand7/10est.Buildability7/10est.Competition7/10est.via Product Hunt22 competitors - Developer tools#7
Plan State Persistence Layer for Claude Code Re-Runs
Build a lightweight plan persistence and idempotency layer for Claude Code that tracks executed plan steps, stores state between runs, and prevents duplicate actions on re-runs. Target solo developers and small teams using Claude Code for multi-step agentic workflows who lose progress when sessions restart or plans re-execute from scratch.
Demand6/10est.Buildability8/10est.Competition8/10est.via Product Hunt11 competitor - Developer tools#8
Developer Context Save and Restore Tool as Micro-SaaS
Build a lightweight SaaS or CLI-plus-web-app called something like Park that lets developers save and restore full work context — open files, terminal state, branch, notes, clipboard snippets — across multiple repos and machines. Sell it as a subscription to solo developers and small teams who constantly context-switch and lose their train of thought. Integrate with VS Code and JetBrains as a plugin for frictionless adoption.
Demand1/10measuredBuildability7/10est.Competition6/10est.via Hacker News33 competitors