StudentSim-for-hire: pre-launch eval harness for AI tutoring products
- Problem
- Edtech companies shipping LLM tutors test them on real kids or on thin scripted prompts. That is slow, risky and hard to repeat. Regressions after a model swap go unnoticed.
- Who pays
- Edtech product and ML teams (Khan-style tutoring apps, curriculum publishers, university online-learning units) pay $1-3k/month per product from their QA or R&D budget.
- Why now
- Microsoft Research just published the simulated-student approach, which validates the method. New frontier models keep arriving and teams constantly need to re-evaluate, so a regression suite is a recurring need.
- First MVP
- Build a hosted service that takes a tutor's API endpoint plus a subject and grade level. It runs a library of simulated learner personas (misconception-prone, disengaged, gaming the system) through multi-turn sessions and scores pedagogy: hints vs answer-leaking, misconception handling. Deliver a CI-friendly report that diffs between model versions.
- Risks & competition
- Simulated students may not correlate with real learner outcomes, and buyers will ask for proof. Microsoft could open-source it, and general eval tools like Braintrust, LangSmith and Promptfoo can be configured for the same job. The edtech market is small and slow to buy.
AI · claude-sonnet-5-5