Today we’re introducing a preview of TutorMoments, a framework to measure whether cutting-edge LLMs can balance one of the hardest trade-offs in education: when to step in and help a student and when to hold back and let the student do more of the work. (View Highlight)
TutorMoments is a replay-based evaluation built off real one-on-one math tutoring sessions. Experienced math teachers go through transcripts collected from a U.S. tutoring program and flag the moments where a tutor had to choose between making a problem easier to get started on and pushing the student to do more of the reasoning themselves. TutorMoments then takes the transcript up to that decision point, hands it to a language model, and has the model take over as the tutor in a simulated session – with the student played by another language model – to see what the LLM tutor does. (View Highlight)
Told only to “tutor well,” we find that models tend to over-help by giving too much support and rarely pushing students to do deeper thinking. Spelling out the trade-off (when to help versus when to hold back) in the tutor’s prompt improves performance, but it doesn’t close the gap to human tutoring that consistently fits the moment, and LLMs still differ widely in how reliably they make that call. (View Highlight)
As part of our commitment to open research, we’re releasing a dataset of de-identified tutoring transcripts, the code for running our replay pipeline, and the model tutor replays of the key moments we evaluated in those transcripts for reproducibility. We hope TutorMoments gives educators, researchers, and the teams building AI tutors a sharper way to ask how a model handles the pedagogical decisions that matter most—and helps the field build tutors that adapt to each student instead of doing the work for them. (View Highlight)
Ask a good math tutor for help and you’ll likely get a question back like, “What do you know about what the problem is asking?” That isn’t unhelpfulness–part of strong teaching is diagnosing what students do know and providing the right support for them in the moment. Immediately volunteering support would rob a student of the intellectual work that helps them learn. Sometimes support is needed; other times what’s most effective is a push to solidify understanding by explaining a correct answer. (View Highlight)
Language models, though, are trained to be helpful, and a helpful assistant tends to do the hard part for you—explaining the concept, laying out the steps, and guiding you to the answer. In a tutoring session, that can cut short the productive struggle—the effortful, sometimes frustrating problem-solving that learning research has long tied to stronger understanding. (View Highlight)
Most benchmarks for language models acting as tutors don’t capture this tension. They tend to reward one behavior in particular – never giving away the answer to a problem, say, or always offering a hint – without accounting for whether that was the right move for where the student actually was in their understanding. But good tutoring isn’t a single fixed behavior you can identify across the board. It’s a judgment call: what does this student need, right now, on this problem? (View Highlight)
TutorMoments is built on real tutoring data. The dataset we’re releasing, TutorMoments-Preview, is 462 de-identified, text-only transcripts of real one-on-one math tutoring with U.S. students in grades 2-7, with more than 1,500 teacher-annotated key moments and several thousand free-text annotations from 27 U.S.-based teacher annotators. The transcripts come from a high-dosage tutoring program whose students mostly attend Title I schools, shared under a research clause agreed to by parents and guardians; all data was stripped of identifying details, first by the provider and then through an additional math-aware pipeline. (View Highlight)