Pelayo Arbués

How to Create Your Own Personal AI Benchmark

10 min read

rw-book-cover

Metadata

Highlights

  • Nobody hires a vice president based on their SAT scores. Yet every time a new model drops, AI researchers first check how well it does on a Math Olympiad and a set of multiple-choice trivia, then argue online about whether it’s the smartest model in the world. Wharton professor Ethan Mollick points out that Massive Multitask Language Understanding-Pro (MMLU-Pro), one of the most-cited benchmarks, asks models for the approximate cranial capacity of Homo erectus and the place named in the title of Cheap Trick’s 1979 live album. Those tests measure something—the scores are directionally useful—but not what you need to know: Can this model help you with your job? To answer that, you need to build a personal benchmark. (View Highlight)
  • joined Every earlier this year to run our technology consulting practice, so I spent most of my time getting models to write, build dashboards, and assemble slide decks. With each new model, I developed an instinct for when and how to use it. Claude Opus 5 felt argumentative. Claude Fable 5 felt like talking to a genius. GPT-5.6 Sol felt like a safe pair of hands. But when friends and colleagues asked me for recommendations, I couldn’t always defend my choices. It was also time-consuming to test every model on every type of task. I often missed opportunities to use something better or cheaper. So, inspired by Mollick’s piece about why you should test models on your own work instead of trusting benchmarks, I built a personal one: a small, private test that tells me whether I like working with a model, without spending all my time testing—because I have a job to do. (View Highlight)
  • Now I’m head of evaluations (evals) at Every, testing new models from the frontier labs for the qualities we value in our work. I want to help everyone on the team build personal benchmarks for their own work. Building personal benchmarks has three levels, and each can make you more confident you’re using the right model for your job. If you set up your personal benchmarks correctly, you’ll know when to switch between Astra and Fable, and whether you can save money using a smaller, cheaper model like Luna or Haiku. (View Highlight)
  • Fill your back pocket with failures
    If your current daily driver (the model you use for most things) performs well on certain tasks—screening your email or building dashboards—it’s less useful to keep testing new models on those tasks unless they can deliver the same results more cheaply. Instead, think about the last few times AI failed you. Steve Yegge calls these back-pocket evals. “Whenever I give a project to a model, and it can’t do the project, I add it to my pocket-eval list,” he wrote. “Then every time a new model drops, it’s like Christmas.” When a new model can make progress on tasks AI previously failed at, you know it’s worth adopting. (View Highlight)
  • Let me take you through the process I’ve been using with the team to help you build your first benchmark. Review the past month’s work and identify 10 tasks. If you’re using the Claude or Codex desktop app, you can ask AI to help collect this list: (View Highlight)
  • The desktop apps store past sessions in a folder the agent can access, so it can search them for tasks it didn’t get right the first time. For each failed task, make sure you capture all the context needed to complete it: any reference files and any advice you’d give an intern doing the work. The context for most of my consulting tasks includes an NPS dashboard, a client PowerPoint deck, an article I published, and the research document from when I had to choose a hotel for business travel—all tasks I completed.
    Then open Claude Code and tell it to run all 10 tasks in parallel on several models and show the outputs side by side. Ask it to build an eval viewer—an HTML page that displays each model’s responses for comparison. Do the following until you form an opinion on the outputs:
    • Compare a large model like Fable against a small one like Haiku.
    • Compare models from different providers, like Anthropic and OpenAI.
    • Run the same task with the same model three times to see how much it varies.
    You may have seen a model perform surprisingly well on tasks you haven’t recently assigned it. You may also have noticed differences in outputs across models and providers. (View Highlight)
  • Turn your taste into tests
    Using the models yourself is the best way to choose your daily driver, but not everybody can afford to try every new one. It’s also hard to stay consistent in your assessments unless you’re rigorous. Your opinion on a model may fall when you’re in a bad mood or soar if it comes through for you on an important task. (View Highlight)
  • Start teaching AI your preferences by switching on voice mode for your favorite model and dictating what you liked or disliked about its output for each task:

    “This headline was a banger.”
    “It sounds like AI writing.”
    “You didn’t follow my brand guidelines.”
    The more opinions, the better. That’s your taste, and it will guide your models toward your standards. Ask the model to turn this unstructured feedback into repeatable evals—a rubric for grading its work.
    Once a model understands what you like, it can apply the checks you would use when reviewing the outputs and judge them according to your preferences. Run these evals whenever you benchmark a new model to check whether its response matches your expectations. One AI model does the task and another checks the work—a technique called LLM-as-a-judge. For example, if you’re unenthused by the model’s headlines, you could ask: (View Highlight)

  • Resist the urge to ask AI to score model outputs on a 100-point scale. A binary yes or no works best. AI is bad at assigning ratings because its ratings tend to cluster around 4/5 or 70/100. An LLM will give you a number, but it doesn’t have the precision the number implies. Instead, it forms a rough qualitative judgment—bad, acceptable, good—and then generates a plausible number for that category. A model that got 70 out of 100 on conciseness might not be clearly better than one that got 65/100. Would you replace an entire workflow to gain those extra five points? Pass/fail tests on three to 10 specific criteria tell you more than a score. If a model passes all the checks you would use in your work, you should use it. (View Highlight)
  • When I ran my consulting private benchmark—a set of 10 tasks from dashboard building to writing articles and PowerPoint creation—recently, the cheapest OpenAI model, Luna, came within a few points of Fable. I checked Luna’s outputs and found that it performed well on those tasks. Does that make Luna the better model? No. It means Luna was good enough for me to use on those tasks. They sit below what Yegge calls the discernment horizon—the “good enough” line that most models meet, which makes it hard to tell the difference between them. Ask Einstein to make coffee, and he might do a decent job, but you won’t get a sense of how good he is at physics. As Yegge says, “If all you have handy are easy problems, they don’t give a smarter model any room to pull ahead.” Anthropic found that the difference between Opus and Fable is negligible except on substantial and open-ended tasks, like building a multi-agent simulation of a town. I can delegate trivial or routine tasks like dashboard building to Luna, which is hundreds of times cheaper, and send the more ambitious ones to Fable and Sol. (To check Luna’s results, use my personal benchmark viewer to compare its outputs and checks with those of other models.) (View Highlight)
  • Scale it into a real benchmark
    If you want to recommend a model to others, you’ll eventually need more certainty in your benchmark scores, so you’ll need to add both more tasks and more cases per task. You’ll also need to make the tasks harder. Handling one PowerPoint well is less convincing than generating 20 good ones per week; one success can be a fluke, but many indicate a system working well. I’m at this point with my personal benchmark, and it’s where my private test turns into a company engineering problem.
    An easy way to collect more examples is to turn each task into a skill, so you don’t have to repeat your feedback. For example, I have an NPS dashboard skill, a deck skill, and an AI writing skill that captures my taste. Whenever you run a task, capture the brief, output, and feedback, and have the agent file them. Instead of one example for each of your 10 tasks, you’ll have 20 or more per task—enough to answer, “How good is this model at PowerPoint?” (View Highlight)
  • Click index.html to load the evals in a browser and view them side by side. Each task folder contains examples or cases to review whenever you run a benchmark. Claude can create a script that runs all cases at once, so you don’t have to wait hours. The Context folder contains all documents needed for the task. Gold shows the ideal result—in the PowerPoint example, the slides you presented. evals.py is a script that runs the tests and scores the model for that case. Keeping everything in a consistent format lets you navigate the tasks and compare them more easily. (View Highlight)
  • Your benchmark will expire
    Treat your personal benchmark as a living document. When Fable came out, it was so good at open-ended work like analyzing thousands of customer reviews or writing an entire book that I realized I’d been giving AI tasks that were too small. Then my benchmark had to evolve with my more ambitious use of AI. Mollick used to ask AI to draw an “otter on a plane using wifi,” which was a useful quality marker because it was universally hard. Now it’s too easy for most models. That benchmark has been “saturated”—the models are getting it right every time. Mollick escalated his baseline task to “1960s documentary footage of the band’s last concert before the incident with the swarm of otters.”
    This is how you “surf the models,” as Every CEO Dan Shipper calls it. Models are getting better every day. To get the most from them, we have to keep up. AI knows everything that’s written down, but using AI models in your work generates judgment they couldn’t have been trained on by anyone else—no matter how good the intelligence gets. Your benchmark turns that judgment into a repeatable evaluation. If you’re regularly testing how well AI can help you with your job and where it still fails, it’ll be easier to navigate the frantic pace of change in the AI industry and focus on what matters for your job. (View Highlight)

Graph View

Backlinks