We\u2019ve built AI systems that can spend hours hunting across the web, synthesizing information, and writing research reports. But we have almost no way to tell if they\u2019re actually good at this task.The problem runs deeper than it first appears. Traditional benchmarks work fine for closed-form questions with single correct answers. Feed a system a math problem, check if it matches the known solution, move on. But research is different. There are many valid approaches to answering a question about renewable energy policy, and multiple correct answers depending on what sources you integrate and how you weight them. A static answer key doesn\u2019t capture this nuance.There\u2019s a worse problem hiding underneath: static ground truth becomes obsolete. If your benchmark was created last year and a system is researching current events, comparing it to pre-written answers makes no sense. The world has moved on.Current benchmarks also impose a heavy cost. Creating reliable research tasks requires human annotation at scale, which is expensive and slow. Existing approaches either demand painstaking effort to construct each task, assume evaluation criteria are universal (they\u2019re not, a business analyst needs different things than a historian), or fail completely when systems cite sources that don\u2019t exist or skip citations altogether.DeepResearchEval addresses this by automating both the creation of realistic research challenges and the evaluation of how well systems handle them. The insight that ties everything together: you can\u2019t fairly evaluate research systems without task-specific evaluation criteria, and you can\u2019t verify factual claims without an evaluator that actively hunts for evidence rather than checking a static answer key.What makes a real research taskBefore grounding a solution, it helps to think about how real research actually works. A person doesn\u2019t start with a random question. They first think about who they are, what they\u2019re trying to accomplish, and why it matters. A journalist investigating corporate fraud needs different information than a grad student studying historical trade patterns. Their research process, their information needs, and what constitutes a good answer all flow from their identity and stakes.<\/p>\n
Read more<\/p>\n","protected":false},"excerpt":{"rendered":"