Benchmark Framework for Systematic Review Screening Automation

What happened

To address the time-consuming literature screening stage for systematic reviews in evidence-based research, and the susceptibility of traditional LLM evaluation metrics to skew in highly imbalanced category scenarios, a research team built SRBench: a benchmark dataset of 45,064 annotated samples covering 32 curated secondary studies. The team also released a class imbalance-aware evaluation framework, alongside PromptSR—a tool supporting prompt experimentation, experiment management, and result analysis. The suite has been validated for use in typical real-world screening scenarios.

Key facts

Core dataset

SRBench: 45,064 annotated samples, covering 32 secondary studies

Accompanying tool

PromptSR: supports prompt experimentation and result analysis for LLM screening workflows

Evaluation framework features

Adapted to match the class imbalance characteristics of systematic review screening scenarios

Use case

Literature screening for systematic reviews in evidence-based research

Background

Systematic reviews are a core foundation of evidence-based research, requiring significant manual labor to assess paper relevance during literature screening. While LLMs can drastically reduce this workload, existing evaluation methods mostly rely on traditional generic metrics that fail to adapt to the imbalanced data reality of this scenario, where the vast majority of papers are excluded from final inclusion.

Why it matters

This benchmark and tool suite built specifically for systematic review screening solves the longstanding lack of a trustworthy evaluation yardstick for LLMs in this vertical. It helps researchers more efficiently debug and tailor LLM screening prompts, reducing wasted manual effort. Ultimately, the suite accelerates evidence-based research output, providing more reliable evidence production support for all domains dependent on systematic reviews.

What to watch

Future updates to watch include the open-source roadmap for this benchmark and PromptSR tool, and performance differences across major mainstream LLMs tested on this evaluation suite.