SciMDR: Advancing Scientific Multimodal Document Reasoning
300K question–answer pairs with reasoning chains over 20K papers, written for focused passages and then re-embedded in full documents, plus an expert-annotated benchmark.
Research
Published work from our group, ordered along the research loop.
1. Literature
Finding, reading and synthesizing papers.
SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks
An arena where researchers vote between model answers to literature questions, with over 20,000 votes on 47 models, plus a benchmark for automatic judges.
SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature
137K expert-written instructions across 54 literature tasks; fine-tuning on them improves held-out scientific tasks by 70.6% on average.
Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Papers
3,000 expert-annotated questions about the schematic diagrams in 983 papers; 18 frontier multimodal models still trail human experts.
2. Ideation
Proposing and specifying research ideas.
IdeaAnchor: Teaching LLMs to Turn Literature into Research Ideas
Mines published papers for how their ideas grew out of prior work, and uses that structure to train LLMs to turn a set of papers into a research idea.
BackTrend: Evaluating Scientific Weak-Signal Prediction via Backward Reconstruction
Can systems spot research directions before they become mainstream? 66 validated early signals behind 25 mature AI topics; the best system reaches 10.1% F1.
IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications
660 method specifications with gaps that block implementation; across 13 LLMs, finding the gap is the bottleneck, with 9.6% recovered on real-world cases.
Measuring the Gap Between Human and LLM Research Ideas
Gives LLMs the prior papers that inspired real research ideas and compares what they propose with what humans did; LLM ideas cover a narrower, shifted range.
3. Experiment design
Planning studies and designing ablations.
AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Research
2,000 expert-annotated ablation-study design tasks from 677 NLP papers, plus a meta-benchmark showing that automatic judges are unreliable for this task.
4. Discovery & reasoning
Solving scientific problems with knowledge and reasoning.
Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and Reasoning
A suite of scientific reasoning benchmarks and a probing framework that separates knowledge from reasoning; recalling the relevant knowledge is a key bottleneck.
Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving
1,297 expert-annotated university physics problems across six areas; the strongest model tested, o3-mini, reaches 59.9%.
5. Critique & review
Verifying claims, finding limitations, reviewing papers.
ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation
Learns from how authors answered real OpenReview reviews to train a reviewer that suggests concrete revisions, with a 1,000-instance human-curated benchmark.
RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation
Uses rebuttals to learn which review comments led authors to change their papers, and trains a model to write focused, actionable comments.
Can LLMs Identify Critical Limitations within Scientific Research? A Systematic Evaluation on AI Research Papers
A taxonomy of limitation types and a benchmark of synthetic and real limitations in AI papers; retrieving related literature helps LLMs find them.
SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verification
3,000 expert-annotated claims to check against the text, tables and figures of 1,113 papers; 21 multimodal models trail human experts.
6. Communication
Figures and diagrams for scientific papers.
SciSketch: An Open-source Framework for Automated Schematic Diagram Generation in Scientific Papers
An open-source framework that draws a paper's schematic overview diagram, either by planning a layout in graphics code or by generating the image directly.
Evaluation methods
How to evaluate LLM-based agents.
A Survey on Evaluation of LLM-based Agents
How LLM-based agents are evaluated, from planning and tool use to web and software agents, and where evaluation of cost, safety and robustness falls short.