A recent study led by Cornell University researchers, in collaboration with Google, evaluates whether large language models (LLMs) can function as expert-level interpreters of scientific literature. Rather than relying on standard benchmarks, the team focused on a highly specialized domain, high-temperature superconductivity, to test whether AI systems can engage with complex research at the depth required by scientists.
To conduct the study, researchers assembled a panel of 12 domain experts and evaluated six leading LLM systems, including widely used models such as ChatGPT and Claude. These systems were tasked with answering challenging, expert-designed questions derived from a curated body of scientific papers. The goal was not simple fact retrieval but assessing the models’ ability to synthesize information, reason across multiple sources, and provide well-supported explanations.
The findings reveal a mixed picture. Some models demonstrated strong capabilities in extracting relevant insights and generating coherent responses. However, none consistently achieved the level of understanding expected from human experts. In many cases, the models produced answers that appeared convincing but lacked depth, nuance, or full alignment with the scientific evidence.
A key takeaway is that performance varies significantly depending on system design. Models augmented with retrieval mechanisms, which allow them to access and reference specific scientific documents, performed better than standalone systems. These retrieval-based approaches were more likely to provide balanced, evidence-backed answers, highlighting the importance of grounding AI outputs in reliable sources.
The study also underscores the limitations of current evaluation methods. Traditional benchmarks fail to capture the complexity of real scientific reasoning, prompting the researchers to develop more rigorous, expert-driven assessment frameworks.
Overall, the results suggest that while LLMs show promise as tools for navigating scientific literature, they are not yet capable of replacing expert judgment. Instead, their near-term role lies in augmenting human researchers by accelerating literature review and supporting knowledge synthesis, rather than independently driving scientific discovery.