AI Law - International Review of Artificial Intelligence Law
G. Giappichelli Editore

30/05/2024 - Stanford to Augment Study on AI Legal Research Tools’ Hallucinations Amid Methodology Criticisms (USA)

argument: Notizie/News - Legal Technology

According to an article on LawNext, Stanford University is set to augment its recent study which found that AI legal research tools from LexisNexis and Thomson Reuters hallucinate in 17% of queries. The study, conducted by Stanford's RegLab and the Human-Centered Artificial Intelligence research center, revealed that these tools frequently produce hallucinated results, though less than general-purpose AIs like GPT-4. Criticisms have arisen due to the study’s comparison methods, as it assessed LexisNexis’ Lexis+ AI against Thomson Reuters’ Ask Practical Law AI, not its primary AI-Assisted Research tool. Stanford's researchers noted this discrepancy, citing Thomson Reuters' refusal to provide access to the latter tool. Thomson Reuters has now granted access, prompting Stanford to update its findings. Critics argue that the study's conclusions about hallucination risks are undermined by this methodological flaw. LexisNexis defended its product, emphasizing continuous improvements and the importance of linked citations. The study calls for greater transparency and rigorous benchmarking of AI legal tools to ensure their reliability.