Researchers at Stanford conducted an evaluation of the performance of specialized legal research tools developed by LexisNexis and Thomson Reuters (Westlaw). The study scrutinized the veracity of the claims made by these providers regarding the efficacy of their AI-powered platforms, Lexis+ AI and Westlaw's Ask Practical Law AI. The findings indicate a marked reduction in errors when these tools are utilized in comparison to general AI models, such as GPT-4, highlighting a significant advancement in the field. The research documented several instances where these dedicated legal AI tools delivered accurate and comprehensive legal research. However, the study also revealed that these tools are not infallible, with the Lexis+ AI system generating incorrect information in excess of 17% of cases, and Ask Practical Law AI (Practical Law) exhibiting a higher error rate of over 34%. Such statistics underscore the need for ongoing scrutiny and refinement of AI applications in legal research.
However, on May 28, 2024, Stanford University announced it will augment their study, based upon criticism that their study did not review Westlaw’s AI-Assisted Research in Westlaw Precision. Instead, Stanford tested Practical Law, a research tool whose database is limited to content from a collection of how-to guides, templates, checklists, and practical articles, but not statutes or case law. Stanford acknowledged that Practical Law has a limited database. It will be interesting to see whether Stanford’s augmented study will reveal better efficacy of these AI-powered legal research platforms.