Legal Contract Review: Overcoming the Accuracy Hurdle in Legal AI Space

Legal Contract Review: Overcoming the Accuracy Hurdle in Legal AI Space

The legal space is embracing technology at an accelerating pace. However, there's a persistent challenge: ensuring the accuracy of AI for reviewing legal contracts. A recent Stanford University report highlighted this issue, revealing that large language models like GPT, PaLM and LLaMA  struggle with "hallucination." This means they can confidently provide incorrect answers to legal queries, posing a significant hurdle for lawyers relying on AI-powered solutions.

Despite the potential of AI in drafting and analyzing legal contracts, limitations in accuracy and citation have hampered adoption. LegalGraph AI aims to bridge this gap by offering the most accurate and efficient AI based legal tech solution.

Testing Legal AI’s accuracy on Master Services Agreements:

To validate our progress, we conducted a rigorous test of LegalGraph AI on 60 Master Services Agreements (MSAs). Each MSA included 40 critical clauses we wanted the AI to analyze.

Here's what we found:

• Speed: LegalGraph AI analyzed a contract and identified values within 40 clauses in under 8 minutes per contract. This translates to a 75% improvement in review time compared to our previous contract processing speed.

• Accuracy: LegalGraph AI achieved a remarkable average accuracy of 87% in identifying the correct values within the clauses. This represents a 10% improvement from our previous benchmark, demonstrating significant progress.

Identifying Areas for Improvement

While these results are encouraging, there's always room for improvement. Analyzing the test data revealed some areas where the AI needs further refinement. AI struggled with specific clauses related to:

• Termination: The AI had difficulty differentiating between termination for cause, without cause, and for convenience. Additionally, it had challenges understanding "opportunity to cure breach" clauses and notice periods associated with various termination scenarios.

• Price Increase: The AI faced challenges identifying clauses related to price increases and late payment charges, potentially due to variations in how these clauses are worded.

• Data Breach Notification: The AI sometimes struggled with clauses related to data breach notification, highlighting the need for further training on this critical area.

Addressing Limitations and Pushing Accuracy Above 90%

We're actively working to address these limitations and enhance LegalGraph AI's accuracy further. Our approach includes:

• Model Optimization: Ongoing model training and optimization will improve the AI's understanding of nuanced legal language and clause variations.

• Knowledge Base Development: We're creating a dedicated knowledge base specifically focused on termination clauses. This will provide the AI with a comprehensive reference point to understand the different types of termination rights and obligations.

Next Steps: Benchmarking and User Experience

Stay tuned for our next blog post! We'll be comparing LegalGraph AI's performance directly against the popular Vanilla ChatGPT-4 model, providing insights into the accuracy comparisons

Invitation for Feedback and Free Evaluation Offer

We invite you to share your thoughts on our approach to reviewing MSAs. Should we include any additional key legal terms in our evaluation? How can we assist you in benchmarking your legal contracts across these dimensions? Reviewing a contract with LegalGraph AI is now as simple as sending an email! Just send your contract to review@legalgraph.ai and our AI will analyze it automatically. You'll receive a report highlighting the top risks within the contract.

P.S. Your data is encrypted, securely stored, and never shared with any third party. You have complete control and can delete the file at any time.

Special thanks to Nitesh Sharma , Anusheel Pandey , Samyak jain , Mayank saini and Neeraj for for their invaluable contributions to the accuracy testing of the app.