How to Use AI Contract Review to Flag Risky Clauses

A supplier sends a renewal order form on a Friday afternoon. The terms sit in a portal nobody opened, the nonrenewal window closes in nine days, and the business owner wants an answer before Monday. Someone forwards the PDF to legal with a one-line request: can you take a look?

That request is where most AI contract review programs begin, and it's also where plenty of them stall. A tool that returns a confident summary in forty seconds feels like progress until a reviewer asks which section the liability cap came from and the answer points to a page that doesn't exist.

What follows is the process I'd run. The clauses worth flagging, the order to check them in, the vendor terms to settle before anything is uploaded, and the tests that show the tool works on your contracts rather than a vendor's demo set.

Table of Contents

Clauses to Flag and Who Verifies Each One

The economics of a first pass are already settled. A 2024 empirical study of language models on contract-review tasks found the fastest tested model completed a review for about $0.02, against an estimated $74 for a junior lawyer doing the same work, a reduction of more than 99.97% (arXiv, 2401.16212). That gap is why AI earns a place on high-volume paper such as NDAs, procurement terms and standard commercial agreements, and why it doesn't earn the final word on any of them.

The list below does two jobs. It tells the tool what to hunt for, and it tells you who signs off when the tool finds it. Risk tier sets the review path, not the wording of the clause. A medium-tier payment term with clumsy drafting still goes to a finance owner with a spot check. A high-tier liability cap with immaculate drafting still goes to a lawyer.

Clause type What the tool should find Risk tier Who verifies
Limitation of liability Cap amount against fees paid, excluded damages, carve-outs High Lawyer sign-off
Indemnity Mutual or one-way scope, uncapped indemnities, third-party claims High Lawyer sign-off
Termination and renewal Notice period, termination for convenience, auto-renewal window High Lawyer sign-off
Governing law and disputes Jurisdiction, arbitration seat, jury waiver High Lawyer sign-off
Data processing and privacy Subprocessors, transfers outside the EU, deletion duties High Lawyer with privacy lead
Assignment and change of control Consent requirement, triggers on acquisition Medium Lawyer review
Payment and price changes Payment days, late fees, annual price escalators Medium Finance owner with spot check
Confidentiality Duration, scope of definition, exceptions Medium Business owner with spot check
Notice and contact details Addresses, notice methods, deemed receipt Low Spot check

Ask for missing clauses as their own finding class. An absent liability cap is a problem in a different category from a weak one, and it needs a different conversation, so the tool should return a missing status instead of silence. Silence reads as compliance to a reviewer skimming at 5pm.

Operational rule: anything the playbook tiers as high goes to a lawyer before it's accepted, negotiated or relied on. No confidence score overrides that.

If your supplier paper runs through procurement rather than legal, the guide to streamlining procurement with AI contract review covers how contract intelligence fits the buying cycle and where the handoff to counsel belongs.

Privacy and Governance Terms to Check First

Settle the obligations that apply to you as the user of a system, not just the promises on the vendor's page. Under Article 26 of the EU AI Act, deployers of high-risk AI systems must use them according to the provider's instructions, monitor their operation, manage input data, report relevant risks or incidents, and retain logs for at least six months. The same article requires you to assign human oversight to people with the competence, training, authority and support to intervene when the system misreads context (EU AI Act Service Desk).

That oversight requirement is the one that changes your org chart: someone named, trained and resourced is accountable for output a machine produced.

A professional woman reviewing an AI contract checklist with focus on privacy and governance compliance.

Whether a contract-review tool counts as high-risk depends on how you deploy it and what you automate, so put that question to your legal team early rather than at procurement signature. Some AI Act obligations were already applying from 2 February 2025, and the European Commission's regulatory framework guidance frames the requirements around risk management, data quality, documentation, transparency, human oversight, accuracy, cybersecurity and resilience.

Terms to get in writing

  • Processor status: for business clients acting as controllers, we act as processor under a data processing agreement accepted at subscription. Read the data processing agreement before you upload anything.
  • Subprocessor changes: we give at least thirty days' notice before adding or replacing a subprocessor, and you have thirty days after that notice to object.
  • Model provider retention: we hold zero data retention agreements with every AI model provider we work with, and conversations are not training data. That's a contractual position, not a marketing line.
  • Everything outside the training promise: ask about transient processing, cross-border transfers, incident notification, deletion you control, and whether uploaded exhibits and file metadata are kept after the review closes.

Then ask which model handled which step. When one model extracts a clause, another interprets it and an agent pulls external sources, accountability breaks unless you can trace the chain from input to conclusion.

Build a Playbook Before You Upload Anything

A playbook is one page. It states your standard position, your fallback and the line you won't cross, and it travels with every review request so the tool and the reviewer work from identical rules.

  1. Pick the contract types you see. Ten to fifteen, covering NDAs, supplier agreements, SaaS terms, reseller paper and statements of work. Anything outside the list takes a different path.
  2. Write a standard position for every clause in the table above. A liability cap equal to twelve months of fees. Mutual indemnity. Thirty days' notice to terminate. Concrete enough that a tool can mark a clause compliant or deviating without interpretation.
  3. Add a fallback and a walk-away line so a negotiator can move without calling a lawyer at 9pm the night before signature.
  4. Set a confidence threshold per tier. Every high-tier finding goes to counsel regardless of score. Any medium-tier finding below 0.7 goes to a spot check. Low-tier findings get sampled.
  5. Record the jurisdiction each contract type usually falls under. Governing law language that's fine in one country creates real exposure in another.
  6. Pull 40 to 60 signed contracts from the last two years. They become the test set for the accuracy work later in this guide.
  7. Name three roles. The contract owner who submits documents, the reviewing lawyer who approves findings, the privacy lead who checks data terms.

Keep the playbook under version control. Every finding should trace to the rule version that produced it, because a bad result after an edit is a playbook problem at least as often as a model problem.

The underlying task explains why the discipline matters. A 2025 survey of automated legal-contract classification drew on 60,540 publicly filed contracts from 2016 through 2019 and 846,274 individual provisions. The unit of work is the clause, not the document. Your playbook is what decides which clauses.

Seven Steps to Run an AI Contract Review

Order matters more than the model. Run the same seven steps on every agreement and the output becomes a queue a reviewer can clear in one sitting.

The sequence below is the one I'd document for a team, because each step removes a specific failure that shows up repeatedly in pilots.

A diagram illustrating the seven-step process for conducting an effective AI-powered contract review, from conversion to final handoff.

  1. Convert to text without losing location. Keep page, section and table numbers intact. A finding without a location can't be checked quickly, and a reviewer who has to hunt for a clause stops trusting the tool by the third document. Run OCR on scanned pages and compare a sample against the original image.
  2. Split the agreement and extract against a fixed schema. Sections first, then the provision list: indemnity, limitation of liability, assignment, governing law, renewal, data processing. Fixed schema, every time, so results are comparable across contracts.
  3. Compare each provision with your playbook, not with market norms. A generic note that a clause is unusual is noise. The only question that matters is whether it matches your position.
  4. Demand evidence and status together. For each playbook clause, ask for the location, a short verbatim quote, a status of compliant, deviation, missing or unclear, and a confidence score, returned as structured fields. A prompt that requests location, quoted text, status and confidence as JSON beats an instruction to review the contract, every time. Structure pays: a WorldCC evaluation of large language models in contract review found structured reviewer guidance improved accuracy by roughly 10 to 15% across tested models compared with simple question-based prompting, with hallucination rates for foundational fields converging below 1% under those conditions.
  5. Run a challenge pass. A second request that lists which findings lack a supporting quote and why. This catches conclusions the model inferred from a summary instead of the text.
  6. Route by tier and confidence. High-tier or low-confidence findings go to counsel. The rest land in a reviewer queue with evidence attached.
  7. Log every decision. Approve, reject or amend each finding with initials and a date, in the log described next. An undocumented approval is unverifiable three months later.

When the agreement arrives as a set, master agreement plus order form plus an invoice sitting in a spreadsheet, document analysis that reads PDF, DOCX and XLSX files in one case removes the reconciliation step. And if you're setting positions without in-house counsel, contract review guidance for Washington startups covers how to decide them before the tool sees a document.

The most common mistake isn't a hallucinated clause. It's treating a clause that exists as a clause that's acceptable. Presence and wording are separate checks, and liability lives in the second one.

Test Accuracy on Your Own Contracts

A single accuracy percentage is the least useful number a vendor can hand you. It collapses a dozen clause types with wildly different failure costs into one figure, and it says nothing about whether the tool will miss the indemnity buried on page 34 while scoring well on boilerplate.

Prove it on your own paper instead:

  1. Use the 40 to 60 signed contracts you gathered for the playbook. They're representative, already reviewed, and safe to test with.
  2. Have two lawyers annotate each clause independently, then a third person adjudicate disagreements. The adjudicated set is your ground truth.
  3. Measure per clause type. Precision and recall for indemnity, for liability caps, for renewal terms, reported separately. An average hides the clause that matters.
  4. Report missed high-risk clauses as their own safety metric. A tool with strong averages and one silent liability miss isn't ready for unassisted use.
  5. Break the test set on purpose. Altered monetary thresholds, negated obligations, exceptions appended to the end of a clause, swapped defined terms, conflicting schedules, scanned pages, clauses drafted in a second language.

The recall trade-off is the one most pilots get wrong. Push for every possible issue and reviewers drown in false alarms, then start rubber-stamping to clear the queue. The CUAD benchmark shows the shape of that cliff: on more than 500 expert-labeled contracts with 41 clause categories and over 13,000 annotations, a DeBERTa model reached 44.0% precision at 80% recall and only 17.8% precision at 90% recall (CUAD clause extraction analysis). Set the highest recall target on high-tier clauses and accept lower precision on administrative ones.

Published benchmarks are context, not proof. A summary of the LawGeex study reports AI reaching about 94% accuracy identifying risks in nondisclosure agreements against about 85% for experienced lawyers (Zip summary of the LawGeex study). That's a task-specific figure on one contract type. It justifies a first pass on NDAs. It says nothing about your supplier agreement.

Metric What it tells you When to check
Recall on high-risk clauses Share of real issues the tool found Before any rollout, highest priority
Precision by clause type Share of flags that were real issues Per clause type, to size the review burden
Missed critical clauses Count of high-risk clauses the tool never flagged Reported separately as a safety metric
Evidence accuracy Share of quoted spans that match the contract text Every test run
Drift after changes Change in results after a model or prompt update Each time the model or playbook changes

Record each adversarial result beside the issue log, tied to the playbook version and prompt used, so a failure can be traced to the rule that caused it rather than blamed on the model in general.

Organize Flagged Issues for Human Verification

Raw model output isn't a work product. Turn it into a log a reviewer can clear in one sitting and the review becomes auditable. Leave it as a chat transcript and nobody can reconstruct what was checked, by which tool, against which rule.

Validation effort should scale with the consequence of a miss, not with the length of the document. That's the distinction between spot-checking a notice address and reading a limitation-of-liability clause word by word. A Chambers practice guide on AI in legal work frames the same split, with lightweight checks on administrative extraction and clause-level attorney verification for governing law, indemnity, liability, termination, data processing and change of control.

Here's the log worth building. Six fields, one row per finding.

A professional analyzing contract documents with AI software to flag issues and verify legal compliance efficiently.

Log field What to record Example entry
Clause location Section, page and table reference Section 8.2, page 4, table 2
Evidence span Short verbatim quote from the contract One or two sentences copied exactly
Status Compliant, deviation, missing or unclear Deviation
Risk tier High, medium or low from the playbook High
Model confidence Score or low-confidence flag Low
Reviewer decision Approve, reject or amend, with initials and date Amended, JM, 14 March 2026

Sort by risk tier first, then by clause type, so high-tier deviations reach a lawyer before anything else. Keep missing clauses in their own group. A tool that reports an absence may be right about the absence and still need a human to check whether the obligation sits in a schedule or in incorporated online terms. Merge duplicate flags when the same issue appears in the body and again in an exhibit, or the reviewer counts one problem twice.

Link definitions and cross-references into the finding. An obligation that depends on a defined term in an appendix is where general-purpose models fail most.

Then give the reviewer three choices per finding: approve, reject, amend. Keep the tool's original output unchanged in the record so an audit later can see what the model said and what the human changed. The habit of checking AI output against its source is the same one here, applied clause by clause.

Some working rules for the log itself:

  • Store it beside the source file so the evidence travels with the contract.
  • Date and initial every decision. Undated approvals can't be verified later.
  • Require a second initial on amended clauses. The person who changed the wording shouldn't be the only person who approved the change.
  • If your contracts hang off bids, the proposal compliance matrix approach maps tender requirements the same way, one requirement per row with an owner and an evidence reference.

Choose Your Review Path Before the Next Contract

Three paths exist, and many groups require all of them. The key decision is not which tool to purchase. It is which path a given document takes before anyone opens a laptop.

Adoption pressure is already here. 30% of lawyers surveyed by the American Bar Association in 2024 reported using AI, reaching 46% at firms with 100 or more attorneys (ABA Journal). A LegalOn survey that year found 8% of legal professionals saying their teams already used AI for contract review while more than 70% were actively considering it (Legal Dive).

Review path Best for Risk tier Lawyer role
Manual review Rare or unusual agreements Any, low volume Lawyer reads every clause
AI triage with lawyer review Standard NDAs and supplier terms Low to medium Lawyer checks flagged issues and high-tier items
AI-assisted redlines Liability, IP, privacy and termination terms High Lawyer signs every change

Pick by volume, by risk tier, and by how closely your playbook matches your actual standard positions. A playbook that's six months stale pushes everything toward manual review, and the answer there is to fix the playbook rather than to trust the tool more.

The next step is small and specific: run a 20-contract pilot this month using the playbook and the issue log, then compare those results with the annotated test set. That gives you a precision and recall picture for your own paper before you commit to a rollout.

Where the agreement and its supporting documents need to be read together, BromeAI's workspace analyzes PDF, DOCX and XLSX files in one case, so the order form and the invoice can sit alongside the master agreement rather than in a separate tab. Before loading client material, confirm the privacy terms from the earlier gate in writing rather than taking this guide's word for it. One of ours belongs in that conversation: conversations are not training data, and that's a contractual position rather than a marketing claim.

The output of any review is a human decision. The tool's job ends where someone with authority signs.

BromeAI gives your team one workspace for contract work, with model routing, agentic multi-step tasks and document analysis that reads PDF, DOCX and XLSX files together in a single case, so a supplier agreement, its order form and its invoice can be reviewed side by side. Conversations and files are excluded from model training, and zero data retention agreements are in place with every model provider. Start with one pilot contract and visit BromeAI to see how the flagged issues land in a log you can verify.