Skip to content
Vesh AI

Held-out evidence

Measured on contracts it had never seen.

Founder-developed software for Vesh AI, benchmarked on 160 contracts held out from development. 960 review cells across 6 fields, scored on 12 September 2026.

Internal held-out evaluation; not an independent audit.Evidence updated 12 September 2026

Method

How the run was built.

Qwen3 4B with prompt r5, reading eight retrieved passages per review cell. Dense and BM25 retrieval are fused before the model sees anything. The model names passage labels, and Python resolves those labels to exact character ranges in the indexed source.

The contracts come from CUAD, the Contract Understanding Atticus Dataset, a public corpus of annotated commercial contracts. The 160 were held out from development: no prompt, threshold or retrieval parameter was tuned against them. Each was reviewed for 6 fields — parties, effective date, expiration date, governing law, assignment restriction and liability cap.

These results are task-, corpus-, model- and hardware-dependent. They describe this run, not performance on your contracts, which a diagnostic measures separately.

Run
review__train200__qwen-qwen3-4b__p8__r5__f6__from40
Scored
12 September 2026
Software
Private-pilot build 0.1
Dataset
CUAD (CC BY 4.0)
Split
160 contracts held out from development
Model
Qwen3 4B, run in LM Studio (MLX) on the host
Embedding
Nomic Embed Text v1.5
Hardware
Apple M4 laptop with a 10-core GPU and 16 GB of unified memory
Precision
92.2%

When the system asserts a field, how often it is right.

Cells it marked uncertain are excluded, so this is not overall accuracy.

Pass · gate 90%

Recall
81.9%

How many annotated positive fields it finds.

Some annotated fields are missed; the reviewer is there to catch them.

Pass · gate 70%

Citation validity
100.0%

Positive claims resolving to indexed source text.

No citation was fabricated. That does not mean every answer is correct, or that a citation always covers the annotated span.

Pass · gate 100%

Coverage
86.6%

Cells resolved without an uncertain result.

This is not a guarantee of complete extraction: an uncertain cell is handed to the reviewer rather than guessed.

Pass · gate 80%

Below target

Two internal gates in the same run did not pass. Their figures are shared directly with reviewers rather than published here.

Evidence overlap
How often the cited passage covers at least half of the annotated span. A citation can be valid without covering enough of it.
Metadata accuracy
How often normalised dates match the annotation. Dates are the field that most needs a reviewer's check.

Throughput

A hundred and twenty-eight minutes, start to finish.

The held-out run completed in 128 minutes of wall clock — 48.0 seconds per document, against a 60-second internal gate.

Measured on the reference Apple M4 laptop with a 10-core GPU and 16 GB of unified memory. Wall clock is sensitive to hardware and to whatever else the machine is doing, so it is a reference point rather than a guarantee. It is not an NVIDIA figure: performance on NVIDIA GPUs has not yet been benchmarked.

Review path

The reviewer stays in the loop.

  1. 01

    Ingest locally

    Text PDFs are read in place. Scans are rendered and transcribed by a local OCR sidecar.

  2. 02

    Retrieve and inspect

    Dense and BM25 retrieval are fused. A small local model reads only the selected passages.

  3. 03

    Bind the evidence

    The model names passage labels. Python resolves those labels to exact character ranges in the indexed source.

  4. 04

    Review and export

    A human accepts or corrects each cell, then exports the contract evidence matrix as CSV or JSON.

  5. 05

    Destroy on demand

    Private text, chunks, embeddings, review cells and decisions are cleared. The destruction record keeps file names, hashes, sizes and timestamps — not document contents.

Deployment

Cloud is a location. Custody stays with the client.

So far the containerised product has been demonstrated on a single machine under Docker Compose. A client-owned server or cloud tenant is what it is designed for; neither has yet been deployed for a client. Customer material is not sent to a third-party model API.

Vesh maintains development and reference environments using public or synthetic material. A client deployment would use the client's account, keys, policies and bill.

See the architecture and its status
EgressNoneCorpusIn place
YOUR BOUNDARYMODELVENDOR API
The model is brought to the material. The material holds still.
CustodyClientOperator accessWhile engaged
  • On-premiseHardware you already ownA machine in the office or the firm's own rack. Nothing to procure but the model.Demonstrated on a single machine
  • Private tenantA cloud account in your nameYour subscription, your keys, your invoice. We hold operator access only while engaged.Designed for; not yet deployed
  • Air-gappedNo route out at allFor matters where the network is the risk. Models are installed first; after that the review path needs no network.Planned; not yet validated
ResidencyIn-countryIn-regionIn-buildingIn-room
You can end the arrangement without moving a single file.

Scope of the private pilot

An MVP for supervised review, not finished SaaS.

The current release is single-user and single-tenant. Jobs and reviewer decisions are held in memory; there is no authentication, document-management integration or hosted control plane.

Those limits define the private pilot. They are not hidden behind a production claim, and the software does not replace the person who signs the work.

Request a product briefing