Skip to content

okf-ocr

Process scanned or text-less documents in OKF matter folders. Use when files appear in 98_Inbox_ToProcess, when OCR log rows are PENDING, when the user mentions OCR, scanning, digitising documents, extracting tables, or processing the inbox in an OKF matter. Also maps dense documents (expert reports, FRAEW, adjudications) into a verified section index + per-section map (tools/dense_doc_map.py) so the README/Chronology can point into the document instead of at it. Transcribes verbatim via the pipeline (tools/ocr_pipeline.py — PyMuPDF text layer + Gemini vision for scans with PaddleOCR offline fallback + pdfplumber tables + pandas), tags files _OCR/_SRC, updates 97_OCR_Log, the matter README, matter.yaml and Chronology. Self-learning: appends new lessons to learnings.md.

Source: .opencode/skills/okf-ocr/SKILL.md — site rebuilt 2026-09-05.

OKF OCR & Extraction

When to run

  • Files sit in a matter's 98_Inbox_ToProcess (Contentious_Template or NonContentious_Template layout).
  • The user asks to OCR, digitise, or extract tables from documents.
  • 97_OCR_Log\OCR_Log.csv contains PENDING rows whose files are already filed in matter folders (inbox may be empty — process PENDING items in place).
  • A matter has a dense document (expert report, FRAEW, adjudication decision, long advice) that the README/Chronology must reference — map it so references point at sections, not the whole document.

Tooling

tools/ocr_pipeline.py in this skill folder does ALL transcription work — engine precedence:

  • Fast classifier (v1.10, optional, diagnostic) — pdf-inspector (Firecrawl, Rust, pip install pdf-inspector) classifies every PDF text_based/scanned/image_based/mixed in milliseconds and cross-checks the PyMuPDF text layer. Divergence → warnings only; the deterministic text layer stays authoritative for routing. Result goes to _pipeline.json, _V1_OCR_meta.json (classify:), and OCR_Log notes (classify: <type>). Disable with --no-classify.

  • Office files (v1.10, optional) — anydoc (Firecrawl, Rust, pip install firecrawl-anydoc) converts .doc/.docx/.ppt/.xls/.rtf/.epub/... to a clean Markdown <base>_V1_OCR.txt sidecar in Phase 1 (free, local, ~10 ms). Skipped with a note when not installed. anydoc's hosted OCR fallback sends scans to Firecrawl — never route client scans through it; Gemini/PaddleOCR stay the scan engines.

  • Text layer — PyMuPDF get_text() per page (instant, exact for digital PDFs)

  • Scans (default) — Gemini vision API (gemini-3.5-flash-lite), verbatim transcription with {..}-braced uncertainty spans → [LOW CONFIDENCE] / [FIGURE UNVERIFIED] markers. Page batching (v1.8): scan pages go to Gemini in ONE request per batch of --gemini-batch N pages (default 20; user-tested sweet spot 20-40) with --- PAGE n --- markers; a whole multi-page file costs ~1 request. Missing pages (truncation) are silently re-batched at half size; pages that still fail fall to the next engine. --gemini-batch 1 restores the old one-page-per-request behaviour. Daily budget guard: Google free tier caps requests per Cloud project (flash-lite 15 RPM / 1,000 RPD, resets midnight PT) — every Gemini request bumps a shared counter (%TEMP%\okf_gemini_daily.json, all processes/matters); once the daily cap is spent, the chain skips Gemini (auto falls to Muse/Qwen/Mimo) instead of failing. --engine text is for text-layer-only runs and skips scans.
  • Scans (Muse XHigh — highest quality) — Muse Spark 1.2 XHigh via the opencode-go gateway (--engine muse, OpenAI-compatible https://opencode.ai/zen/go/v1, model muse-spark-1.2-contributor alias muse-xhigh, key from auth opencode-go.key/MUSE_API_KEY). Same {..}-brace tagging as Gemini/Qwen/Mimo. Requires workspace opt-in for data collection at https://opencode.ai/workspace/wrk_01KZ9PBDY4PH0VY9KR7K20X140/go (until opted in the engine returns DataPolicyError and the chain falls through soft — check ocr_pipeline.py:1 header). The auto chain is now gemini → muse → qwen → mimo → mistral → paddle (Muse sits directly after Gemini as the XHigh quality engine).
  • Scans (fast path) — Qwen3.8 Max via the opencode-go gateway (--engine qwen, OpenAI-compatible https://opencode.ai/zen/go/v1, model qwen3.8-max, key from auth opencode-go.key/QWEN_API_KEY). Much faster than Gemini on photo/scan batches; same {..}-brace tagging.
  • Scans (gateway fallback, preferred over Gemini free/Mistral) — Mimo v2.5 via the opencode-go gateway: mimo-v2.5 is FREE (flash; PNG images only — the pipeline auto-converts JPG/TIF via PIL). OpenAI-compatible on the same endpoint as qwen/muse (key from auth opencode-go.key/MIMO_API_KEY), same {..}-brace tagging. When vision has issues (Gemini free tier 429ing/recitation, Mistral slow or delayed), fall to muse/qwen/mimo instead of waiting — do not rely on Gemini free or Mistral under delay. The auto/vision/qwen/mimo/muse chains put muse after gemini, mimo between qwen and mistral; --engine muse forces Muse XHigh then falls to qwen/mimo/mistral/paddle. COST RULE: the paid tier kimi-k3 is NOT in any chain and never escalates automatically — mass OCR must not land on a paid model. It is explicit opt-in only, via --mimo-model kimi-k3 on a targeted single-file run. (mimo-v2.5-pro is TEXT-ONLY on the gateway — 404 no image endpoints.) Gateway requests send a browser-like User-Agent (Cloudflare 403/error-1010 blocks urllib's default one).
  • API concurrency gate (v1.7) — --max-concurrency N caps simultaneous API calls (Gemini/Muse/Qwen/Mimo/Mistral combined) ACROSS all pipeline processes, not just one: a token pool guarded by an OS file lock (%TEMP%\okf_ocr_api_gate.lock, msvcrt on Windows / fcntl elsewhere, tokens expire after 10 min so a crashed process can't drain the pool). Default 2 (OKF_MAX_API_CONCURRENCY env overrides; 0 = unlimited). This is the backstop that makes parallel matter agents safe on the opencode-go quota — keep it at 2 unless the user raises the matter cap too. Single-process runs are unaffected (they were serial anyway).
  • Low-confidence auto-resolution (v1.4) — when a page has any flagged line, the pipeline automatically re-reads the WHOLE page at 1.5× DPI with the next available engine (prefers muse/mimo/qwen over Gemini-free/Mistral — see delay policy) and line-reconciles: flagged lines confirmed cleanly by the alt engine get [RESOLVED: <engine>@<dpi>dpi] and stop flagging; lines the alt engine reads DIFFERENTLY get [ALT: ...] but stay flagged; lines BOTH engines doubt stay flagged. A second disagreement pass checks the model's confident lines against the alt read: content-bearing lines (≥6 chars) the engines read differently (0.65–0.99 similarity) get [CONFLICT: <engine> reads "..."] — this catches silent errors like 3MT vs OMT, UG467 vs UG487 that the model never flagged. [CONFLICT] lines keep the page PARTIAL. Disable with --no-resolve; resolutions are reported in the page summary (mistral@300dpi (2 resolved, 0 ALT, 1 CONFLICT)) and in the OCR_Log notes. Engine output varies run-to-run (e.g. a bad Gemini pass that silently swapped product codes), so [CONFLICT]/[LOW CONFIDENCE] residuals are never auto-cleared — a human/agent decides them.
  • Scans (fallback 1) — Mistral OCR (mistral-ocr-latest), ~2–5 s/page, no safety filter — reliably transcribes the boilerplate pages Gemini refuses; tools/mistral_ocr_pipeline.py is a wrapper forcing --engine mistral. Last-resort-only when Gemini/Mimo/Qwen all fail or are delayed — do not default to Mistral.
  • Scans (fallback 2) — PaddleOCR (PP-OCRv6, enable_mkldnn=False — the oneDNN flag avoids a runtime crash), ~50 s/page, per-line scores → [LOW CONFIDENCE n.nnn] markers
  • Tables — pdfplumber extract_tables() (deterministic, structure-exact)
  • Table vision-QA (safety net, ON by default) — table_qa_pass(): for every page with tables (cap --table-qa-max, default 10), renders the table bbox crop and re-transcribes with a vision model (gemini → qwen → mimo → mistral), then NUMBER-ANCHORED verification: every figure pdfplumber extracted must be confirmed by the SIDECAR page text (exact text layer on digital pages) or the vision transcription — substring match against digit-collapsed text (spaced glyphs 0 0 0 2 == 0002). Verdicts: PASS (all figures confirmed) / PARTIAL (unconfirmed figures listed; confirmation-rate <60% on dense CAD grid-glyph tables — pdfplumber's layout-aware extraction is authoritative there) / UNVERIFIED (vision verifier failed but page has a text layer — extraction still exact) / FAIL (no text layer AND figures unconfirmable, or vision error on a scan page). Row-count mismatches are scope noise (vision reads headers/letterhead in the crop as rows), never a failure. Writes <base>_V1_OCR_table_qa.csv next to the source + table QA: summary line in OCR_Log notes. Report-only — nothing auto-corrected. Disable with --no-table-qa. This is the automated half of the "tables always double-checked" rule; the model still spot-checks 1-2 invoice sums + bank balance chains per matter.
  • Table QA (vision, ON by default) — every page with tables (cap 10 pages/file, --table-qa-max N) is bbox-cropped and re-transcribed by a vision model (gemini→qwen→mistral), then number-anchored compared against pdfplumber: every figure pdfplumber extracted must appear in the vision read. Verdict PASS / PARTIAL (<3 missing figures) / FAIL (≥3 missing or vision error); row-count mismatch is scope noise (vision reads letterhead/headers as rows) and is NOT a failure. Writes <base>_V1_OCR_table_qa.csv next to the source + a table QA: summary line in the OCR_Log notes. Skips re-run when the CSV exists; --force redoes. Report-only — nothing auto-corrected. Disable with --no-table-qa. This catches silent figure swaps/column shifts pandas never sees.
  • Financial tables cross-check — tools/table_vision_check.py <matter_root>: for invoice/fee-note PDFs, vision reads line-item amounts + stated totals and cross-checks against the tables CSV (line-item count parity, sum-of-items vs stated totals, vision vs pdfplumber agreement). Verdict MATCH/FLAG/NO-TABLES → 00_Admin\table_vision_check.csv. Run it when a matter has invoices/fee notes with arithmetic flags.
  • Arithmetic — pandas column-sum vs stated total, auto-flags [ARITHMETIC ERROR: ...] / [TOTAL OK: ...] in the CSV
  • Outputs — <name>_V1_OCR.txt sidecar next to source + <name>_V1_OCR_tables.csv in the matter's Extracted_Data folder + <name>_V1_OCR_meta.json provenance sidecar (producer/creator, creation date, digital-vs-scan origin, text-layer coverage, image counts, DPI hints, portfolio embedded-file count; EXIF for images — GPS as presence only, never coordinates) + <name>_pipeline.json machine-readable summary
  • OCR_Log auto-update — one row per processed file is upserted into the matter's 97_OCR_Log\OCR_Log.csv keyed on (File, Destination): created if absent, replaced on re-runs (no duplicates). Quality = qa overall_flag, tables flag PASS/NO, notes list page methods + sidecar + a meta: provenance line (e.g. meta: Aspose.Words 23.6.0 | created 20241001 | pure scan (0/45 text pages)). Disable with --no-log. The log folder is found via --matter or by walking up from the target until 97_OCR_Log is found.

tools/apply_schema.py applies the OKF folder schema to a flat matter dump: - --skeleton — create the template tree (contentious/noncontentious, auto detects 02_Pleadings vs 02_Contract_Documents) - --scan --client <kw> --opponent <kw> --third <kw> — classify every file → moves.csv (no changes); counterparty keywords route email stubs by subject; content keywords (invoice/agreement/plan/certificate/...) route documents; identical-size small image pairs → decorations; UNCLASSIFIED items printed for human review. Non-contentious construction rules added 2026-08-05: SOA/redline/risk-analysis → 02_Contract_Documents\04_Amendments_Variations, collateral warranties + licence agreements → same folder, bonds/guarantees/PCG/NBIT → 04_Pre_Contract\03_Bonds_Guarantees_Insurance, professional appointments/fee proposals → 03_Professional_Appointments, tender packs/schedule of works → 04_Pre_Contract\01_Tenders_Proposals, front-end info/annexure/site plan → 01_Main_Contract, communications logs + .html working files → 00_Admin. email from X to Y on DATE.pdf portfolio exports are treated as email (counterparty keywords match party names in the filename). - --apply [--purge-decorations] — perform the moves (refuses to overwrite existing files) - --dedup [--apply] — sha256 byte-identical captures → consolidated to one copy (dupes to _thread_parts\ on apply); reports version families (draft/final/signed/Vn/dated copies of the same document) to 97_OCR_Log\version_families.csv — name them as ONE version chain at filing time, but note a family may still contain DISTINCT documents after (N)-stripping (review before renaming). Run BEFORE --scan. - --init --ref <no> --client <n> --opponent <n> — create 97_OCR_Log\OCR_Log.csv header, README.md, matter.yaml stubs if missing. Template-aware since 2026-08-05: non-contentious matters get the construction README header (parties, project, contract form, contract sum, key dates) + construction legend + matter.yaml with contract_form/contract_sum/professional_team — no longer inherits the contentious (02_Pleadings) legend. - --report — missing-evidence warnings (zero-byte .gdoc stubs, decoration candidates, generic-named images) - --email-threads [--apply] — email chains: clio exports the SAME thread many times (13+ captures of one subject is normal). Groups by normalised subject (strips Parsed Email -, export timestamps, Re_/Fw_/RE_/Fwd_/Automatic_reply, punctuation), merges single-member groups into multi-member groups by substring, keeps the latest stub capture + every real file (distinct correspondence), consolidates duplicate captures (and identical sha256 content) into <folder>\_thread_parts\ on --apply. Writes 97_OCR_Log\email_threads.csv thread map. Run on every new batch before filing. - --check — warn if new/unprocessed files were added since the agent last ran (unfiled root files, files with no OCR_Log row and no sidecar, non-empty 98_Inbox_ToProcess); exit 1 if anything found. 00_Admin is never checked (working files). Run this whenever the user says "new files" / "files added" — then process them via the pipeline. - tools/dense_doc_map.py — map dense documents (expert reports, FRAEW, adjudications): parses the document's own ToC into a canonical section list, locates each section's start page in the body text layer, and emits <base>_V1_OCR_index.md (mechanical, page numbers verified) + <base>_V1_OCR_map.md (AI-fill stub: per-section Substance / Key holdings / Chronology relevance, all [UNVERIFIED] until confirmed against the verbatim source). ToC-less docs fall back to single-line numbered titles (level 1-2 only). Run it on any dense document the README/Chronology must reference; README points to the map, map points to sections, sections point to verbatim pages. - tools/commslog_archive.py — MANDATORY for every matter with a Master Communications Log: convert the clio log (plus any 00_Admin\*Daily Communications Log*.txt, same format — merged chronologically, overlap removed by subject+body dedup) into 00_Admin\Email_Archive\: per-thread folders of individual-message .md files. Nested email chains are unpacked into separate messages, boilerplate segments (Mimecast banners, signatures) merged into parents, duplicates removed (subject+body hash), threads named from 97_OCR_Log\email_threads.csv. Nomenclature: thread folder + NNN ordinal — every message has a stable ID thread/NNN (0001-style zero-padded chronological position; ties by body hash), filename NNN_YYYYMMDD_Sender_Subject.md, a plain-text _V1_OCR.txt sidecar per message (written automatically, same convention as OCR sidecars — 1:1 with the .md, so every archived message is uniformly searchable), per-thread index.md table, and stub_index.md mapping every .gdoc stub to thread folder + message range. Every sidecar ends with an Attachments: field — real name+size when the source carries a manifest (.eml/.msg/portfolio embeds), otherwise (no manifest in source log). A message quoted in other chains exists once, at its origin position. .gdoc stubs themselves are unreadable Drive pointers while GDFS isn't streaming — but their content lives in the Master/Daily Communications Logs, so the archive is complete without Drive access; the logs stay the verbatim source. Run: python tools/commslog_archive.py <matter_root> (re-runs clear the output folder first). - tools/eml_split.py — message-level email decomposition for every .eml/.msg: splits each chain into its component messages → <source>_split\NNN_YYYYMMDD_Sender_Subject.md (one file per unique message, dedup by subject+body hash, per-source index.md), extracts attachments (sha-deduped, _N suffix on collisions) to 03_Productions\06_Originals\_eml_attachments\. Tags per message: ID, From, To, Cc, Sent, Subject, Attachments, Attachments extracted, Sensitivity, Source (+ Archive ID cross-ref when the subject matches an Email_Archive message). Sensitivity auto-detected from body keywords (without prejudice / privileged / confidential / legal advice). Run AFTER building the Email_Archive; then cross-source dedup: collapse (subject, body) dupes across ALL _split dirs (the same message is quoted in every chain capture that contains it) — keep the canonical copy (longest source-dir name), record deletions in 00_Admin\eml_split_dedup_manifest.csv, and REGENERATE the affected index.md files. Run: python tools/eml_split.py <matter_root>. - tools/native_structure.py — append a [DOCUMENT OUTLINE] appendix to sidecars from native source structure: PDF fitz.get_toc() (fallback heading-scan), DOCX Heading 1-3 styles, XLSX sheet names + dimensions. Idempotent (skips sidecars already containing the marker); skips admin/working files and .gdoc pointers. Run in Phase 1 for born-digital docs so every sidecar carries its structural outline. - tools/table_vision_check.py — vision cross-check of extracted financial tables (invoices, fee notes, outlays, bills). Renders the table pages and asks a vision model (Mistral by default — markdown tables natively; Gemini fallback) to read line items + labeled totals, then cross-checks against the pdfplumber extraction: line-item parity, totals agreement (per-invoice Total preferred over account-level Outstanding Balance/Total Amount Outstanding, subtotal lines excluded, adjacent label/amount lines handled). Verdict per file MATCH/FLAG → 00_Admin\table_vision_check.csv. Catches what the pandas column-sum auto-flag misses: pdfplumber empty-table extractions (McAuley Outlays: full ledger + £11,934.32 recovered; invoice_3497: true £54,836.28 vs artifact 295), and account-vs-invoice total confusion (invoice_3446 £3,481.20 balance vs £2,348.40 invoice total). Run in Phase 2b on all financial PDFs. - README stubs from --init carry a NEW FILES banner reminding to run the agent when files are added.

The Gemini key is picked up automatically from opencode's auth store (~/.local/share/opencode/auth.json → google.key) or GEMINI_API_KEY; the Mistral key from the same file (mistral.key) or MISTRAL_API_KEY — no extra setup. Without keys the pipeline falls back to PaddleOCR (--engine paddle forces it; --engine mistral forces Mistral).

Key behaviours: - Resume-safe — the sidecar is rewritten after EVERY page and re-runs skip pages already transcribed (--pages 1-7 chunked runs merge with existing sections instead of replacing them) - Never destructive — an existing good page section survives even if a re-run fails on that page (--force re-does everything) - RECITATION guard — if Gemini's safety filter blocks a page (it refuses verbatim copies of standard legal boilerplate), the pipeline retries a rephrased prompt, then Mistral OCR, then PaddleOCR for that page automatically - Google Drive quirk — sidecars on H: are written in place, NOT via temp-file+rename (GDFS reverts os.replace())

Run it per file or per folder: python <skill>/tools/ocr_pipeline.py <target> --matter <matter_root> --json <out.json> [--engine auto|vision|muse|qwen|mimo|mistral|paddle|text] [--pages N-M] [--force] [--no-resolve] [--no-table-qa] [--table-qa-max N] [--max-concurrency N] [--muse-model muse-spark-1.2-contributor] [--mimo-model mimo-v2.5]. The paid kimi-k3 is explicit opt-in only (--mimo-model kimi-k3) for a targeted single-file run — never in the default chains. Muse requires workspace opt-in (see above).

The model's job is: triage, run the pipeline, QA the output against the source (vision spot-check if available), file, log, maintain knowledge links, learn.

Order matters: dedup → text-extract → vision → emails → file

Canonical order (see Workflow): Phase 0 dedup/decoration cleanup FIRST — run --dedup --apply (sha256 byte-identical captures → _thread_parts\; reports version families draft/final/signed/Vn/dated to 97_OCR_Log\version_families.csv, name as ONE version chain at filing; a family may still contain DISTINCT documents after (N)-stripping — review before renaming) so the scan plan is smaller and nothing downstream is wasted → Phase 1 born-digital text extraction + quality assessment (free, offline, exact — the bulk of any corpus) → Phase 2 vision OCR only for the no-text remainder (the expensive tail) → Phase 2b post-OCR quality check (batch-level verification: cross-engine no-text verdicts, CONFLICT/ALT artifact triage, residuals never auto-cleared, arithmetic-artifact recognition, case-ref discrepancy flagging) → Phase 3 email layer (.eml/.msg conversion, content dedup, --email-threads --apply BEFORE --scan; since 2026-08-05 --email-threads --apply is deliberately decoupled from the plain --apply branch — it will NOT apply moves.csv anymore, it used to, and an unreviewed plan applied 840 moves before review on 01403) → Phase 4 file/consolidate (--scan, review moves.csv, then --apply (or --scan --apply after review), sidecars follow sources, log/README/yaml/chronology once). The Email_Archive build (commslog_archive.py) is MANDATORY in Phase 3 for every matter with a Master Communications Log — one .md sidecar per unique message, thread/NNN IDs, merged with the Daily Communications Logs; the archive is the reviewable email evidence layer and the matter is incomplete without it.

Self-learning

The skill remembers what it learns. learnings.md sits next to this SKILL.md and is the skill's memory.

  • Before a batch: read learnings.md and apply known quirks (tool behaviors, file-type gotchas, engine bugs).
  • After a batch: append any NEW lessons to learnings.md — one dated bullet each. Never delete old entries.
  • What to record: file-type quirks (e.g. .gdoc stubs, .msg output layout, PDF portfolio wrappers), engine behaviors (PaddleOCR flags, pdfplumber table catches), model limitations, environment gotchas, efficiency techniques.
  • If a lesson saved real time or prevented an error, it MUST be recorded — that is how the skill improves.

Parallel multi-matter batches (the list workflow)

When the user hands a list of matters to work through (refs, folders, or inboxes), treat it as a queue processed by parallel workers — never one big sequential slog, never more than the quota allows.

Concurrency rules

  • MAX_PARALLEL_MATTERS = 2 (default) — at most 2 matters in flight at once. Every API call (Gemini/Qwen/Mistral) shares one quota and the opencode-go gateway rate-limits (HTTP 429) are the binding constraint; the pipeline's --max-concurrency gate (default 2) is the hard backstop. Tune both numbers together, and only if the user says the plan allows more.
  • One okf-worker subagent per matter (Task tool — the agent is defined at .opencode/agent/okf-worker.md). Launch the first 2 workers in the SAME message (multiple Task calls in one response run in parallel; calling, waiting, then calling again is sequential and defeats the purpose). As each finishes, launch the next queued matter.
  • Never two agents on the same matter; one writer per file. Workers own their matter folder; the main agent owns the skill folder, learnings.md, and any batch-level files. Workers must never edit the skill folder — tools/SKILL.md are read-only for them (parallel-writer hazard: a second writer silently overwrites edits; see learnings 2026-08-06).

Worker brief (per matter)

The okf-worker agent file carries the standing rules; the main agent adds per-matter specifics:

  1. Matter folder path + template type (contentious / non-contentious) + ref; any user scope limits (e.g. "structure + core OCR only", "skip the giant bundles").
  2. Worker must read SKILL.md + learnings.md FIRST and follow the canonical phases 0→4 in order.
  3. Every pipeline call runs under the shared gate (--max-concurrency 2 or the OKF_MAX_API_CONCURRENCY env default — do not pass a higher number). Prefer --engine muse (XHigh quality, requires opt-in) or --engine qwen/--engine mimo for scan-heavy matters (fast, free on the gateway; mimo-v2.5 free is PNG-only, auto-converted). If Gemini free or Mistral are delayed/429ing, do NOT wait on them — muse/qwen/mimo are the fallbacks.
  4. Rate-limit protocol: on repeated HTTP 429s do not hammer — runs are resume-safe; stop that file/matter, note remaining pages (resumed_pages vs scan_pages_ocr in the pipeline JSON), and report. The gate should make this rare.
  5. Writes: inside the assigned matter folder only. The README contract still applies to the matter README. 5b. NotebookLM staging sync — build or refresh the NotebookLM_<ref> sibling staging folder via the okf-notebooklm skill: python .opencode/skills/okf-notebooklm/tools/okf_nblm_build.py <matter_root> --check; rebuild when dirty (exit 1), and BUILD it if no staging folder exists yet (every OKF matter gets one). Sources over 500k words fail validation — split the bundle (usually a raw comms-log/00_Admin mega-bundle; processing the matter's OCR backlog resolves it). Never write into the okf-notebooklm skill folder.
  6. Return the fixed-format report below — the main agent merges.

Worker report format (return verbatim)

MATTER: <ref>
STATUS: <complete|partial|blocked>
FILES: <processed / deduped / OCR'd / skipped>
QA: <PASS|PARTIAL|FAIL> + residual lines needing human decision
EMAIL: <archive messages / threads / attachments>
STAGING: <synced|dirty|none> + rebuild output summary
PENDING: <remaining items + reason>
README: <updated Y/N>
NEW LESSONS: <one-liners for learnings.md, or "none">

Main-agent coordination

  • Keep a batch status table (queued / in flight / done + verdict) and show progress to the user.
  • Merge worker reports into ONE final summary; collect NEW LESSONS and append to learnings.md once (single-writer; dedupe overlaps across workers).
  • PARTIAL/FAIL residuals and BLOCKED matters follow the never-guess rule: stay flagged for human decision, or ask the user.

Workflow — canonical order of operations

Order matters: free & offline first, expensive & API-bound last. Dedup and born-digital text extraction are instant and local; vision OCR is the slow/paid tail and must only see what is actually left. Per-file pipeline runs happen once each; all writes (log, README, yaml, chronology) happen ONCE per batch, not per file.

Phase 0   Recon & dedup          (cheap, prevents wasted work)
Phase 1   Born-digital extract   (free, offline, exact)
Phase 2   Vision triage          (expensive tail — only the remainder)
Phase 2b  Post-OCR quality check (batch-level verification, after ALL OCR)
Phase 3   Email knowledge layer  (independent of OCR)
Phase 4   File & consolidate     (writes once)

Phase 0 — Recon & dedup (once per batch)

  1. Read learnings — load learnings.md (same folder as this skill) and apply known quirks to the current batch.
  2. List & census — list the target folder (or PENDING rows in 97_OCR_Log\OCR_Log.csv; parse with the Python csv module — filenames contain commas, never split on ,). Count by extension to find the shape: PDFs, .eml/.msg, images, .gdoc stubs, signature files, oversized files.
  3. sha256 dedup FIRST — hash all files (skip unreadable .gdoc stubs), group identical bytes, keep ONE copy per hash — prefer the most descriptive filename. (1)/(2) suffixed files are usually DIFFERENT versions (different sizes), NOT dupes — verify by hash before touching them. Record removals for the log.
  4. Decoration cleanup — delete email-signature noise: base64-named files (*==), Outlook-*, image00X.*, clio/branding logos (powered-byclio.png, clio-border.png, welcome_email_*, front-composer*, LogoM365.png, microsoft.png), ~WRD*.jpg, (Optimized).txt Document AI error stubs. Keep .gdoc stubs as thread pointers (content lives in the communications logs). Note "SKIPPED: decoration" for the log.
  5. Classify what remains: Document (PDFs, scans, images of letters/contracts/submissions) → process; Already-text (.docx, real .txt, text-layer PDFs) → text extract (Phase 1); Outlook .msg → convert (Phase 3); Broken stubs (PDF portfolio wrappers "open this PDF portfolio in Acrobat X") → strip; extract embedded members and file them in place of the wrapper.

Phase 1 — Born-digital text extraction (free, offline, exact)

  1. Text-layer extraction for ALL PDFs — every PDF gets page.get_text() written to <base>_V1_OCR.txt sidecars immediately (instant, no vision). This is the bulk of any corpus and costs nothing. Office files (.docx/.xls/.ppt/.rtf/.epub/...) get the same sidecar via anydoc (fast local Markdown conversion, v1.10); the pdf-inspector classifier cross-checks PDF text layers (diagnostic warnings only).
  2. Assess text-layer quality — per PDF count extractable chars, text pages, and "gibberish pages" (alpha-char ratio < 0.25). Classify: good (searchable, done) / garbled-mixed (a few image/photo pages among good text — usually covers/index tables/photos, not bad OCR; the text is already extracted) / no-text (scans — go to Phase 2). Record the assessment with the batch.
  3. Extract PDF metadata (title, author, subject, pages, first-page text) + image EXIF (Pillow _getexif(): DateTimeOriginal, Make, Model, GPS) → 00_Admin\Image_Metadata.csv + .md. Use it to judge filename descriptiveness (rename only if a name is genuinely non-descriptive AND the metadata is a better label; never for versioned email captures) and to capture evidence dates/geolocation. HEIC: EXIF unreadable without pillow_heif — derive date from iOS filename (YYYYMMDD_HHMMSS..._iOS.heic). 8b. Append native structure outlines — run python <skill>/tools/native_structure.py <matter_root> so every born-digital sidecar carries a [DOCUMENT OUTLINE] appendix (PDF bookmarks/heading scan, DOCX headings, XLSX sheets).

Phase 2 — Vision triage (the expensive tail)

  1. Send ONLY the remainder to the vision pipeline — the no-text scans and (if worth it) the garbled pages: python <skill>/tools/ocr_pipeline.py <target> --matter <matter_root> --json <batch.json>. The auto chain is gemini → muse (XHigh, opt-in) → qwen → mimo (free) → mistral → paddle with low-confidence auto-resolution (kimi-k3 paid is opt-in only, never automatic). Delay policy: if Gemini free or Mistral are 429ing/slow, prefer --engine muse/--engine qwen/--engine mimo rather than waiting on them. Skip: photos (not documents), oversized bundles (>10MB — file as evidence, don't OCR), and already-text PDFs (extracted in Phase 1). Rotate sideways photos via PIL (Image.rotate(270, expand=True)) before re-running on those files.
  2. QA the vision output — confidence-driven, not blind:
    • PASS: vision pages clean (no {..} flags); PaddleOCR pages ≥0.90 mean. Verify metadata + key figures only.
    • PARTIAL (any flagged lines): most were reconciled by the auto-resolver; review ONLY the residuals — [LOW CONFIDENCE...] / [FIGURE UNVERIFIED...] / [CONFLICT: ...] lines against the source page; otherwise mark Quality_Check = PARTIAL and note the lines.
    • FAIL (any page flagged FAIL): stop and report — never guess figures.
    • Tables: ALWAYS double-check tables against the source, EVEN with zero auto-flags — the pipeline only flags arithmetic mismatches (pandas column-sum vs stated total) and even those are often FALSE positives (cross-table parse artifacts: per-line mini-table headers like £295.00 summed across tables, item/Doc#/U-value columns summed that were never meant to total). Zero flags ≠ correct table. Minimum spot-check per matter: re-sum 1-2 invoice line-item columns against stated totals, verify bank-statement balance chains (prev ± paid/withdrawn = new), confirm key columns (dates, prices, U-values, item numbers) against the sidecar, and eyeball row/column structure on 2-3 tables. Annotate OCR_Log notes when flags are artifacts; never re-run for artifact flags.
    • Capture metadata WHILE checking: document_type, parties, date, key figures.
  3. Map dense documents (when present) — for reports/advice with a ToC (expert reports, FRAEW, adjudications), run python <skill>/tools/dense_doc_map.py "<file.pdf>" → <base>_V1_OCR_index.md (mechanical section tree, pages verified from the text layer) + <base>_V1_OCR_map.md (per-section rows, all [UNVERIFIED]). Fill map rows for the sections that matter (key findings, opinions, conclusions, costs) from the verbatim text; drop [UNVERIFIED] only when the pinpoint (page/section/paragraph) is confirmed — same rule as the case-law skill. README and Chronology reference map:§4.6 p44-46.

Phase 2b — Post-OCR quality check (batch-level verification, after ALL OCR)

  1. Batch QA pass — after every page of OCR (text-layer AND vision) is written, verify the whole batch against the pipeline JSON + sidecars, NOT per-page:
    • Cross-engine no-text verdicts: a scan that produced no text must be confirmed by ALL engines — Gemini empty + Mistral placeholder/hallucination + PaddleOCR 0 lines. Mistral HALLUCINATES on blank/dark pages (repeated LaTeX like \( \therefore m = \frac{3}{11} \), fake ![img-0.jpeg] placeholders, fabricated "Source: ..." URLs). Only when every engine agrees "no text" is the page logged as such (blank page / dark photo / non-document image). Never trust one engine alone.
    • CONFLICT/ALT artifact triage — [CONFLICT: ...] / [ALT: ...] lines are often FORMATTING variants, not content errors. Classify each:
    • Line-split artifacts: alt engine merges lines the base splits ("...within 10 days," + "supply and fit a new steel beam." → one line). Content identical → not a real conflict.
    • Footnote/superscript formatting: legal scans (Scots law authorities: $^{302}$, ¹⁵⁴ superscripts) differ only in footnote markup. Citations identical → formatting-only.
    • Dash/quote variants: en-dash vs hyphen, straight vs curly quotes → cosmetic.
    • TRUE conflicts (real silent errors: 3MT/OMT, UG467/UG487, Page 1/Page 7 header-vs-footer): keep flagged, page stays PARTIAL, resolve against the source page.
    • Residuals are never auto-cleared — engine reads are NON-DETERMINISTIC run-to-run (a later pass can produce DIFFERENT silent errors). Only the reconciled output (or 2-of-3 verdicts) is trustworthy; [CONFLICT]/[LOW CONFIDENCE]/[HANDWRITTEN] lines stay for human/agent decision.
    • Signature/handwriting flags: [LOW CONFIDENCE] on signature lines (msGA, We, em) = handwriting, not transcription error — log as PARTIAL, don't chase engines.
    • Arithmetic flags: [ARITHMETIC ERROR] on invoice bundles is USUALLY cross-table parse artifacts (per-line mini-table headers like £295.00 summed across tables) — verify source maths, annotate, don't re-run. Genuine errors only where the source truly doesn't balance.
    • Vision cross-check of financial tables — run python <skill>/tools/table_vision_check.py <matter_root> (Mistral engine default). The pandas auto-flag misses pdfplumber empty-table extractions and merged-cell errors; vision independently reads the totals. Interpret FLAGs: per-invoice Total vs account Outstanding Balance (both may be correct), and cross-check the vision grand against the sidecar text. Record the interpretation in QA_Verdict.
    • Table double-check EVEN with zero flags — auto-flags only catch column-sum mismatches; verify tables unconditionally: re-sum invoice line items vs stated totals, walk bank-statement balance chains (prev ± paid/withdrawn = new), confirm key columns (dates, prices, U-values, item/Doc# codes) against the sidecar, eyeball structure on 2-3 tables. Zero flags ≠ correct table.
    • No-text photo pages inside documents (fire-strategy covers, furniture/stonework photos in email chains, bundle index tables) — not OCR failures; the text is already extracted. Log, don't re-OCR.
    • Case-ref / content discrepancies across documents (e.g. GLW-A728-22 vs -23): file BOTH as-is, flag in README/Chronology — never "correct" OCR or document content to make refs agree.
    • Re-run the cross-layer dupe check — 0 dupes across Email_Archive / _split / attachments before the matter is complete.

Phase 3 — Email knowledge layer (independent of OCR)

  1. Convert .msg files (all at once) — python -m extract_msg --out <staging_dir> "file.msg" → <timestamp> <subject>\message.txt + attachments. Envelope (From/To/CC/Date/Subject/body) is a correspondence record; each attachment is a document.
  2. Convert .eml to sidecars — parse with the stdlib email module (policy.default); write <basename>_eml.txt IN THE SAME DIR as the source (naming trap: splitext-base matching fails across dirs — generate sidecars next to sources, never in a separate admin folder). Attachments → 03_Productions\06_Originals\_eml_attachments\. 18b. Split emails into individual messages (eml_split.py) — python <skill>/tools/eml_split.py <matter_root> decomposes every .eml/.msg chain into its component messages: <source>_split\NNN_YYYYMMDD_Sender_Subject.md per unique message (dedup by subject+body hash) with tags From/To/Cc/Sent/Subject/Attachments/Attachments extracted/Sensitivity/Source (+ Archive ID when it matches an Email_Archive message). Extracts attachments (dedup by sha) to _eml_attachments\. Nested Outlook items are parsed recursively — a .msg/.eml attached inside a .msg/.eml is itself parsed (its messages → <att_dir>\<name>_split\, tagged Nested in: <parent>), and its attachments recursed into, until exhaustion. Guards: --max-depth N (default 12) and a sha256 set of container bytes that both breaks cycles (a self-containing .msg is parsed once) and dedupes attachment extraction across all sources (13 thread captures extract once). This is the per-message view for LLM use; the raw _eml.txt sidecar remains for verbatim fidelity. 18c. Cross-source split dedup (REQUIRED after 18b) — the same message is quoted inside EVERY chain capture that contains it, so it appears once per capture: dedup (subject, body) across ALL _split dirs — keep the canonical copy (longest source-dir name = most specific capture), delete the rest, record deletions in 00_Admin\eml_split_dedup_manifest.csv (deleted → kept_as), then REGENERATE the affected _split\index.md files from the remaining messages. Also sha-dedup _eml_attachments\ (keep the longest filename — Outlook signature images recur under different names). Re-run the cross-layer dupe check until 0. 18d. Attachment coverage check (MANDATORY — the gate) — python <skill>/tools/email_attachment_check.py <matter_root> parses every .eml/.msg in the matter RAW (stdlib email / extract_msg; .msg falls back to the sibling .md ## Attachments listing when extract_msg is missing), compares every declared attachment against what exists in _eml_attachments\ and the _split dirs, writes 97_OCR_Log\email_attachment_check.csv, and exits 1 when any declared attachment is missing or when no extraction has run at all. Fix by (re-)running eml_split before Phase 4; a matter is not complete with missing attachments. Failure mode this exists to catch (matter 01730, 2026-08-12): email conversions declared ## Attachments: (66 JA certificate PDFs, a 258 KB Deduction Notice, subcontractor EOT claims, an earlier baseline programme) while eml_split.py was never run — the documents were locked inside the .eml/.msg binaries, invisible to --check, missing_evidence.py and every downstream tool. Declaring an attachment without extracting it is a FAIL, not a note.
  3. Dedup .eml content — by (normalized_subject, body_sha256) (body = text/plain+text/html whitespace-collapsed); keep the most descriptive filename. Separate from stub dedup.
  4. Consolidate thread stubs — --email-threads --apply groups the .gdoc stub captures into threads (one kept pointer each), writes 97_OCR_Log\email_threads.csv. Run BEFORE --scan so the scan plan is smaller (and since 2026-08-05 --email-threads --apply is decoupled from plain --apply — it will NOT apply moves.csv).
  5. Build the Email_Archive — MANDATORY for every matter with a Master Communications Log — python <skill>/tools/commslog_archive.py <matter_root>. Splits the Master log + all 00_Admin\*Daily Communications Log*.txt (same clio format, merged chronologically, overlap removed by subject+body dedup) into 00_Admin\Email_Archive\<thread>\NNN_YYYYMMDD_Sender_Subject.md — one .md per unique message (envelope + body, stable thread/NNN ID, per-thread index.md, stub_index.md mapping every .gdoc stub). This is the reviewable/searchable evidence layer for the WHOLE email corpus: real .eml/.txt files AND stub-only threads (their content comes from the logs, so the archive is complete without Drive access). Re-runs clear the output folder first. The archive is never optional — a matter is not complete without it; the README must link it.

Phase 4 — File & consolidate (writes once)

  1. File outputs — move/copy as needed:
    • Filed copy: <dest>\YYYYMMDD_Description_FromTo_V1_OCR.<ext> (increment V# on later versions). If the source already sits at its destination with a dated name, keep it — just ensure _OCR sidecar + tables CSV exist.
    • Raw scan: original unaltered as _SRC in 03_Productions\06_Originals (contentious) or 02_Contract_Documents\05_Execution_Exchange (non-contentious)
    • Sidecars must follow their sources — when a PDF moves, move its _V1_OCR.txt alongside (else --check flags the PDF as new). Do this after apply.
  2. Log — append ALL staged rows to 97_OCR_Log\OCR_Log.csv in ONE write (Python csv module). Update any rows that were PENDING to PASS/FAIL/SKIPPED instead of duplicating.
  3. Update README + matter.yaml + Chronology — on EVERY change, not only at batch end. The README is the living state of the matter: any change to the corpus (files processed, files moved/re-filed, quality flags changed, QA verdicts resolved, sidecar annotations added, dedup/thread results, missing-evidence findings, counts) MUST be reflected in the README before the session ends — see "README contract" below. Single pass: add all documents to README Document Index (ISO dates), sync matter.yaml (documents, links.figures, links.case_law, links.expert), and append event rows to 00_Admin\Chronology.md. Note in README: intentional skips (oversized bundles, drawings), mixed-matter folders left untouched, and the large-corpus --check caveat (filed emails/photos aren't in OCR_Log — expected at scale). 24b. Run the missing-evidence test — python <skill>/tools/missing_evidence.py <matter_root> scans all correspondence (sidecars, Email_Archive, Master/Daily Communications Logs, _eml.txt) for references the schema should satisfy — explicit attachments ("I attach X", "Attachments: Y"), bills/invoices ("bill 2553", "invoice 3463"), dated letters ("our letter of 16 September 2025", "letter from X dated ..."), production numbers ("production 6/1") — and checks each against the filed corpus (filenames, OCR_Log, _eml_attachments\, inventories). Output: 97_OCR_Log\missing_evidence.csv + a MISSING-HIGH / LOW report. Every MISSING-HIGH finding becomes a README missing-evidence entry (per the README contract); LOW items are eyeball-checked. Re-run after every batch — a reference satisfied by a newly filed document drops out automatically.
  4. Learn — append new lessons to learnings.md (dated bullets). 25b. NotebookLM staging sync — build or refresh the NotebookLM_<ref> sibling staging folder via the okf-notebooklm skill: python .opencode/skills/okf-notebooklm/tools/okf_nblm_build.py <matter_root> --check; rebuild when dirty (exit 1), and BUILD it if no staging folder exists (every OKF matter gets one). Sources over 500k words fail validation — split the bundle (usually a raw comms-log/00_Admin mega-bundle; processing the matter's OCR backlog resolves it). Report the result in the STAGING line.
  5. Summary — files processed / deduped / text-extracted / vision-OCR'd / skipped (reasons) / tables extracted + arithmetic errors / quality flags / remaining PENDING / post-OCR QA verdict (per-batch: PASS / PARTIAL-with-residuals / FAIL; list residual [CONFLICT]/[LOW CONFIDENCE] lines needing human decision).

README contract (the matter's living state)

The matter README is a living document, not a batch-end artefact. Every change to the matter MUST be reflected in the README before the session ends — a user reading the README must see the current truth about the corpus. This is a standing rule across ALL phases (0-4), not just consolidation.

Triggers — any of these requires a README edit

  • New files processed / OCR'd (counts in the Document Index line: OCR_Log.csv (N rows))
  • Documents moved, re-filed, or routing corrected (incl. counterparty reclassification, e.g. "X is our correspondent firm, not opponent")
  • Quality flags changed (PASS ↔ PARTIAL ↔ FAIL) — keep the QA residuals section in sync
  • QA verdicts resolved or new ambiguities flagged (e.g. vision-model resolutions, figure verifications, human-decision items)
  • Sidecar annotations added (metadata headers, # Inventory:, # Structure:, # Vision QA)
  • Email layer changes (threads, Email_Archive rebuild, attachment indexes)
  • Missing / referenced-but-absent evidence discovered or resolved (incl. missing_evidence.py MISSING-HIGH findings)
  • Production/inventory mappings (which file = which production number)
  • Zips/bulk exports handled or intentionally left
  • Anything the previous run reported as "to do" (e.g. re-OCR after re-export)

Sections that must stay current

README section Invariant
Parties & Case names/roles; opponent agents vs own correspondent firm clearly distinguished
Document Index every substantive document with location + quality; the OCR_Log (N rows) count line
Email Archive link + message count after every rebuild
Missing / referenced-but-absent every known gap with rationale (stubs, corrupt files, bulk sources left in place)
QA residuals (PARTIAL) one bullet per PARTIAL file; resolved items annotated RESOLVED = ... with basis; open human-decision items explicitly labelled HUMAN DECISION
OCR pipeline quality current PASS/PARTIAL/FAIL counts; pipeline behaviour changes worth knowing
Production numbers / inventory 6/x-style mappings with file names

Discipline

  • Update the README in the SAME session as the change — never "leave for later".
  • When a change supersedes a previous README statement, EDIT the old line, don't append a contradicting one.
  • matter.yaml (documents, key_dates, links) and Chronology.md follow the same rule for their scopes.
  • If a change is reverted (e.g. a move undone), the README must be reverted with it.

Rules

  • Parallel batches (list workflow): max 2 matters in flight; one okf-worker subagent per matter; workers never write the skill folder; one writer per file; every pipeline call runs under the shared --max-concurrency gate (default 2). The main agent is the single writer for learnings.md and any batch-level files, and merges all worker reports.
  • Every matter with a Master Communications Log MUST have 00_Admin\Email_Archive built (tools/commslog_archive.py <matter_root>) — one .md sidecar per unique email (envelope + body, thread/NNN IDs, index.md + stub_index.md). A matter is not complete without it; the README must link it. The archive covers stub-only threads too — their content comes from the Master + Daily Communications Logs.
  • The README is the living state of the matter — every change (processed files, re-filing/routing corrections, quality-flag changes, QA verdicts, sidecar annotations, email-layer results, missing-evidence, production mappings) must be reflected in the README in the same session (see "README contract"). Never leave a change un-documented; edit superseded statements rather than appending contradictions.
  • The what's-missing test runs on every matter — tools/missing_evidence.py <matter_root> after filing/consolidation: anything correspondence refers to (attachments, bills/invoices, dated letters, production numbers) that the schema cannot satisfy is a MISSING-HIGH finding and MUST be listed in the README missing-evidence section (and chased: filed elsewhere? in the bulk export? truly absent?). A reference satisfied later drops off the list automatically on re-run.
  • Email dedup = unique individual MESSAGES, not threads (user directive) — every unique message stored separately with a stable ID; threads are only grouping folders, never the dedup unit. The Email_Archive (from logs), the _split dirs (from .eml), and the raw _eml.txt sidecars are three views of the same corpus: verbatim / per-message-tagged / archive.
  • Attachments ARE essential email content — ~30% of emails carry them (drawings, photos, fee letters, evidence). Every email sidecar/archive message MUST list Attachments: (names) and the extraction path; extract them to _eml_attachments\ via eml_split.py and VERIFY with email_attachment_check.py (step 18d) — a declared-but-unextracted attachment is a FAIL (01730: certificates/deduction notices/sub claims were locked inside .eml/.msg because eml_split was skipped).
  • Tables are ALWAYS double-checked against the source, EVEN when the pipeline reports zero flags — auto-flags only catch column-sum mismatches (and those are usually false-positive parse artifacts); a quiet table can still have wrong cells. Minimum per matter: re-sum 1-2 invoice line-item columns vs stated totals, verify bank-statement balance chains, confirm key columns (dates/prices/codes) against the sidecar, eyeball structure on 2-3 tables. Zero flags ≠ correct table.
  • Never edit, redact, or "fix" document content.
  • If scan quality prevents verbatim transcription, stop and report — never guess figures.
  • Confidentiality: default engine is vision API (Gemini) with the user's own key; --engine paddle is the fully-offline mode (PaddleOCR models cached locally). Never paste document content outside this matter's context.

Example

User: "OCR the inbox" — run the canonical sequence: Phase 0 recon + --dedup + decoration cleanup; Phase 1 born-digital text extraction + quality assessment + metadata/EXIF; Phase 2 vision OCR only for the no-text remainder; Phase 2b post-OCR quality check (cross-engine no-text verdicts, CONFLICT/ALT artifact triage, residuals never auto-cleared, arithmetic artifacts, case-ref discrepancies); Phase 3 email layer (convert .eml/.msg → _eml.txt sidecars + eml_split.py per-message dirs + cross-source dedup with manifest + --email-threads + Email_Archive build via commslog_archive.py); Phase 4 --scan → review moves → --apply → sidecars follow sources → consolidate log/README/yaml/chronology once → append learnings → summarise: files processed / deduped / text-extracted / vision-OCR'd / unique messages archived / attachments extracted / tables extracted + arithmetic flags / quality flags + QA verdict / remaining PENDING. Every change made along the way updates the README in the same session (README contract). If a filed document is dense (expert report, FRAEW), map it (step 11) so its references resolve to sections.

User: "work through this list: 01234-A, 01235-B, 01236-C" — treat as a batch queue (see "Parallel multi-matter batches"): launch okf-worker subagents for the first 2 matters in the SAME message; each runs phases 0→4 on its own matter under the shared --max-concurrency 2 API gate; as each finishes, launch the next. Collect the fixed-format reports, merge into one summary, append lessons once.