VerdictTank — Multimodel Proposal Review Platform

1System Overview

VerdictTank scores an uploaded proposal or RFP response against a rubric using a panel of independent LLM judges, then aggregates their verdicts into a single, defensible PASS / BELOW decision with per-criterion breakdowns and actionable fixes.

The Consensus Problem

A single LLM asked "is this proposal good?" produces a fragile answer: score drift between runs, sycophancy toward confident prose, sensitivity to prompt phrasing, and no way to detect when the model is simply wrong. VerdictTank treats scoring as a consensus problem, not a single-inference problem. Three heterogeneous judges (different model families, different training corpora) score the same artifact under an identical rubric. Divergence between judges is a signal — it flags criteria where the artifact is genuinely ambiguous — and agreement is the measure of confidence. The platform's core value is not any one model's opinion; it is the reproducible, auditable aggregation of independent opinions.

Architecture Decisions

Component Map

  ┌──────────────────────────────────────────────────────────────────────────┐
  │                            verdicttank.com (Caddy)                         │
  │  TLS termination · HTTP/2 · automatic certs · reverse proxy · rate limit   │
  └───────────────┬───────────────────────────────────────────┬──────────────┘
                  │                                           │
        ┌─────────▼─────────┐                       ┌─────────▼─────────┐
        │   API (FastAPI)   │  ◀── stateless ──▶    │  Worker (arq/RQ)  │
        │  auth · sessions  │                       │  judge dispatch    │
        │  webhooks · gate  │                       │  aggregate · fixit │
        └───┬───────┬───────┘                       └───┬───────────┬────┘
            │       │                                   │           │
      ┌─────▼──┐ ┌──▼─────┐                       ┌─────▼───┐  ┌────▼─────────┐
      │Postgres│ │ Redis  │                       │ LiteLLM │  │  Wasabi S3   │
      │ state  │ │queue + │                       │ gateway │  │ artifacts +  │
      │ + DDL  │ │ cache  │                       │ 3 judges│  │ corpus (iso) │
      └────────┘ └────────┘                       └────┬────┘  └──────────────┘
                                                       │
                          ┌────────────────────────────┼────────────────────────┐
                          ▼                            ▼                          ▼
                   Anthropic family            OpenAI family             DeepSeek family
                   (judge slot A)               (judge slot B)            (judge slot C)

  Observability: Prometheus scrapes API+Worker+LiteLLM /metrics → Grafana dashboards + alerts.
  

2Multimodel Scoring Pipeline

Three independent judges score the same sanitized artifact in parallel through the LiteLLM gateway. Per-criterion scores are aggregated deterministically into an overall score and a binary verdict.

Judge Roster (via LiteLLM)

Aliases resolve to concrete provider models inside LiteLLM's config.yaml. Swapping a judge's underlying model is a LiteLLM config change plus a reload — no application deploy. Weights are configurable per rubric; default is equal (1.0) so all three judges count identically.

Parallel Dispatch

  sanitized_artifact + rubric
            │
            ├──────────────┬──────────────┐         all three fire concurrently
            ▼              ▼              ▼          (asyncio.gather, per-judge timeout)
       ┌────────┐    ┌────────┐    ┌────────┐
       │Judge A │    │Judge B │    │Judge C │       each: prompt → LiteLLM → JSON score
       │ 8s TO  │    │ 8s TO  │    │ 8s TO  │       on timeout/error → fallback chain
       └───┬────┘    └───┬────┘    └───┬────┘
           └─────────────┼─────────────┘
                         ▼
                  ┌──────────────┐
                  │  Aggregator  │  weighted mean per criterion → overall → PASS/BELOW
                  └──────┬───────┘
                         ▼
                   verdict record (persisted)
  

Dispatch uses asyncio.gather(return_exceptions=True). A judge that raises or times out does not abort the panel; its slot enters the fallback chain independently. The panel is valid if at least 2 of 3 judges return a well-formed score (quorum = 2). With only 1 valid judge the session transitions to DEGRADED and the verdict is flagged low-confidence.

Scoring Schema (judge output contract)

Every judge is instructed to return strict JSON matching this schema. The response is parsed and validated; a schema violation triggers the fallback chain (stage 4, reformat).

{
  "$schema": "verdicttank/judge-score/v4",
  "type": "object",
  "required": ["judge","criteria","overall","rationale_summary"],
  "properties": {
    "judge":   { "type": "string", "enum": ["judge-anthropic","judge-openai","judge-deepseek"] },
    "criteria": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["id","score","weight","rationale"],
        "properties": {
          "id":        { "type": "string" },              // rubric criterion id, e.g. "req_coverage"
          "score":     { "type": "number", "minimum": 0, "maximum": 100 },
          "weight":    { "type": "number", "minimum": 0, "maximum": 1 },
          "rationale": { "type": "string", "maxLength": 600 },
          "evidence":  { "type": "array", "items": { "type": "string" } } // quoted spans from artifact
        }
      }
    },
    "overall":            { "type": "number", "minimum": 0, "maximum": 100 },
    "rationale_summary":  { "type": "string", "maxLength": 1200 },
    "flags":              { "type": "array", "items": { "type": "string" } } // e.g. "missing_pricing"
  }
}

Aggregation

Aggregation is deterministic and published so a customer can reproduce the verdict by hand.

1. Per-criterion aggregate. For criterion c, compute the judge-weighted mean: agg_c = Σ(judge_weight_j × score_jc) / Σ(judge_weight_j) over judges that returned a valid score for c.
2. Criterion-weighted overall. OVERALL = Σ(rubric_weight_c × agg_c) / Σ(rubric_weight_c).
3. Agreement metric. spread_c = max_j(score_jc) − min_j(score_jc). Any criterion with spread_c ≥ 25 is tagged contested and surfaced in the report.
4. Confidence. confidence = clamp(1 − mean(spread_c)/100, 0, 1), degraded by 0.15 per missing judge below quorum.
5. Verdict. OVERALL ≥ threshold → PASS else BELOW. Default threshold = 75; configurable per rubric, stored on the verdict for auditability.

6-Stage Fallback Chain (per judge slot)

Each judge slot independently walks this chain. The chain guarantees a slot either yields a valid score or is cleanly marked UNAVAILABLE (counting against quorum) — it never hangs.

Latency Budgets

Error handling — pipeline

• All judges fail (quorum lost): session → FAILED; API returns 503 with Retry-After; no charge recorded against the session's review credit.
• Partial panel (2 of 3): verdict produced, confidence reduced, response includes "degraded": true and the unavailable slot id.
• Malformed after reformat (stage 4 fails): slot treated as UNAVAILABLE; raw body stored for post-mortem in judge_raw_output.
• Wall-clock cap hit: in-flight judge calls are cancelled; if quorum already met the partial verdict is committed as TIMEOUT_PARTIAL, else FAILED.

3Fix-It Engine

After a verdict, the Fix-It Engine converts each weak criterion into a ranked, actionable fix, then re-scores the revised artifact to produce a measured before/after delta.

Failure Modes (what Fix-It detects)

Prioritization Formula

Fixes are ranked by expected verdict impact per unit of effort, so the customer fixes the things that move the needle first.

Fixes are returned sorted by priority_c descending. Each carries an estimated_lift = rubric_weight_c × gap_c — the maximum points the overall score can gain if that criterion reaches target_score. The UI shows a running "projected verdict if all applied" sum, clamped so OVERALL ≤ 100.

Rescore Delta

When the customer submits a revised artifact (or accepts inline suggestions), the engine re-runs the panel and reports a real, measured delta — never a promise.

{
  "$schema": "verdicttank/rescore-delta/v4",
  "session_id": "rs_9f3a...",
  "before": { "overall": 68.4, "verdict": "BELOW", "confidence": 0.71 },
  "after":  { "overall": 81.2, "verdict": "PASS",  "confidence": 0.83 },
  "delta":  { "overall": 12.8, "verdict_changed": true },
  "per_criterion": [
    { "id": "req_coverage", "before": 55.0, "after": 84.0, "delta": 29.0, "fix_applied": "fix_01" },
    { "id": "clarity",      "before": 72.0, "after": 78.0, "delta":  6.0, "fix_applied": "fix_03" }
  ],
  "rescored_at": "2026-08-11T14:22:08Z"
}

Optimization: if only a subset of criteria changed (detected by artifact diff), the engine re-scores just those criteria across all three judges and reuses unchanged criterion scores — cutting rescore latency and cost. Full re-scores are forced when the artifact diff exceeds 30% of tokens.

Single-Tier Binary Verdict — PASS / BELOW

VerdictTank issues exactly one of two verdicts. There is no letter grade, no star rating, and no intermediate "maybe" tier. This is deliberate (AD-04): a buyer needs a defensible go/no-go.

The numeric OVERALL, per-criterion sub-scores, confidence, and contested flags are always returned alongside the binary verdict for transparency — the binary is the decision, the numbers are the justification.

Error handling — Fix-It

• No weak criteria (already PASS): engine returns an empty fix list with "status":"no_action_needed"; UI shows the passing report only.
• Rescore panel loses quorum: the before verdict is preserved unchanged; response carries "rescore_status":"degraded" and does not overwrite the stored verdict.
• Artifact unchanged from prior submission (hash match): rescore is skipped; returns 409 Conflict with the existing verdict to avoid a redundant paid run.
• Compliance floor still breached after fixes: verdict stays BELOW even if OVERALL crosses threshold; the blocking criterion is named in blocking_criteria.

4Data Flow

A review moves through six durable stages. Each stage has an input contract, an output contract, and a persisted state transition. The pipeline is crash-safe: recovery resumes at the last committed stage.

6-Stage Pipeline

  ①INGEST → ②SANITIZE → ③DISPATCH → ④AGGREGATE → ⑤FIXIT → ⑥PERSIST/DELIVER
     │           │            │            │           │            │
   upload     redact PII   3 judges     weighted   ranked fixes  verdict +
   + parse    (idempotent)  parallel     mean →     + rescore     webhook +
   artifact                 (LiteLLM)    PASS/BELOW  delta         report URL
  

① INGEST — input contract

{ "$schema":"verdicttank/ingest/v4",
  "tenant_id":"t_8821", "rubric_id":"rub_default_v4",
  "artifact":{ "filename":"proposal.pdf", "mime":"application/pdf",
               "bytes_b64":"JVBERi0x...", "sha256":"a91f..." },
  "options":{ "threshold":75, "webhook_url":"https://buyer.example/hooks/vt" } }

Parser extracts plain text (PDF/DOCX/MD/TXT). Artifact stored to Wasabi at s3://vt-artifacts/{tenant_id}/{session_id}/original. Output: artifact_text + token_count.

② SANITIZE — output contract

{ "$schema":"verdicttank/sanitized/v4",
  "session_id":"rs_9f3a", "artifact_text":"...[REDACTED:EMAIL]... [REDACTED:PHONE]...",
  "redactions":[ {"type":"EMAIL","count":3}, {"type":"PHONE","count":1}, {"type":"SSN","count":0} ],
  "sanitize_hash":"7c2e..." }

Deterministic regex + NER redaction. Idempotent: re-running yields the identical sanitize_hash. Only the sanitized text is ever sent to a judge (AD-06).

③ DISPATCH — output contract (per judge, see §2 scoring schema)

{ "session_id":"rs_9f3a", "judge_results":[
    {"judge":"judge-anthropic","status":"ok","overall":66,"criteria":[...]},
    {"judge":"judge-openai","status":"ok","overall":71,"criteria":[...]},
    {"judge":"judge-deepseek","status":"substituted","overall":68,"criteria":[...]} ],
  "quorum_met":true, "degraded":false }

④ AGGREGATE — output contract

{ "$schema":"verdicttank/verdict/v4",
  "session_id":"rs_9f3a", "overall":68.4, "verdict":"BELOW", "threshold":75,
  "confidence":0.71,
  "criteria":[ {"id":"req_coverage","agg":55.0,"spread":18,"contested":false},
               {"id":"clarity","agg":72.0,"spread":9,"contested":false},
               {"id":"differentiation","agg":60.0,"spread":31,"contested":true} ],
  "blocking_criteria":[], "computed_at":"2026-08-11T14:20:55Z" }

⑤ FIXIT — output contract (see §3)

{ "session_id":"rs_9f3a", "status":"action_needed",
  "fixes":[ {"id":"fix_01","criterion":"req_coverage","mode":"MISSING_REQUIREMENT",
             "priority":24.8,"estimated_lift":9.0,"text":"Add SOC 2 evidence section..."} ],
  "projected_overall":81.2 }

⑥ PERSIST / DELIVER — output contract

{ "session_id":"rs_9f3a", "state":"COMPLETED",
  "report_url":"https://verdicttank.com/r/rs_9f3a",
  "webhook_delivered":true, "verdict":"BELOW" }

Review Session State Machine

                         ┌───────────────────────────────────────────────┐
                         │                                               │
   (POST /reviews)       ▼                                               │
   ─────────────▶ [ RECEIVED ]                                           │
                     │ ingest+parse ok                                   │
                     ▼                                                   │
                [ SANITIZED ]                                            │
                     │ redaction committed                              │
                     ▼                                                   │
                [ DISPATCHING ] ──quorum lost──▶ [ FAILED ] ◀──wall-clock cap (no quorum)
                     │ ≥2 judges ok                     │
                     ▼                                  │ (terminal, 503 to caller)
                [ AGGREGATED ] ──1 judge only──▶ [ DEGRADED ]
                     │ verdict computed                 │ (low-confidence verdict stored)
                     ▼                                  ▼
                [ SCORED ] ◀──────────────────────────┘
                  │        │
      fixes needed│        │already PASS / no action
                  ▼        ▼
             [ FIXIT_READY ]   [ COMPLETED ]───(webhook + report)───▶ (terminal)
                  │  rescore submitted                 ▲
                  ▼                                    │ verdict updated
             [ RESCORING ] ──quorum lost──▶ keeps prior verdict, back to [ SCORED ]
                  │ new verdict computed               │
                  └────────────────────────────────────┘

   Timeout guard: any non-terminal state exceeding 25s wall-clock → [ TIMEOUT_PARTIAL ]
   (if quorum already met, commit partial verdict → SCORED) else → [ FAILED ].
   Crash recovery: worker restart reloads last committed state; completed judge calls
   are cached by (session_id, judge, sanitize_hash) and never re-billed.
  

*TIMEOUT_PARTIAL is terminal for the original run but a new rescore may reopen the session into RESCORING.

5API Surface

REST/JSON over HTTPS. Base URL https://api.verdicttank.com/v4. All bodies UTF-8 JSON. All timestamps RFC 3339 UTC. All IDs are prefixed opaque strings.

Authentication

• API keys — Authorization: Bearer vt_live_XXXX for server-to-server. Keys are tenant-scoped, hashed at rest (argon2id), and carry a role (viewer, reviewer, admin). Test keys prefixed vt_test_.
• Session JWT — short-lived (15 min) bearer for browser clients, issued by POST /auth/token, refreshed via POST /auth/refresh. Contains tenant_id, role, exp. Signed HS256 with a rotated secret.
• Webhook signature — outbound webhooks are signed X-VT-Signature: sha256=HMAC(body) with the tenant's webhook secret.

Rate Limiting

Token-bucket per API key, enforced in Redis. Limits returned on every response:

X-RateLimit-Limit: 60
X-RateLimit-Remaining: 57
X-RateLimit-Reset: 1754923200
Retry-After: 12          # only on 429

Endpoints

• POST/auth/token — exchange API key or credentials for a session JWT.
• POST/reviews — create a review (ingest artifact, start pipeline).
• GET/reviews/{id} — fetch a review session + verdict.
• GET/reviews/{id}/verdict — verdict + per-criterion detail only.
• GET/reviews/{id}/fixes — Fix-It ranked fix list.
• POST/reviews/{id}/rescore — submit revised artifact, get before/after delta.
• GET/reviews — list tenant reviews (paginated, filterable by verdict/state).
• GET/rubrics · POST/rubrics — manage scoring rubrics.
• GET/webhooks · POST/webhooks · DELETE/webhooks/{id} — manage webhook endpoints.
• GET/whitelabel/status — 5-condition gate ledger (see §8).
• GET/healthz · GET/metrics — liveness + Prometheus.

POST /reviews — request

POST /v4/reviews
Authorization: Bearer vt_live_XXXX
Content-Type: application/json

{ "rubric_id":"rub_default_v4",
  "artifact":{ "filename":"proposal.pdf","mime":"application/pdf","bytes_b64":"JVBERi0x..." },
  "options":{ "threshold":75, "webhook_url":"https://buyer.example/hooks/vt", "async":true } }

POST /reviews — response (202 async)

HTTP/1.1 202 Accepted
Location: /v4/reviews/rs_9f3a
{ "session_id":"rs_9f3a", "state":"RECEIVED", "poll_url":"/v4/reviews/rs_9f3a",
  "estimated_ready_s":6 }

With "async":false the call blocks up to the 25 s wall-clock cap and returns 200 with the full verdict inline.

GET /reviews/{id} — response (200)

{ "session_id":"rs_9f3a", "state":"SCORED", "tenant_id":"t_8821",
  "verdict":"BELOW", "overall":68.4, "threshold":75, "confidence":0.71, "degraded":false,
  "criteria":[ {"id":"req_coverage","agg":55.0,"spread":18,"contested":false,
                "rationales":{"judge-anthropic":"...","judge-openai":"...","judge-deepseek":"..."}} ],
  "blocking_criteria":[], "report_url":"https://verdicttank.com/r/rs_9f3a",
  "created_at":"2026-08-11T14:20:49Z", "computed_at":"2026-08-11T14:20:55Z" }

POST /reviews/{id}/rescore — request/response

POST /v4/reviews/rs_9f3a/rescore
{ "artifact":{ "filename":"proposal_v2.pdf","mime":"application/pdf","bytes_b64":"..." } }

→ 200  { ...rescore-delta/v4 object (see §3)... }
→ 409  { "error":"unchanged_artifact","message":"sha256 matches prior submission" }

Webhooks

Registered endpoints receive signed POSTs on state transitions. Events: review.scored, review.completed, review.failed, review.rescored.

POST {webhook_url}
X-VT-Signature: sha256=9a3c...
X-VT-Event: review.scored
X-VT-Delivery: whd_44a1

{ "event":"review.scored", "session_id":"rs_9f3a", "verdict":"BELOW",
  "overall":68.4, "occurred_at":"2026-08-11T14:20:55Z" }

Delivery: at-least-once with exponential backoff (max 6 attempts over ~1h). Consumers must be idempotent on X-VT-Delivery. Verify by recomputing HMAC-SHA256(secret, raw_body) and constant-time comparing to X-VT-Signature.

Error Model

All errors share one envelope. HTTP status conveys the class; error conveys the code.

{ "error":"quorum_lost",
  "message":"Only 1 of 3 judges returned a valid score.",
  "session_id":"rs_9f3a", "request_id":"req_7c2e", "retryable":true }

6Data Model

PostgreSQL 16. Multi-tenant with tenant_id on every business table and row-level security enforced by the query layer. All money is integer cents; all scores are numeric(5,2).

Entity Relationships

  tenants ──1:N── api_keys
     │
     ├──1:N── rubrics ──1:N── rubric_criteria
     │
     ├──1:N── review_sessions ──1:N── judge_results ──1:N── judge_criterion_scores
     │              │
     │              ├──1:1── verdicts
     │              └──1:N── fixes
     │
     ├──1:N── webhooks
     └──1:1── whitelabel_ledger
  

DDL

CREATE TABLE tenants (
  tenant_id     TEXT PRIMARY KEY,                 -- 't_8821'
  name          TEXT NOT NULL,
  plan          TEXT NOT NULL DEFAULT 'free'
                CHECK (plan IN ('free','pro','enterprise','whitelabel')),
  data_role     TEXT NOT NULL DEFAULT 'controller'
                CHECK (data_role IN ('controller','joint_controller','processor')),
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE api_keys (
  key_id        TEXT PRIMARY KEY,                 -- 'ak_...'
  tenant_id     TEXT NOT NULL REFERENCES tenants ON DELETE CASCADE,
  key_hash      TEXT NOT NULL,                    -- argon2id(secret)
  role          TEXT NOT NULL CHECK (role IN ('viewer','reviewer','admin')),
  mode          TEXT NOT NULL CHECK (mode IN ('live','test')),
  last_used_at  TIMESTAMPTZ,
  revoked_at    TIMESTAMPTZ,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX ix_api_keys_tenant ON api_keys(tenant_id);

CREATE TABLE rubrics (
  rubric_id     TEXT PRIMARY KEY,                 -- 'rub_default_v4'
  tenant_id     TEXT REFERENCES tenants ON DELETE CASCADE,  -- NULL = system rubric
  name          TEXT NOT NULL,
  threshold     NUMERIC(5,2) NOT NULL DEFAULT 75.00,
  version       TEXT NOT NULL DEFAULT 'v4',
  is_active     BOOLEAN NOT NULL DEFAULT true,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE rubric_criteria (
  criterion_id  TEXT PRIMARY KEY,                 -- 'req_coverage'
  rubric_id     TEXT NOT NULL REFERENCES rubrics ON DELETE CASCADE,
  label         TEXT NOT NULL,
  weight        NUMERIC(4,3) NOT NULL CHECK (weight BETWEEN 0 AND 1),
  is_mandatory  BOOLEAN NOT NULL DEFAULT false,
  hard_floor    NUMERIC(5,2) NOT NULL DEFAULT 0.00, -- mandatory floor; below = forces BELOW
  sort_order    INT NOT NULL DEFAULT 0
);
CREATE INDEX ix_criteria_rubric ON rubric_criteria(rubric_id);

CREATE TABLE review_sessions (
  session_id    TEXT PRIMARY KEY,                 -- 'rs_9f3a'
  tenant_id     TEXT NOT NULL REFERENCES tenants ON DELETE CASCADE,
  rubric_id     TEXT NOT NULL REFERENCES rubrics,
  state         TEXT NOT NULL DEFAULT 'RECEIVED'
                CHECK (state IN ('RECEIVED','SANITIZED','DISPATCHING','AGGREGATED',
                                 'SCORED','FIXIT_READY','RESCORING','DEGRADED',
                                 'COMPLETED','TIMEOUT_PARTIAL','FAILED')),
  artifact_sha  TEXT NOT NULL,                    -- sha256 of original artifact
  sanitize_hash TEXT,                             -- idempotency key for judge cache
  s3_key        TEXT NOT NULL,                    -- vt-artifacts/{tenant}/{session}/original
  threshold     NUMERIC(5,2) NOT NULL,
  webhook_url   TEXT,
  charged       BOOLEAN NOT NULL DEFAULT false,   -- billing guard (FAILED never charged)
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  updated_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX ix_sessions_tenant_state ON review_sessions(tenant_id, state);

CREATE TABLE judge_results (
  id            BIGSERIAL PRIMARY KEY,
  session_id    TEXT NOT NULL REFERENCES review_sessions ON DELETE CASCADE,
  judge         TEXT NOT NULL,                    -- 'judge-anthropic' | ...
  status        TEXT NOT NULL CHECK (status IN ('ok','substituted','unavailable')),
  overall       NUMERIC(5,2),
  fallback_stage SMALLINT NOT NULL DEFAULT 1,     -- which of the 6 stages produced this
  latency_ms    INT,
  raw_output    JSONB,                            -- persisted for replay/post-mortem
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  UNIQUE (session_id, judge)                      -- idempotent per (session,judge)
);

CREATE TABLE judge_criterion_scores (
  id            BIGSERIAL PRIMARY KEY,
  judge_result_id BIGINT NOT NULL REFERENCES judge_results ON DELETE CASCADE,
  criterion_id  TEXT NOT NULL,
  score         NUMERIC(5,2) NOT NULL CHECK (score BETWEEN 0 AND 100),
  rationale     TEXT,
  evidence      JSONB
);
CREATE INDEX ix_jcs_result ON judge_criterion_scores(judge_result_id);

CREATE TABLE verdicts (
  session_id    TEXT PRIMARY KEY REFERENCES review_sessions ON DELETE CASCADE,
  verdict       TEXT NOT NULL CHECK (verdict IN ('PASS','BELOW')),
  overall       NUMERIC(5,2) NOT NULL,
  threshold     NUMERIC(5,2) NOT NULL,
  confidence    NUMERIC(4,3) NOT NULL,
  degraded      BOOLEAN NOT NULL DEFAULT false,
  criteria_agg  JSONB NOT NULL,                   -- [{id,agg,spread,contested}]
  blocking_criteria JSONB NOT NULL DEFAULT '[]',
  computed_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE fixes (
  fix_id        TEXT PRIMARY KEY,                 -- 'fix_01'
  session_id    TEXT NOT NULL REFERENCES review_sessions ON DELETE CASCADE,
  criterion_id  TEXT NOT NULL,
  mode          TEXT NOT NULL,                    -- MISSING_REQUIREMENT | WEAK_EVIDENCE | ...
  priority      NUMERIC(7,3) NOT NULL,
  estimated_lift NUMERIC(5,2) NOT NULL,
  body          TEXT NOT NULL,
  applied       BOOLEAN NOT NULL DEFAULT false
);
CREATE INDEX ix_fixes_session ON fixes(session_id);

CREATE TABLE webhooks (
  webhook_id    TEXT PRIMARY KEY,
  tenant_id     TEXT NOT NULL REFERENCES tenants ON DELETE CASCADE,
  url           TEXT NOT NULL,
  secret_hash   TEXT NOT NULL,
  events        TEXT[] NOT NULL DEFAULT '{review.scored,review.completed}',
  active        BOOLEAN NOT NULL DEFAULT true,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE whitelabel_ledger (
  tenant_id           TEXT PRIMARY KEY REFERENCES tenants ON DELETE CASCADE,
  dpa_signed          BOOLEAN NOT NULL DEFAULT false,
  billing_active      BOOLEAN NOT NULL DEFAULT false,
  corpus_isolated     BOOLEAN NOT NULL DEFAULT false,
  branding_approved   BOOLEAN NOT NULL DEFAULT false,
  subprocessor_ack    BOOLEAN NOT NULL DEFAULT false,
  unlocked_at         TIMESTAMPTZ,
  updated_at          TIMESTAMPTZ NOT NULL DEFAULT now()
);

Row-level security policies (omitted for brevity) restrict every business table to tenant_id = current_setting('app.tenant_id'). The app sets this GUC per request from the authenticated key, so a query can never span tenants even on a coding error.

7Security

Customer proposals are confidential business documents. The security model assumes every judge is an external sub-processor and every tenant is mutually distrustful.

PII Sanitization

Sanitization (data-flow stage ②) runs before any bytes leave the trust boundary for a model call. It is deterministic and idempotent so a verdict is reproducible and the redaction is auditable.

• Detectors: regex passes for EMAIL, PHONE, SSN/EIN, credit-card (Luhn-checked), IBAN, street address; plus a NER pass for PERSON and ORG when the tenant enables aggressive mode.
• Replacement: each match becomes a typed token [REDACTED:EMAIL]. A per-session salted map (never sent to judges) allows post-hoc rehydration in the customer's own report view only.
• Idempotency: sanitize_hash = sha256(sanitized_text) is the judge-cache key — re-scoring identical sanitized text reuses cached judge results and never re-bills.
• Audit: the redactions count vector (no raw values) is stored on the session.

Corpus Isolation

A tenant's reference corpus (past winning proposals, style guides) is never visible to another tenant's scoring context. Enforced at three layers (AD-07):

• Storage: S3 prefix s3://vt-corpus/{tenant_id}/… with a bucket policy denying cross-prefix reads; each tenant's IAM-scoped credentials cannot list outside their prefix.
• Database: row-level security keyed on tenant_id GUC (see §6).
• Retrieval: corpus lookups always inject WHERE tenant_id = :current; the query builder refuses to run a corpus query without a bound tenant.

Sub-Processor Training Guard

Customer content must never train a third-party model (AD-08). Enforced at the LiteLLM boundary:

• Every judge call sets provider no-train / zero-data-retention flags where offered (e.g. OpenAI store:false, Anthropic no-retention headers).
• DeepSeek traffic is routed only through a contracted no-retention endpoint; if that route is unavailable, the DeepSeek slot is marked UNAVAILABLE rather than falling back to a retaining endpoint — the training guard overrides the fallback chain.
• LiteLLM request logs store metadata + token counts only, never artifact text.
• Contractual: each provider is a named sub-processor in the DPA with a no-training clause; the tenant's subprocessor_ack ledger flag records acknowledgment (see §8).

Encryption

RBAC

Roles are carried on the API key and in the session JWT. Every mutating endpoint checks role before touching state; a role violation returns 403 forbidden. Cross-tenant access (valid key, wrong tenant's resource id) returns 404 not_found — the platform does not reveal that another tenant's resource exists.

Error handling — security

• Sanitizer failure: fail-closed → session FAILED, no dispatch, alert fired.
• Training-guard route down: affected judge slot UNAVAILABLE; guard never bypassed.
• RLS/GUC unset: queries against business tables raise and abort the request (defense in depth against a missing tenant context).
• Expired/rotated JWT secret: old tokens fail verification → 401; clients refresh via /auth/refresh.

8White-Label Gate

White-Label makes VerdictTank a processor acting on a customer's behalf under their brand. Processor-role features stay locked behind a 5-condition ledger until every prerequisite is verifiably satisfied (AD-11). Any locked call returns HTTP 423 Locked.

5-Condition Ledger

Backed by the whitelabel_ledger table (§6). All five booleans must be true before unlocked_at is stamped and White-Label endpoints activate.

GET /whitelabel/status — response (200)

{ "tenant_id":"t_8821", "unlocked":false, "unlocked_at":null,
  "conditions":{
    "dpa_signed":true,
    "billing_active":true,
    "corpus_isolated":false,     // ← blocking
    "branding_approved":false,   // ← blocking
    "subprocessor_ack":true
  },
  "blocking":["corpus_isolated","branding_approved"] }

HTTP 423 Response (locked feature access)

Any White-Label-scoped endpoint (custom-branded report render, processor-mode review, tenant-branded webhook) invoked while the ledger is incomplete returns:

HTTP/1.1 423 Locked
Content-Type: application/json

{ "error":"whitelabel_locked",
  "message":"White-Label features are locked until all onboarding conditions are met.",
  "tenant_id":"t_8821",
  "blocking":["corpus_isolated","branding_approved"],
  "status_url":"/v4/whitelabel/status",
  "retryable":false }

423 is chosen deliberately over 403: the resource is not forbidden by role, it is temporarily locked pending a state change the tenant can resolve. The blocking array tells the caller exactly which conditions remain, and status_url points at the live ledger. Once all conditions clear, the same call succeeds with no code change on the client side.

9Deployment Topology

Single netcup RS 4000 host running Docker Compose. Caddy terminates TLS and reverse-proxies to the app tier; Prometheus + Grafana provide observability; Wasabi S3 holds artifacts and corpus.

Host

Topology Diagram

  Internet
     │ 443
  ┌──▼───────────────────────────────────────── netcup RS 4000 (app3) ──────────────┐
  │                                                                                   │
  │  ┌────────────┐   verdicttank.com / api.verdicttank.com                          │
  │  │   Caddy 2  │   TLS · HTTP/2 · rate-limit · reverse proxy                        │
  │  └─────┬──────┘                                                                    │
  │        │ (docker private net: vtnet)                                               │
  │   ┌────▼─────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌────────────────┐  │
  │   │  api     │   │ worker   │   │ postgres │   │  redis   │   │    litellm     │  │
  │   │ FastAPI  │◀─▶│  arq     │◀─▶│    16    │   │    7     │   │    gateway     │──┼─▶ providers
  │   │ (uvicorn)│   │          │   │  (vol)   │   │  (vol)   │   │  (3 judges)    │  │  (Anthropic/
  │   └────┬─────┘   └────┬─────┘   └──────────┘   └──────────┘   └────────────────┘  │   OpenAI/
  │        │              │                                                            │   DeepSeek)
  │   ┌────▼──────────────▼────┐    ┌────────────┐   ┌───────────┐                     │
  │   │      /metrics          │───▶│ Prometheus │──▶│  Grafana  │  dashboards+alerts   │
  │   └────────────────────────┘    └────────────┘   └───────────┘                     │
  │                                                                                     │
  │   nightly: pg_dump + artifact sync ───────────────────────────────────────────────┼─▶ Wasabi S3
  └─────────────────────────────────────────────────────────────────────────────────┘
  

docker-compose.yml (reference)

services:
  caddy:
    image: caddy:2
    ports: ["80:80","443:443"]
    volumes: ["./Caddyfile:/etc/caddy/Caddyfile","caddy_data:/data"]
    networks: [vtnet]
    depends_on: [api]

  api:
    build: ./app
    command: uvicorn vt.main:app --host 0.0.0.0 --port 8000 --workers 4
    env_file: [.env]
    networks: [vtnet]
    depends_on: [postgres, redis, litellm]
    healthcheck:
      test: ["CMD","curl","-fsS","http://localhost:8000/healthz"]
      interval: 15s

  worker:
    build: ./app
    command: arq vt.worker.WorkerSettings
    env_file: [.env]
    networks: [vtnet]
    depends_on: [postgres, redis, litellm]

  postgres:
    image: postgres:16
    environment: [POSTGRES_DB=vt, POSTGRES_USER=vt]
    env_file: [.env]              # POSTGRES_PASSWORD
    volumes: ["pg_data:/var/lib/postgresql/data"]
    networks: [vtnet]

  redis:
    image: redis:7
    command: ["redis-server","--appendonly","yes"]
    volumes: ["redis_data:/data"]
    networks: [vtnet]

  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    command: ["--config","/app/config.yaml"]
    volumes: ["./litellm/config.yaml:/app/config.yaml"]
    env_file: [.env]              # provider keys, no-retention routes
    networks: [vtnet]

  prometheus:
    image: prom/prometheus
    volumes: ["./prometheus.yml:/etc/prometheus/prometheus.yml","prom_data:/prometheus"]
    networks: [vtnet]

  grafana:
    image: grafana/grafana
    volumes: ["grafana_data:/var/lib/grafana"]
    networks: [vtnet]
    depends_on: [prometheus]

networks: { vtnet: { driver: bridge } }
volumes: { caddy_data: {}, pg_data: {}, redis_data: {}, prom_data: {}, grafana_data: {} }

Caddyfile (reference)

verdicttank.com {
  encode gzip zstd
  reverse_proxy api:8000
  header Strict-Transport-Security "max-age=31536000; includeSubDomains"
}
api.verdicttank.com {
  encode gzip zstd
  rate_limit { zone api { key {remote_host}; events 60; window 1m } }
  reverse_proxy api:8000
}

Observability

• Prometheus scrapes /metrics from api, worker, and litellm every 15 s.
• Key metrics: vt_review_latency_seconds (histogram), vt_judge_fallback_stage (counter by stage), vt_quorum_lost_total, vt_verdict_total{verdict}, vt_sanitize_failures_total.
• Grafana alerts: p95 end-to-end > 18 s (5 min), quorum-loss rate > 1% (10 min), any sanitizer failure (immediate), a judge slot stuck at fallback stage ≥5 (5 min).

Backups & DR

• Nightly pg_dump → s3://vt-backups/pg/ (Wasabi), 30-day retention.
• Artifact + corpus objects already durable in Wasabi (SSE); versioning enabled on vt-artifacts.
• Config-as-code: compose file, Caddyfile, litellm config.yaml, and DDL migrations in the infra repo — full host rebuild from repo + .env + latest pg dump.

Error handling — deployment

• Container crash: Compose restart: unless-stopped; api healthcheck failure removes it from Caddy upstream until healthy.
• Postgres unavailable: api returns 503; worker retries with backoff; no data loss (state is committed transactionally).
• LiteLLM down: all judge slots walk the fallback chain to stage 6; sessions go DEGRADED/FAILED rather than hang; alert fires.
• Wasabi unreachable: ingest fails fast with 503; nightly backup retries and alerts on repeated failure.

10Infrastructure Tracker

Canonical record of the deployed footprint. The product domain is verdicttank.com. rfptank.com is retained only as legacy/defensive and serves no production traffic.

Domains & DNS

TLS

• Certificates issued and auto-renewed by Caddy (ACME / Let's Encrypt) for verdicttank.com, www.verdicttank.com, api.verdicttank.com.
• TLS 1.3, HSTS max-age=31536000; includeSubDomains. CAA locks issuance to the ACME CA.

Origin & Storage

Backup Schedule

Deployed Document Map