Loading experiment data…

Audit Copilot Portal

LLM pipeline for internal-audit findings — RAG · generation · criticality · summary · quality

Pipeline heads at a glance

the four learned heads that turn a raw finding into a full audit write-up

Generator lift — fine-tuning vs base

semantic similarity to gold write-ups (before → after)

Criticality models — best macro-F1

top 8 classifiers · High/Moderate/Low

Findings dataset — criticality mix

Key findings

Generator leaderboard

fine-tuned decoder/seq2seq models drafting the full finding JSON · click a row for detail

Metrics are on the held-out test findings. SemSim / BERTScore are semantic; JSON% and Schema% measure structural validity; Crit-acc is the criticality field agreement.

Before → after (SemSim)

fine-tuning lift per model

Structural validity

JSON-parsable & perfect-schema %, after fine-tuning

Per-field quality — champion generator

Risk / root-cause / recommendation each scored separately (ROUGE-L, SemSim, rescaled BERTScore).

Criticality leaderboard

High / Moderate / Low severity · ranked by macro-F1 · click a row for the confusion matrix

Macro-F1 — top models

Accuracy vs High-recall

catching High-severity items is the safety metric · color = family

Executive-summary models

condense a finding to ≤2 sentences / ≤40 words · ROUGE-L + SemSim + style constraints

Summarizer quality

ROUGE-L vs SemSim

Style-constraint adherence

length / sentence / no-em-dash / no-banned-words

Finding-quality regressors

Predict a 1–5 human quality rating from engineered + embedding features. Lower MAE and higher Pearson-r / within-1 are better.

How the retriever was evaluated

IR metrics for the RAG layer — bi-encoder ranking vs cross-encoder rerank

Each finding's title + description is used as a query against the other 847 findings (indexed exactly as the live pipeline: all-mpnet-base-v2 embeddings over word-chunks, cross-encoder ms-marco-MiniLM rerank of the top-30). There are no human relevance labels, so relevance is derived automatically two ways: Paraphrase — findings sharing the query's exact title (the augmentation produced 2–7 paraphrases per finding; a strict known-item test), and Topical — findings in the same Root Cause Category (7 classes; a broad-recall test). Metrics are macro-averaged; random is an illustrative uniform-ranking baseline.

Paraphrase retrieval — ranking quality

known-item · 499 queries · ~3.7 relevant each

Paraphrase retrieval — Recall@k

share of the finding's paraphrases found in top-k

Topical retrieval — ranking quality

same root-cause category · 848 queries · base rate 15.8%

Reranker lift

cross-encoder vs bi-encoder-only on the key metrics

All retrieval metrics

bi-encoder vs +reranker vs random · higher is better

Runs on a normal CPU — no GPU needed

measured inference feasibility for the two neural heads on a 10th-gen-class laptop CPU

The generator (Pythia-410M + LoRA) and summariser (BART-base) are the only compute-heavy heads; criticality (TF-IDF + logistic regression) and quality (XGBoost) are classical and run in well under a millisecond. Numbers below are measured wall-clock on CPU, warm-up excluded, with inference threads capped to emulate 10th-gen laptop core budgets (4 threads ≈ i5-10400, 8 threads ≈ i7-10700).

Weight footprint — FP32 vs INT8 vs INT4

OpenVINO / NNCF weight compression · on-disk MB

Generator latency / finding

180-token JSON draft, greedy

Summariser latency / finding

≤2-sentence summary, beam=4

FP32 runtime — PyTorch vs OpenVINO

same precision; graph optimization alone ~halves generator latency

Full-pipeline latency breakdown

per-head ms/finding @8 threads (OpenVINO, FP32)

Measured numbers

weight footprint (MB) + latency (per finding) on CPU

Analyze a finding

enter a raw audit observation — the pipeline drafts the full finding, classifies severity, summarizes, scores quality and retrieves references

Vector knowledge base

…

The copilot grounds every generation in two separate Qdrant collections. Browse or search them here, and add your own corporate documents (policies, standards, SOPs) to the corporate collection so they surface as references.

📥 Ingest corporate documents

.txt · .md · .csv · .pdf · .docx · .xlsx

Audit Copilot — an LLM pipeline for internal-audit findings

Course MAAI902 — Final Project University University of Wollongong in Dubai (UOWD) Task NLP · RAG · generation · classification

Internal auditors spend hours turning a raw observation — “call recordings were not retained for the required five years in 5 of 15 sampled cases” — into a complete, well-structured finding: a risk statement, a root cause, a recommendation, and a criticality rating. This project builds and evaluates an LLM pipeline that drafts that whole write-up from a short title and description, grounded in past findings and corporate policy through retrieval-augmented generation.

The system is organized as four learned heads over a shared retrieval layer, each benchmarked independently across many model families. This portal lets you browse every experiment, compare models, and run the full pipeline live on your own text.

The heads

  • Generator — a fine-tuned language model (Qwen-0.5B, Pythia-160M/410M, T5-v1.1, ByT5) that takes the title + description and emits a valid JSON finding (criticality, risk, root_cause, recommendation). Evaluated before vs after fine-tuning.
  • Criticality classifier — predicts High / Moderate / Low severity. Forty variants were tried: classical TF-IDF + linear/logistic/ordinal models, frozen MPNet embeddings + LightGBM/RBF-SVM, RoBERTa, and instruction-tuned Pythia/Qwen — each with a real and an augmented training-data version.
  • Summarizer — condenses a finding into a ≤2-sentence, ≤40-word executive summary (BART, FLAN-T5, T5-v1.1, Pythia), scored on ROUGE-L, semantic similarity and style constraints.
  • Quality regressors — predict human 1–5 ratings for Completeness, Conciseness and Clarity from engineered readability features plus embeddings (XGBoost, SVR, Ridge, RandomForest, HistGB).

Retrieval (RAG)

Both the live copilot and evaluation ground generation in two Qdrant vector collections — past audit findings and corporate documents — embedded with all-mpnet-base-v2 and re-ranked with a ms-marco-MiniLM cross-encoder. The Try it tab shows the top similar findings and policy references alongside each generation.

Metrics glossary

  • SemSim / ContentSemSim — cosine similarity of sentence embeddings to the gold text (overall and content-only).
  • ROUGE-L — longest-common-subsequence overlap with the reference.
  • BERTScore (F1 / rescaled) — contextual-embedding token overlap; rescaled is baseline-adjusted.
  • JSON-parsable % / Perfect-schema % — share of generations that parse as JSON and match the exact 4-field schema.
  • Criticality accuracy — agreement of the generator's severity field with ground truth.
  • Macro-F1 / High-recall — classifier quality overall and specifically on the safety-critical High class.
  • MAE / Pearson-r / within-1 — quality-regressor error, correlation, and share of predictions within 1 point of the human rating.

Runs on ordinary hardware

Every head is small enough to run inference on a standard consumer CPU (e.g. an Intel i5/i7). The generators are ≤0.5 B parameters with LoRA adapters, the classical criticality and quality models are milliseconds-per-call, and retrieval uses a compact embedding model — so the whole copilot works locally without a GPU.

Try it · Copilot

The Try it tab runs the real models. Type a finding (or pick an example), choose a model per head, and hit Run pipeline to get the drafted risk / root-cause / recommendation, a severity verdict with class probabilities, an executive summary, quality scores, and the retrieved references — all from a single call.

This tab needs the local backend: run python serve.py in the portal folder and open http://localhost:8000. The backend loads the trained heads from the local workspace (or streams them from GCS) and runs inference on demand. Charts and leaderboards work fully static without it.