RAG, Agents, Evals, and LLMOps for Product Safety

RecallGraph AI

Built a production-style local AI engineering system: source-shaped CPSC/openFDA/NHTSA fixtures, normalized RecallRecord models, idempotent backfill, SQLite persistence with a Postgres/pgvector path, hybrid retrieval, deterministic multi-agent orchestration, cited risk briefs, eval gates, trace exports, readiness reports, audit trail, recovery drill, and GitHub Actions validation.

★85 Validation
Start a project like this

Challenge

Product-safety risk data is fragmented across recall agencies, enforcement feeds, complaints, tables, notices, and future PDFs/images. A basic chat wrapper cannot prove source coverage, retrieval quality, citation faithfulness, or operational readiness.

Solution

Built a production-style local AI engineering system: source-shaped CPSC/openFDA/NHTSA fixtures, normalized RecallRecord models, idempotent backfill, SQLite persistence with a Postgres/pgvector path, hybrid retrieval, deterministic multi-agent orchestration, cited risk briefs, eval gates, trace exports, readiness reports, audit trail, recovery drill, and GitHub Actions validation.

Impact Metrics

85
Validation
tests
4/4
Sources
fixture coverage
4/4
Retrieval
labeled cases
2
Backfill
idempotent runs

Results

  • • 85-test validation suite passes with only 2 expected optional-dependency skips
  • • 4/4 configured public-source fixtures covered across CPSC, openFDA food, openFDA device, and NHTSA
  • • 4/4 labeled retrieval cases pass with 1.0 mean recall@k and 1.0 mean reciprocal rank
  • • Citation faithfulness gate passes with 0 unsupported controlled risk terms and 0 unlinked source mentions
  • • Backfill drill proves safe reruns with 4 records, 8 chunks, 4 raw payloads, and 2 ingestion-run history rows

Business Impact

Demonstrates senior AI engineering depth: not just prompt wiring, but the data contracts, retrieval gates, agent traces, LLMOps artifacts, and production boundaries needed to operate a trustworthy AI workflow.

Architecture

RecallGraph AI: RAG, Agents, Evals, and LLMOpsPublicCPSC / FDA / NHTSAIngestfixtures + backfillStoreSQLite / pgvectorRetrievehybrid rankingAgentsverify + scoreBriefcited answerSource Health4/4 fixture coverageQuality Gatesretrieval + faithfulnessLLMOps Evidencetraces, audit, recoveryGitHub-ready local baseline: make validate, 85 tests, no credentials by defaultPython • RAG • multi-agent workflow • eval harness • SQLite • pgvector path

Portfolio project runs locally without credentials, cloud resources, paid model APIs, or live network calls. Hosted auth, tenancy, source completeness, and model-provider integrations remain explicitly deferred.

Interactive Workbench

Explore the fixture-backed analyst console with risk filtering, cited evidence, agent traces, eval gates, and a controlled natural-language panel.

Open Workbench

Production-Style AI Engineering Loop

A proven approach combining statistical rigor, automation, and AWS best practices.

1

Source Contracts

Represent CPSC, openFDA, and NHTSA as source adapters with dry-run URL builders, cached fixtures, freshness checks, and explicit live-network gates.

2

RAG Baseline

Normalize records into evidence chunks, rank with local hybrid retrieval, and keep pgvector parity available behind an explicit Docker target.

3

Agent Orchestration

Run a deterministic planner, retriever, evidence verifier, risk scorer, and report writer so the workflow can be tested before adding LLM variance.

4

Eval + Ops Gates

Gate retrieval quality, answer quality, faithfulness, latency, readiness, audit coverage, recovery, and idempotent ingestion through local validation and CI.

Scale & Scope

85
Validation
local tests across ingestion, retrieval, agents, API, storage, evals, security, and ops
4
Source Coverage
source-shaped fixture families with 4 records, 8 chunks, and raw payload persistence
9+
Operational Gates
production gates including readiness, performance, faithfulness, audit, recovery, backfill, and migration checks

Technology Stack

  • Python 3.11 package with stdlib-first runtime
  • RAG and hybrid retrieval
  • Deterministic multi-agent orchestration
  • Offline retrieval, answer, and faithfulness evals
  • SQLite local repo with Postgres/pgvector migration path
  • FastAPI and LangGraph optional adapters
  • GitHub Actions validation workflow
  • Docker pgvector parity target

Need a similar solution?

Let's replicate this success within your organization with a tailored engagement plan.

Start a ConversationView Our Process