MLOps case studies

Engineering decisions, architecture, and verified evidence.

These projects are designed around production failure modes. Metrics are identified as controlled or synthetic results where applicable; none are presented as customer-production outcomes.

Architecture at a glance

Three systems. Five operating concerns.

A concise comparison of how each deep dive moves from governed inputs to release, service, observation, and recovery.

Concern	FraudShield	RAGOps	SignalOps
Data & lineage	DVC stages, feature validation, tracked training metadata	Versioned knowledge inputs, retrieval and citation evaluation	Licensed admission, trusted timestamps, point-in-time manifests and feature contracts
Release gate	Model-regression policy after controlled retraining	Retrieval, grounding, safety, abstention, and latency thresholds	Purged walk-forward evidence, signed promotion policy, stress matrix, holdout lockbox
Serving boundary	Typed FastAPI container; Kubernetes and Terraform definitions	Dockerized grounded-response API	Authorized alerts to paper simulation only; no live brokerage execution
Monitoring	Prometheus metrics, probes, PSI drift, autoscaling definitions	Grounding quality, citation coverage, refusal behavior, latency	Calibration, PSI drift, feed health, queue backlog, execution quality, reconciliation
Recovery & rollout	Promotion blocks regressions; operational alerts define response	Release rejection and explicit abstention on insufficient evidence	Shadow champion–challenger, kill switches, incident evidence, exact-model rollback
Evidence status	4 automated tests; deployment artifacts are definitions	6 automated tests; 7-case synthetic evaluation	206 automated tests; PostgreSQL and AWS resources are not deployed

01 · End-to-end delivery

FraudShield

A transaction-risk platform built to demonstrate reproducible model delivery rather than model accuracy alone. The pipeline versions stages and metadata, blocks regressions, packages a typed API, and defines day-two operations.

DVCMLflowFastAPIKubernetesTerraform

Download project ↓

Controlled evaluation0.838 ROC-AUC and 0.757 recall on held-out synthetic data

Release controlRegression-aware promotion gate

OperationsPrometheus alerts, PSI drift, probes, autoscaling

Data lineageDVC stages, feature validation, training metadata, and versioned model outputs preserve the development path

Deployment boundaryDocker, Kubernetes, and Terraform artifacts are included as tested project definitions; no production deployment is claimed

Verification4 automated tests pass

DATAValidated featuresDVC stages

→

TRAINCandidate modelTracked metadata

→

GATEQuality policyPromote or block

→

SERVEFastAPI imageKubernetes

→

OBSERVEMetrics and driftPrometheus

CI/CD

Regression-aware build path

GitHub Actions installs the pinned Python requirements, runs four tests, retrains the controlled fixture, executes the promotion gate, and builds a commit-addressed Docker image.

OPERATIONS

Guardrails before serving

Schema checks and the model regression policy precede API packaging. Prometheus configuration, probes, autoscaling, and PSI drift signals define the intended operating contract.

02 · Evaluation-first LLMOps

RAGOps

A source-grounded support service that treats retrieval, prompts, citations, abstention, and safety as versioned release concerns. Candidates must meet explicit quality and safety thresholds.

RAGFastAPIDockerGitHub Actions

Download project ↓

QUERYUser requestSafety screen

→

RETRIEVERelevant sourcesVersioned knowledge

→

ANSWERGrounded responseCite or abstain

→

EVALUATEQuality suiteRelease decision

CI/CD

Evaluation-gated container

GitHub Actions runs six tests, the seven-case synthetic evaluation suite, the release gate, and a commit-addressed Docker build. Retrieval or safety regressions stop the build path.

GUARDRAILS

Ground, cite, or abstain

Versioned knowledge, citation checks, latency limits, prompt-injection screening, and unsafe medical-advice refusals are evaluated together before release.

03 · Risk-controlled operations

SignalOps

An authorized Discord-bot alert service that parses messages into paper orders, applies deterministic policy, and records every decision. User-token automation and live brokerage execution are excluded.

FastAPISQLiteDiscord BotPrometheus

Download project ↓ Verify downloads ↗

Trusted data boundaryIndependent-source consensus, authenticated feeds, ordered sequences, outage failover, gap detection, split adjustment, and signed ingestion evidence

Historical provenanceProvider-neutral licensed-source imports with revision pins, UTC/as-of timestamps, sequence and gap enforcement, plus raw, normalized, and manifest SHA-256 verification

Final-holdout lockboxSigned development, embargo, and holdout partitions with label-availability checks, one-time access receipts, and automatic rejection of post-open tuning

Licensed dataset admissionOperator-attested license rights, immutable revisions, signed freeze receipts, and a single final-holdout policy

Reproducible experiment lineageSigned candidate bundles bind admitted development data, code, environment, seed, parameters, operator, and report

Versioned feature contractsHash-bound types, sources, lookbacks, label horizons, and point-in-time availability checks

End-to-end research workflowFail-closed import, admission, point-in-time features, walk-forward evaluation, and signed lineage orchestration

Authorized research-run CLIStrict operator contract verifies license text, hashes every input, executes the point-in-time workflow, and atomically writes immutable signed development evidence

Independent evidence verificationValidates HMAC signatures, canonical hashes, original input hashes, development-only status, and dataset, report, receipt, and lineage cross-links without rerunning training

Adaptive abstention policyFold-local threshold selection targets calibration precision and abstains when evidence is insufficient

Secure cloud definitionsHTTPS, private tasks, digest-pinned images, least privilege, Secrets Manager injection, non-root runtime, and dropped capabilities

Deployment policy gateIndependent HCL parsing enforces 12 stable controls for TLS, private networking, image integrity, least privilege, secret injection, encryption, non-root execution, and paper-only mode; CI blocks every finding

Disaster-recovery definitionsEncrypted Multi-AZ PostgreSQL, private database access, 35-day backups, deletion protection, mandatory final snapshot, and versioned KMS-encrypted evidence under 90-day COMPLIANCE Object Lock

Risk controlsStale, duplicate, symbol, stop, and 2% allocation gates

SimulationPaper-only latency, quote-quality, spread, nonlinear impact, liquidity, partial-fill, fee, expiry, gap-aware stop/target, exposure-limit, kill-switch, and reconciliation model

Model governancePoint-in-time datasets with SHA-256 manifests, purged walk-forward splits, deterministic rule and statistical baselines, fold-local calibration, confidence-based abstention, modeled trading costs, block-bootstrap intervals, hashed reports, signed immutable artifacts, model lineage, tamper-evident audit events, aligned shadow evaluation, calibration and PSI drift gates, and evidence-triggered rollback

OperationsPostgreSQL definitions for feed checkpoints, atomic idempotency, transactional outbox, leased quarantine replay, automatic failback, dead letters, and reconciliation; Docker Compose observability plus Terraform for ECS/Fargate

Evidence statusPython controls and SQL definitions are automated-test verified. PostgreSQL and AWS infrastructure are not deployed; no commercial market-data feed is connected.

Verification206 automated tests pass across behavior, datasets, baselines, calibration, drift and shadow monitoring, registry integrity, execution stress, portfolio safety, reconciliation, transactional recovery, operations, and Terraform policy checks

ALERTOfficial bot eventAuthorized channel

→

PARSETyped signalPrice and stop

→

GATERisk policyAccept or reject

→

SIMULATEPaper fillNo live broker

→

OBSERVEAudit and metricsScorecards

CI/CD

Fail-closed verification

GitHub Actions runs 206 behavioral and integration tests, the independent Terraform policy scanner, Docker Compose validation, Terraform formatting and validation, and a commit-addressed Docker build.

ROLLOUT & RECOVERY

Shadow first, promote with evidence

Incident readiness: champion–challenger shadow windows monitor calibration, PSI drift, expectancy, drawdown, and regimes. Signed promotion gates, kill switches, reconciliation, incident evidence, and exact-model rollback constrain paper operation.

SYSTEM HEALTH

Operational failure signals

Feed mode, timestamp integrity, recovery streak, quarantine backlog, latency, quote quality, queue retries, dead letters, and reconciliation checkpoints expose data and service health.

DEPLOYMENT STATUS

Implemented controls; definitions remain definitions

Adversarial promotion gate: stress scenarios must pass before promotion. Python and SQL controls are test-verified. PostgreSQL, AWS/Fargate, Multi-AZ recovery, and immutable evidence storage are definitions—not deployed services.