Concord
Distributed Order Processing System

About the project

Concord is a distributed order-processing system built to prove out, from first principles, how to keep data consistent across independently-owned microservice databases when a multi-step business transaction can fail partway through — and to prove those guarantees hold up under real Kubernetes-level failure, not just a local demo. Three independent Spring Boot services (Order, Inventory, Payment) each own their own PostgreSQL schema: no shared database, no two-phase commit.

What I'm building (phased roadmap)

• Transactional Outbox — each service writes its business-data change and an outbox row in the same local transaction; a poller relays outbox rows to Kafka, so the DB write and the event publish can never silently diverge on crash.
• Saga orchestration — a central orchestrator coordinates the reserve → charge → confirm sequence and triggers compensating actions (release inventory, refund payment) if any step fails, including a failure after a compensation has partially run.
• Idempotent consumers — Redis-backed processed-event tracking so a duplicate or redelivered Kafka message cannot double-charge or double-reserve stock.
• Fault-injection testing — a Testcontainers-based integration test that deliberately stops a service mid-saga and asserts the orchestrator detects the failure, compensates correctly, and leaves the order in a terminal FAILED state rather than a stuck PENDING.
• Containerization — multi-stage Dockerfiles per service and a Docker Compose environment wiring all services with Postgres, Kafka and Redis for local development.
• Kubernetes deployment — hand-written manifests (Deployment, Service, ConfigMap, Secret, readiness/liveness probes) built first for understanding, then packaged as a Helm chart; a HorizontalPodAutoscaler added and load-tested to justify its presence.
• Cloud infrastructure — a minimal real cluster (EKS/GKE, or a single-VM k3s to control cost) provisioned with Terraform, with the Helm chart deployed to it.
• CI/CD — a GitHub Actions pipeline running tests on every pull request and building, pushing, and deploying images on merge to main.
• Observability — OpenTelemetry distributed tracing so a single order's saga can be followed end-to-end across all three services, plus Prometheus and Grafana dashboards tracking saga success/failure and compensation-trigger rates — not just generic infrastructure metrics.
• Platform-level chaos testing — killing a live pod mid-saga in the real cluster and confirming both Kubernetes' rescheduling and the saga's application-level recovery logic hold up together.

Feature highlights

• Transactional Outbox pattern for atomic DB-write + event-publish.
• Orchestration-based Saga with explicit, idempotent compensating transactions.
• Redis-backed idempotent event consumers.
• Testcontainers fault-injection test proving recovery from a mid-transaction service failure.
• Kubernetes deployment via hand-written manifests and a Helm chart, with a load-tested HPA.
• Terraform-provisioned cloud infrastructure and GitHub Actions CI/CD.
• OpenTelemetry tracing and Prometheus/Grafana observability focused on business-level saga metrics.
• Platform-level chaos test proving recovery under a real Kubernetes pod failure.