TECHNICAL BRIEF · APPLICATION
Payment, fulfillment, and recovery for an eSIM commerce workload.
A Go and PostgreSQL system integrating hosted payments, external suppliers, per-unit provisioning, and support operations. Its main design problem is not checkout. It is making supplier side effects recoverable when processes and callbacks fail.
- Runtime
- Go modular monolithReact storefront + admin; PostgreSQL source of truth
- Integration boundary
- Four supplier adaptersPayments, accounting, eSIM activation, inventory/package state
- Operational model
- Durable jobs + reconciliationAt-least-once delivery; idempotent completion and operator recovery
React surfaces
Storefront: browse, checkout, order/QR. Admin: orders, fulfillment repair, alerts, history.
buyer + operatorGo modular monolith
HTTP handlers, domain services, session/CSRF controls, correlation IDs, six background runners.
validate + decidePostgreSQL
Orders, payment attempts, per-unit fulfillments, inventory cache, durable jobs, append-only lifecycle history.
source of truthRepository-evidenced delivery
667 commits from 12 March through 18 August 2026 record an iterative delivery path: customer-facing flows, test and release controls, canonical fulfillment recovery, then health and throughput work.
This is the source-controlled delivery phase within the wider independent engineering period.
Research and architecture. Product definition, payment/eSIM provider evaluation, and platform design.
Commerce foundation. Go/PostgreSQL monolith, payment/eSIM/accounting adapters, deterministic provider mocks, durable jobs, inventory sync, and integration/E2E tests.
Release and recovery controls. Staging deployment, readiness gates, race detection, PostgreSQL integration testing, rollback, encrypted R2 backups, and canonical fulfillment/event history.
Platform integration. Workload CI moved onto the ARC runner pool; runner and backup tooling stabilized for self-hosted execution.
Operational hardening. Payment recovery, public-order protection, reconciliation replay coverage, separate health probes, and a measured PostgreSQL queue baseline.
Application mechanisms
Each component exists to make a risky state transition explicit: money verification, supplier provisioning, or recovery from an ambiguous remote outcome.
External calls remain outside local database transactions. This is compensating recovery, not a distributed transaction.
-
Checkout ledger + payment verification
Mechanism: a client idempotency key binds checkout to one order and payment-attempt state; callback receipts are deduplicated and re-verified at the gateway. Why: double-clicks, retries, and unsigned callbacks must not create another paid order. How: persist before redirect, unique constraints, server-side verification, and polling fallback.
Solves: one verified payment completes one order; duplicate callbacks cannot create another completion. -
PostgreSQL jobs + durable fulfillment intent
Mechanism:
FOR UPDATE SKIP LOCKEDclaims and fenced leases process per-unit work; selected eSIM and package intent are stored before supplier calls. Why: workers and providers can fail between steps. How: at-least-once delivery, provider-state checks, and idempotent adopt-or-retry.Solves: crash-safe, idempotent provisioning over at-least-once delivery. -
Reconciliation + operator recovery
Mechanism: a five-minute sweep selects aged fulfillments and paid orders with incomplete downstream work. Why: remote outcome and local persistence can disagree. How: query provider truth; adopt a completed action, retry recorded intent, or hold state and raise a deduplicated operator alert.
Solves: provider drift becomes a bounded recovery path, not an invisible ticket queue.
Representative recovery trace
A compact example of the ambiguous case the system is designed to handle. IDs and timestamps are illustrative.
The normal lifecycle is recorded in append-only event history. Alert rows are a separate mutable operational overlay.
Verified payment enqueues fulfillment for one purchased unit.
Activation/package call reaches the provider. The worker stops before local completion persists.
Reconciliation sees a fulfillment older than the 15-minute threshold and queries supplier state.
Provider reports the package active. The service adopts it locally and appends a fulfillment event.
Order remains fulfilled. No duplicate package purchase. No operator action required.
Guardrails
- Bounded work: 10 fulfillments and 50 stuck orders per sweep.
- Age thresholds: 15 minutes for fulfillment, 30 for paid orders.
- Fail closed: provider read failure leaves local state unchanged.
- Human boundary: persistent cases become deduplicated inbox alerts. No destructive cleanup is automatic.
- History: assignment, failure, detach, recycle, and correction remain queryable over time.
Capacity boundaries and known limits
No published HTTP/API load test exists. The queue mechanism is measured against real ephemeral Postgres; the rest of the values below are intentional configuration boundaries, not throughput claims.
Measurement date: 2026-08-17. Local Linux workstation, Postgres 15 in Lima, command-scoped SSH tunnel.
| Scope | Measured or configured result | Meaning |
|---|---|---|
| Concurrent queue claim | 710 jobs/s 8 workers, batch 10 | Real SKIP LOCKED claim path. Queue partitioning works without a coordinator. |
| Full queue cycle | 55 jobs/s claim → handler → completion | Real Runner and lease bookkeeping, using a no-op handler to isolate queue cost. |
| Contention correctness | 0 lost · 0 duplicate completions 400 jobs, 12 workers | Proves no duplicate completion under this test shape. Delivery remains at-least-once by design. |
| Payment recovery | 100 attempts / 5 min | Configured polling batch. Lost callbacks drain across consecutive cycles rather than disappearing. |
| Reconciliation | 10 fulfillments + 50 orders / 5 min | Configured bounded recovery work. Provider calls have 30-second timeouts. |
What this does not claim: HTTP requests-per-second, an availability SLA, multi-region operation, or exactly-once delivery. The next measurement is an HTTP-tier load harness; it is not needed to demonstrate current queue correctness or recovery behavior.
Current scope
The core transaction and recovery paths are operational. The remaining work is deliberate sequencing, not an attempt to disguise instability.
Hosted payments and supplier calls are production integrations. Provider identities, customer data, credentials, and production captures are excluded.
In place
- Hosted payment verification, polling fallback, and duplicate protection
- Per-unit fulfillment with durable intent and provider-state adoption
- Invoice and fulfillment branches with independent retry and operator repair
- Append-only lifecycle history, alert inbox, readiness, runner health, and release rollback
Deliberately next
- HTTP-tier load harness and repeatable soak run
- Automated gateway refund execution; requests and accounting workflow are modeled, money movement remains manual
- Second provider onboarding through the existing adapter seam
- Application runtime migration to k3s only if it earns its operational cost
Technical walkthrough: I can show the delivery gates, recovery boundaries, queue baseline, and reconciliation behavior end to end.