Notification Gateway - Multi-Platform Payment System

Three payment platforms, three home-grown ways of telling someone a payment had failed. The brief for this engagement was to replace all of them with one gateway — a single Open API that turns a payment error into a delivered notification across email, SMS, webhook and in-app channels, and keeps doing so while the platforms around it change. Four months in 2024, a team of three to four; I was the senior full-stack developer.

The brief

PAYMENT ARINDO, PAPRIKA and ASSASTA each carried their own notification code. The same concerns — retries, provider credentials, status tracking — were implemented three times in three styles, and every new platform would mean a fourth. Operations had no single answer to the only question that matters when a payment fails: was anyone told?

The engagement called for a centralized notification gateway: one service that accepts error and status events from any payment platform, routes them to the right channels under per-platform rules, retries until delivery is confirmed or escalated, and exposes all of it through an Open API designed for platforms that did not exist yet. Around it, a Next.js dashboard for live delivery status, error analytics and runtime configuration.

Constraints

• Delivery guarantees. A dropped payment-error notification is a failed payment nobody investigates. Acceptance had to mean at-least-once delivery: persist first, acknowledge second, retry with exponential backoff until a channel confirms — and surface anything that exhausts its retries.
• Provider flakiness. Email and SMS providers rate-limit, time out and degrade partially; the third-party hop is the least reliable link in the chain. A misbehaving provider could not be allowed to block ingestion or starve the other channels.
• Multi-channel routing. The same event fans out differently per platform — webhook and email for one, SMS and in-app for another — and administrators change those rules at runtime. Routing had to be configuration, not code.
• No maintenance window. Payment traffic runs around the clock. Releases had to ship into live traffic without dropping in-flight notifications, and we shipped throughout the four months.
• Peak-hour bursts. Notification volume spikes with payment volume. We set the design target at sustaining 10K+ notifications a minute, with horizontal scaling as the lever rather than vertical headroom.
• A small team operates it. Three to four people built and ran this, so boring, observable patterns won. Every operational question — latency percentiles, per-platform error rates, delivery success — had a dashboard before launch.

Approach & decisions

The shape of the system is a thin synchronous edge over an asynchronous core. A producer hands the gateway an event and gets back an acknowledgement and a notification ID within a tight response budget; everything that can fail slowly — providers, retries, fan-out — happens behind a queue, where slowness is absorbed instead of propagated.

A service, not a shared library

The first decision was what kind of thing this should be. I rejected two alternatives before writing code. Leaving notification logic inside each platform — the status quo — meant every reliability improvement landed three times and drifted immediately. Packaging the logic as a shared internal library was more tempting: one implementation, familiar distribution. But a library still couples every fix to three coordinated releases, and the operational behaviour we needed — queues, circuit breakers, independent scaling — does not live comfortably inside someone else's process.

So: a standalone NestJS service behind a versioned Open API. An adapter per platform normalizes inbound payloads at the edge, and a strategy per channel isolates delivery quirks on the way out. Onboarding a new platform is an adapter and an API contract — the integration work that used to be repeated per provider, per platform, now happens once, in one place.

A queue between accept and deliver

The defining constraint was that provider flakiness must never become caller latency. Synchronous fan-out in the request path was rejected first: one slow SMS provider would push every caller past its timeout. I also considered building the pipeline on Redis Streams, since Redis was already in the stack — workable, but per-channel routing, dead-lettering and acknowledgement semantics are RabbitMQ's native vocabulary, and I did not want to re-implement broker behaviour in application code.

The accept path stays deliberately thin — validate, persist, enqueue, respond — and was held to a sub-100ms response budget. Dispatch workers consume per channel, retry with exponential backoff, and trip a per-provider circuit breaker so a degraded provider sheds load instead of consuming it. Redis stayed for what it is good at: hot routing rules and rate counters.

Delivery status as a record, not a log line

"Was anyone told?" had to be answerable per notification, months later. Every accepted event becomes a row in PostgreSQL (via Prisma) with explicit status transitions — accepted, dispatched, then delivered or failed per channel. Producers read that status two ways: a polling endpoint and signed webhooks back into their own systems. I rejected deriving status from centralized logs; the ELK stack stayed for forensics, but a contract deserves a schema, not a query over log lines. The Next.js dashboard reads the same ledger — live status over WebSocket, error analytics per platform.

Releases without a maintenance window

There is no quiet hour in payments, so the maintenance-window option was rejected outright. Deployments roll on Kubernetes with zero unavailable replicas, gated by readiness probes; shutdown is graceful in the precise sense — a terminating worker stops consuming, drains its in-flight deliveries, then exits. Failed health checks roll back automatically.

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 0

That fragment is the whole philosophy: capacity may briefly exceed target during a deploy, but it may never dip below it.

Evidence

A note on the numbers: this engagement predates my discipline of capturing client-attributable metrics, so I state only what the system itself supports and label design goals as design goals.

• System facts. One gateway unifies notification delivery for three payment platforms — PAYMENT ARINDO, PAPRIKA and ASSASTA — behind a single versioned Open API, fanning out to four channels: email, SMS, webhook and in-app. One integration replaces per-provider delivery work; a new platform onboards with an adapter and a contract, not a rebuild.
• Design goals. Designed to sustain 10K+ notifications per minute through horizontal scaling and queue buffering; the synchronous accept path was held to a sub-100ms response budget; every release shipped under a zero-downtime strategy — rolling updates with maxUnavailable: 0 and connection draining — from the first deploy.
• Instrumentation. Latency percentiles (p50/p95/p99), per-platform error rates and per-channel delivery success were tracked in Prometheus and Grafana from launch. The operational questions had dashboards, not anecdotes.

Stack notes

• Core service: NestJS and TypeScript. Repository, adapter and strategy patterns keep platform- and channel-specific code at the edges; Swagger/OpenAPI documentation is served by the service itself, with a sandbox environment for integrators.
• Data: PostgreSQL with Prisma as the ledger of record; Redis for the routing-rule cache and rate counters; RabbitMQ for dispatch.
• Dashboard: Next.js with server components and a WebSocket stream for live delivery status and per-platform analytics.
• Operations: Docker and Kubernetes; Prometheus and Grafana for metrics; ELK for logs.
• Security: API keys with request signing and IP allowlisting at the edge; TLS 1.3 in transit; AES-256 at rest; audit logging on administrative actions.