Webhook Processing Engine

Overview

The Webhook Processing Engine is a hardened, production-ready ingestion pipeline designed for teams that cannot afford to lose a single event. Whether you are integrating with Stripe, GitHub, Twilio, or any third-party service that pushes webhooks, this system gives you the confidence that every payload will be received, verified, and processed exactly once.

Built on Node.js and TypeScript with Redis-backed queuing and PostgreSQL persistence, the engine handles the full lifecycle of webhook processing: receiving, validating, deduplicating, dispatching, retrying, and dead-lettering. It has been designed from the ground up with observability in mind, so you always know the state of every event flowing through your system.

Architecture

The engine follows a staged pipeline architecture:

Ingestion Layer -- An HTTP endpoint receives incoming webhooks, captures the raw payload and headers, and immediately returns a 202 Accepted response. This decouples receipt from processing and ensures third-party services never time out waiting for your business logic.
Verification Stage -- Each provider has a pluggable signature verifier. HMAC-SHA256, RSA, and Ed25519 strategies are included out of the box. Custom verifiers can be added by implementing a single interface.
Deduplication & Idempotency -- A Redis-backed idempotency store checks each event's unique identifier before processing. Duplicate deliveries (which are common with Stripe, for example) are silently discarded with full audit logging.
Processing Queue -- Verified, unique events are enqueued onto a BullMQ job queue with configurable concurrency, priority, and rate limiting. Handlers are registered per event type, keeping your business logic cleanly separated.
Retry & Dead-Letter Queue -- Failed jobs are retried with exponential backoff and jitter. After a configurable maximum number of attempts, events land in a dead-letter queue where they can be inspected, replayed, or manually resolved.
Persistence & Audit -- Every event, including its raw payload, processing status, timestamps, and error details, is persisted to PostgreSQL. This gives you a complete, queryable audit trail.

Key Features

Signature Verification -- Verify HMAC-SHA256 (Stripe, GitHub), RSA (Twilio), or Ed25519 signatures. Add custom verifiers with a single class.
Idempotent Processing -- Redis-backed deduplication ensures each event is processed exactly once, even when providers send duplicates.
Dead-Letter Queue -- Events that fail after all retry attempts are moved to a DLQ with full context, allowing manual inspection and replay.
Rate Limiting -- Per-provider rate limits prevent downstream services from being overwhelmed during burst traffic.
Monitoring Hooks -- Built-in hooks for logging, metrics emission, and alerting. Integrates with Prometheus, Datadog, or any custom exporter.
Graceful Shutdown -- In-flight jobs are allowed to complete before the process exits, preventing data loss during deployments.
Provider Presets -- Pre-configured handlers for Stripe, GitHub, Shopify, and Twilio. Adding a new provider takes minutes.

What's Included

Full TypeScript source code with strict type checking
Comprehensive test suite (unit and integration tests with >90% coverage)
Docker Compose setup for local development (Node.js, Redis, PostgreSQL)
Database migration files (Prisma)
API documentation with OpenAPI 3.0 spec
Deployment configuration for Railway, Render, and Docker
Environment variable reference and configuration guide

Usage Example

import { WebhookEngine } from "./engine";
import { StripeVerifier } from "./verifiers/stripe";

const engine = new WebhookEngine({
  redis: process.env.REDIS_URL,
  database: process.env.DATABASE_URL,
});

engine.registerProvider("stripe", {
  verifier: new StripeVerifier(process.env.STRIPE_WEBHOOK_SECRET),
  handlers: {
    "checkout.session.completed": handleCheckout,
    "invoice.payment_failed": handleFailedPayment,
  },
});

engine.start({ port: 3001 });

Who Is This For?

This system is built for backend engineers and teams who are integrating with multiple third-party services and need a reliable, observable, and maintainable way to handle incoming webhooks. If you have ever lost events because your webhook handler threw an unhandled exception, or spent hours debugging duplicate processing, this engine solves those problems permanently.

It is especially valuable for teams building payment systems, marketplace integrations, or any event-driven architecture where webhook reliability is a business-critical requirement.

Overview

Architecture

The engine follows a staged pipeline architecture:

Ingestion Layer -- An HTTP endpoint receives incoming webhooks, captures the raw payload and headers, and immediately returns a 202 Accepted response. This decouples receipt from processing and ensures third-party services never time out waiting for your business logic.

Verification Stage -- Each provider has a pluggable signature verifier. HMAC-SHA256, RSA, and Ed25519 strategies are included out of the box. Custom verifiers can be added by implementing a single interface.

Deduplication & Idempotency -- A Redis-backed idempotency store checks each event's unique identifier before processing. Duplicate deliveries (which are common with Stripe, for example) are silently discarded with full audit logging.

Processing Queue -- Verified, unique events are enqueued onto a BullMQ job queue with configurable concurrency, priority, and rate limiting. Handlers are registered per event type, keeping your business logic cleanly separated.

Retry & Dead-Letter Queue -- Failed jobs are retried with exponential backoff and jitter. After a configurable maximum number of attempts, events land in a dead-letter queue where they can be inspected, replayed, or manually resolved.

Persistence & Audit -- Every event, including its raw payload, processing status, timestamps, and error details, is persisted to PostgreSQL. This gives you a complete, queryable audit trail.

Key Features

Signature Verification -- Verify HMAC-SHA256 (Stripe, GitHub), RSA (Twilio), or Ed25519 signatures. Add custom verifiers with a single class.

Idempotent Processing -- Redis-backed deduplication ensures each event is processed exactly once, even when providers send duplicates.

Dead-Letter Queue -- Events that fail after all retry attempts are moved to a DLQ with full context, allowing manual inspection and replay.

Rate Limiting -- Per-provider rate limits prevent downstream services from being overwhelmed during burst traffic.

Monitoring Hooks -- Built-in hooks for logging, metrics emission, and alerting. Integrates with Prometheus, Datadog, or any custom exporter.

Graceful Shutdown -- In-flight jobs are allowed to complete before the process exits, preventing data loss during deployments.

Provider Presets -- Pre-configured handlers for Stripe, GitHub, Shopify, and Twilio. Adding a new provider takes minutes.

What's Included

Full TypeScript source code with strict type checking

Comprehensive test suite (unit and integration tests with >90% coverage)

Docker Compose setup for local development (Node.js, Redis, PostgreSQL)

Database migration files (Prisma)

API documentation with OpenAPI 3.0 spec

Deployment configuration for Railway, Render, and Docker

Environment variable reference and configuration guide

Usage Example

import { WebhookEngine } from "./engine"; import { StripeVerifier } from "./verifiers/stripe"; const engine = new WebhookEngine({ redis: process.env.REDIS_URL, database: process.env.DATABASE_URL, }); engine.registerProvider("stripe", { verifier: new StripeVerifier(process.env.STRIPE_WEBHOOK_SECRET), handlers: { "checkout.session.completed": handleCheckout, "invoice.payment_failed": handleFailedPayment, }, });

engine.start({ port: 3001 });

Who Is This For?

It is especially valuable for teams building payment systems, marketplace integrations, or any event-driven architecture where webhook reliability is a business-critical requirement.