Stackvora.tech

Designing Webhooks Customers Love

Webhooks provide an asynchronous, event-driven mechanism for real-time data exchange between systems. Unlike traditional polling, which inefficiently…

Webhooks provide an asynchronous, event-driven mechanism for real-time data exchange between systems. Unlike traditional polling, which inefficiently consumes resources, webhooks push notifications to subscribers whenever a predefined event occurs. However, effectively designing and implementing webhooks requires careful consideration of reliability, security, and consumer experience, as they inherently rely on external infrastructure.

Payload Design and Structure

The webhook payload is the core data transfer unit. Its design directly impacts usability and integration effort. A robust payload includes essential metadata and a stable representation of the affected resource.

  • Core Metadata: Every webhook event should include standard fields for identification and versioning.
    • id (UUID v4 or similar): A unique identifier for the specific event instance. This is crucial for idempotency and traceability.
    • type (string): A descriptive string identifying the event, e.g., customer.created, invoice.paid, order.updated. Consider a hierarchical naming convention.
    • created_at (ISO 8601 string): The timestamp when the event occurred, in UTC. Example: 2023-10-27T14:30:00Z.
    • api_version (string): The version of your API contract that the webhook payload adheres to, e.g., 2023-10-27 or v2. This allows consumers to handle changes gracefully.
  • Stable Object Snapshot: Always send a complete, stable snapshot of the relevant object's state at the time of the event, not a differential. Consumers should not need to store previous states or make multiple API calls just to reconstruct an object's current state. For example, for an invoice.paid event, include the full invoice object.
  • Concise Payloads, API for Details: Keep the initial webhook payload lean. Include only the most commonly needed data. For richer details or related objects, advise consumers to fetch them via your main API using an included ID. For instance, an order.created webhook might include order_id, customer_id, total_amount, and status. If the consumer needs the full line item details, they can make an authenticated GET request to /api/v1/orders/{order_id}.
  • Consistent Schemas: Document your payload schemas meticulously, ideally using OpenAPI/JSON Schema, and adhere to them. Avoid introducing breaking changes to existing event types.

Security Best Practices

Webhooks deliver sensitive data to arbitrary endpoints, making security paramount. Protecting against tampering, replay attacks, and unauthorized access is critical.

Request Signing and Verification

Every webhook request must be signed to allow consumers to verify its authenticity and integrity.

  1. HMAC-SHA256 Signature: Generate a signature using HMAC-SHA256. The secret key is unique to each consumer's endpoint. The signed content should be the raw request body. Some implementations also include the timestamp in the signed data to prevent timestamp tampering.
  2. Signature Header: Send the signature in a dedicated HTTP header, e.g., X-Webhook-Signature. A common format includes a timestamp and the signature, separated by a comma: t={timestamp},s={signature}. Example: X-Webhook-Signature: t=1678886400,s=a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0u1v2w3x4y5z6a7b8c9d0e1f2.
  3. Timestamp and Replay Protection: Include a timestamp in the signature header (e.g., t=1678886400). Consumers should compare this timestamp to their current time and reject requests where the difference exceeds a small threshold (e.g., 5 minutes). This mitigates replay attacks where an attacker captures and resends an old, valid webhook.
  4. Consumer Verification Process: 
    
# Python example for webhook signature verification
import hmac
import hashlib
import time

WEBHOOK_SECRET = "shhhhh_its_a_secret" # This would be fetched from storage for the given endpoint
REQUEST_BODY = b'{"id": "evt_123", "type": "customer.created", ...}' # Raw bytes of the request body
RECEIVED_SIGNATURE_HEADER = "t=1678886400,s=a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0u1v2w3x4y5z6a7b8c9d0e1f2"

# 1. Parse header
parts = dict(part.split('=', 1) for part in RECEIVED_SIGNATURE_HEADER.split(','))
timestamp = int(parts['t'])
signature = parts['s']

# 2. Check timestamp for replay attacks
current_time = int(time.time())
if abs(current_time - timestamp) > 300: # 5 minutes skew
    raise ValueError("Webhook timestamp is too old or too new, possible replay attack.")

# 3. Construct signed payload
signed_payload = f"{timestamp}.{REQUEST_BODY.decode('utf-8')}".encode('utf-8')

# 4. Compute expected signature
expected_signature = hmac.new(
    WEBHOOK_SECRET.encode('utf-8'),
    signed_payload,
    hashlib.sha256
).hexdigest()

# 5. Compare signatures
if not hmac.compare_digest(expected_signature, signature):
    raise ValueError("Webhook signature mismatch, potential tampering.")

print("Webhook signature verified successfully.")

Secret Rotation

Implement a mechanism for consumers to rotate their webhook signing secrets. Provide an overlap window where two secrets (current and previous) are simultaneously valid. This allows consumers to update their systems without service interruption. For instance, when a new secret is generated, your system signs with both secrets for a configurable period (e.g., 24-48 hours), then drops the old secret. The signature header would contain two s= entries, e.g., t={timestamp},s1={sig1},s2={sig2}.

Delivery Guarantees and Reliability

Your webhook system must be resilient to network failures, consumer endpoint downtime, and processing delays.

At-Least-Once Delivery

Guarantee at-least-once delivery. This means your system will attempt to deliver an event until it receives a successful HTTP status code (2xx) from the consumer. Due to retries, consumers must be prepared to receive the same event multiple times.

Idempotency

Consumers must process events idempotently. They should use the id field of the webhook payload to uniquely identify and deduplicate events. For example, before processing an invoice.paid event, the consumer should check if an invoice with that id has already been marked as paid.

Retry Strategy

Implement an exponential backoff with jitter for failed deliveries. This spreads out retries and prevents thundering herd problems. A typical schedule:

  • 1st retry: 1 minute after failure
  • 2nd retry: 5 minutes after failure
  • 3rd retry: 30 minutes after failure
  • 4th retry: 2 hours after failure
  • 5th retry: 6 hours after failure
  • Further retries: Potentially up to 24-48 hours with increasing intervals.
Define a maximum number of retries (e.g., 10-15 attempts over 24-48 hours) before considering an endpoint as failed.

Endpoint Disabling and Notification

If an endpoint consistently fails (e.g., returns non-2xx status codes for all retries over a 24-hour period), automatically disable it. Notify the owner of the webhook endpoint via email or other communication channels about the persistent failure and disablement. Provide clear instructions on how to re-enable it.

Tools for Consumers

  • Delivery Log: Offer a dashboard where consumers can view a log of all webhook deliveries, their status (pending, sent, failed), HTTP response codes, and payload details.
  • Replay Endpoint/Feature: Provide a mechanism (e.g., a "Replay Event" button in the dashboard or an API endpoint) for consumers to manually re-send past events, particularly useful after an endpoint has been down or for testing.

Consumer Expectations and Documentation

Clear communication and documentation significantly reduce support burden and improve the developer experience.

Quick Responses

Consumers' webhook endpoints should return a 2xx HTTP status code (e.g., 200 OK, 202 Accepted) as quickly as possible, ideally within 1-2 seconds. This acknowledges receipt and prevents your system from retrying. All heavy processing triggered by the webhook should be handled asynchronously in a separate worker process.

Timeouts

Clearly document the timeout duration your system waits for an HTTP response from the consumer's endpoint (e.g., 5-10 seconds). After this duration, the delivery attempt is considered a failure and will be retried according to your schedule.

Source IP Ranges

Publish the range of source IP addresses from which your webhooks will originate. This allows enterprise customers to whitelist these IPs in their firewalls, preventing blocking issues.

Comprehensive Documentation

Your webhook documentation should cover:

  • All available event types and their detailed JSON schemas.
  • Security mechanisms: signature generation, verification, and secret rotation.
  • Delivery guarantees: at-least-once, idempotency requirements, retry schedule.
  • Best practices for consumer endpoints: quick responses, asynchronous processing.
  • Information on source IP ranges, timeouts, and how to use the delivery log and replay features.

Testing and Development Experience

Empowering developers to test and integrate effectively reduces frustration and support tickets.

"Send Test Event" Button

Include a "Send Test Event" button in your developer dashboard. This allows consumers to trigger a sample webhook event (for a selected event type) to their configured endpoint. This is invaluable for initial setup and debugging.

Local Tunneling Guide

Provide documentation and recommendations for local tunneling tools (e.g., ngrok, LocalTunnel, or similar). These tools create a publicly accessible URL that tunnels requests to a developer's local machine, enabling them to receive webhooks during local development.


# Example ngrok command for local webhook testing
ngrok http 8080
# This will output a public URL (e.g., https://your_subdomain.ngrok.io)
# that you can configure as your webhook endpoint for local development.

Sandbox Environment

Offer a dedicated sandbox or staging environment where developers can register webhooks and receive events without affecting live production data. This is crucial for integration testing and new feature development.

Common Pitfalls

  • Synchronous Processing: Consumers attempting to process the entire event synchronously within the webhook handler, leading to timeouts and retries from the webhook sender. Always process asynchronously.
  • Lack of Idempotency: Failure by consumers to design their systems to handle duplicate events, leading to incorrect state or data corruption. Always use the event id for deduplication.
  • Ignoring Security Headers: Consumers not verifying signatures or timestamps, making them vulnerable to malicious requests or replay attacks.
  • Unstable Endpoints: Consumer endpoints that are frequently down, slow, or return inconsistent error codes, causing your webhook system to constantly retry and eventually disable the endpoint.
  • Inadequate Documentation: Vague or missing documentation on event schemas, security, and delivery mechanisms leads to integration headaches and increased support costs.

Back to the knowledge base · Ask the AI assistant