Webhook Integration for Email Archival | MailParse

Introduction

Webhook integration enables real-time email delivery into your archival system, which is critical when audit trails, discovery, and search are non-negotiable. Instead of polling or building complex IMAP bridges, a webhook-first design streams parsed messages, headers, and attachments to your backend the moment they arrive. With retry logic, signature verification, and idempotent processing, teams can guarantee that each email is stored, indexed, and discoverable without gaps.

For developer teams, the payoff is faster ingestion, simpler pipelines, and strong guarantees around data integrity. A webhook-integration fed with structured MIME parsing yields predictable JSON, which can be persisted to object storage and search indices for long-term email-archival, compliance, and legal holds. Platforms like MailParse make this pattern straightforward, combining instant inboxes with signed webhook delivery and structured payloads that map cleanly to storage and indexing engines.

Why Webhook Integration Is Critical for Email Archival

Archival is only as strong as your intake. Webhooks provide determinism, speed, and verifiability that traditional mailbox scraping lacks.

• Real-time ingestion - Emails stream into your system within seconds, which supports time-sensitive compliance and rapid investigations.
• Reliability through retries - Automatic retry logic brings resilience if your endpoint is temporarily unavailable. Idempotent handlers ensure no duplication during replays.
• Payload signing - HMAC-signed requests and timestamp checks protect against tampering, spoofing, and replay attacks. This is foundational for audit-grade archives.
• Structured MIME parsing - Inbound messages are normalized into JSON with clear sections for headers, parts, inline content, and attachments. Clean structure simplifies indexing and downstream automation.
• Lower operational overhead - No IMAP maintenance, no polling intervals, no mailbox state drifts. You focus on storing, indexing, and serving search queries.
• Compliance alignment - Immutable raw storage, metadata normalization, and policy-driven retention and legal holds satisfy regulatory frameworks while keeping workflows maintainable.

Combined, these properties raise your archival pipeline from best effort to dependable infrastructure.

Architecture Pattern

A robust email-archival pipeline usually follows a streaming architecture where the webhook acts as the intake valve and the rest of the stack provides durability, indexing, and query capabilities.

Core components

• Inbound email service - Receives mail, parses MIME, signs payloads, and pushes to your webhook endpoint in near real-time. MailParse is a common choice for developer teams.
• Webhook receiver - Stateless HTTP service that verifies signatures, enforces idempotency, and enqueues work for downstream processing.
• Message queue - Smooths spikes, isolates transient failures, and allows controlled concurrency.
• Durable storage tiers:
  • Object storage for raw MIME and attachments. Use content-addressed keys for deduplication and integrity checks.
  • Relational or document database for normalized metadata, headers, and routing information.
  • Search index (for example, Elasticsearch or OpenSearch) for full-text queries across subject, body, headers, and attachment text.
• Processing workers - Extract, transform, load. Perform attachment text extraction, file type detection, and PII redaction as needed.
• Compliance services - Retention policies, legal holds, access controls, and audit logging.

Data flow

1. Inbound service parses the email and triggers webhook delivery with signed headers.
2. Receiver validates the request, computes an idempotency key, and acknowledges quickly.
3. Worker consumes from the queue, persists raw MIME first, then writes normalized metadata and attachments, then indexes searchable content.
4. Compliance layer applies retention schedules and legal holds, adds audit log entries, and enforces authorization checks for retrieval.

This pattern provides clear separation of concerns. Your webhook remains thin, your workers scale horizontally, and your storage and indexing layers are optimized for their specific roles.

Step-by-Step Implementation

1) Configure webhook delivery

• Choose a public HTTPS endpoint such as POST https://archive.example.com/webhooks/email.
• Set a signing secret and a header scheme with a timestamp and signature, for example X-Webhook-Timestamp and X-Webhook-Signature.
• Set retry parameters. Common defaults are exponential backoff, jitter, and a retry cap. Ensure the sender only retries on non-2xx responses.

Provider configuration is usually minimal. With MailParse, you point an inbox or a domain catch-all to your webhook and enable payload signing with a single secret.

2) Implement a thin, fast receiver

The receiver’s job is to verify authenticity, validate freshness, and enqueue. Do not do heavy processing on this path. Return 2xx only after verification and enqueue succeed.

// Pseudocode
function handleWebhook(req, res) {
  const ts = req.headers['x-webhook-timestamp'];
  const sig = req.headers['x-webhook-signature'];
  const rawBody = req.rawBody; // exact byte string

  // Reject stale requests, for example older than 5 minutes
  if (Math.abs(now() - parseInt(ts)) > 300000) return res.status(400).end();

  // Compute HMAC using your shared secret
  const expected = hex(hmacSha256(secret, ts + '.' + rawBody));
  if (!timingSafeEqual(expected, sig)) return res.status(401).end();

  // Idempotency key derived from provider event id or message-id
  const idempotencyKey = req.headers['x-event-id'] || req.body.headers['message-id'];
  if (seenBefore(idempotencyKey)) return res.status(200).end();

  enqueue('email-archive', {key: idempotencyKey, payload: req.body});
  markSeen(idempotencyKey);
  return res.status(202).end();
}

3) Understand the parsed payload

A high-quality payload exposes the MIME tree and useful metadata. Expect a JSON structure similar to the following:

{
  "event_id": "evt_01HV...",
  "received_at": "2026-04-24T10:25:11Z",
  "envelope": {"to": ["archive@yourdomain.com"], "from": "client@example.com"},
  "headers": {
    "message-id": "<CAF1234@example.com>",
    "date": "Wed, 24 Apr 2026 10:25:09 +0000",
    "from": "Client <client@example.com>",
    "to": "Archive <archive@yourdomain.com>",
    "subject": "SOW - Q2 Renewal",
    "dkim-signature": "...",
    "received": ["... hop 1 ...", "... hop 2 ..."]
  },
  "mime": {
    "content_type": "multipart/mixed",
    "parts": [
      {
        "content_type": "multipart/alternative",
        "parts": [
          {"content_type": "text/plain; charset=utf-8", "body_text": "Hello..."},
          {"content_type": "text/html; charset=utf-8", "body_html": "<p>Hello</p>"}
        ]
      },
      {
        "content_type": "application/pdf",
        "filename": "SOW.pdf",
        "content_id": null,
        "size": 182344,
        "sha256": "a7b8...",
        "storage_ref": null
      }
    ]
  },
  "raw_mime": {"size": 274321, "sha256": "9f3c...", "download_url": "..." }
}

For archival, the most important items are the immutable raw MIME, high-fidelity headers, standardized body parts, and attachment metadata.

4) Persist raw MIME first

• Write the raw bytes to object storage with a content-addressed key, for example emails/raw/9f/3c/9f3c...eml.
• Store hash algorithms and sizes for integrity checks. Re-verify hashes during background scrubs.
• Record a pointer to the raw object in your metadata database along with the event id and message-id.

5) Store normalized metadata

• Create a row or document for the message with envelope addresses, timestamp, subject, and critical headers such as Message-ID, DKIM-Signature, and Received chain.
• Normalize addresses and domains, for example lowercase, punycode conversions, and Unicode normalization.
• Track threading via In-Reply-To and References headers if present to support conversation-level legal review.

6) Persist attachments and inline parts

• For each attachment, compute a SHA-256 and store to object storage. Keep metadata including filename, content-type, size, content-id, and disposition.
• Run content-type detection to avoid trusting filenames. Consider scanning for malware before indexing or serving downloads.
• For inline images, store content-id mappings so HTML bodies can be reconstructed during review.

7) Index for search and discovery

• Extract text from both text/plain and text/html parts, sanitize HTML, and strip boilerplate signatures where possible.
• Perform OCR or PDF text extraction on attachments as policy allows. Index the text with pointers back to storage refs.
• Index key headers and structured fields for filtering: sender domain, authentication results, SPF, DMARC alignment, and Received hops.

8) Apply retention and legal holds

• Assign lifecycle policies: standard retention windows, archival transitions to cold storage, and deletion schedules.
• Legal holds override deletion. Store hold reasons and audit all changes. Expose a search interface that honors holds and access controls.

9) Return consistent API responses

Your webhook should return HTTP 202 Accepted after enqueue, not after full processing. If verification fails, respond with 4xx. If your queue or storage is down, respond with 5xx so the sender retries. With MailParse, 2xx will mark the event delivered and anything else will trigger retry logic with backoff.

Testing Your Email Archival Pipeline

Testing email-based workflows is about coverage across formats, size, and encoding edge cases. Build repeatable, automated suites and a replay harness.

Message diversity

• Plain text and HTML multipart, including tricky charsets like ISO-2022-JP and windows-1252.
• Attachments with different encodings: base64, quoted-printable, and binary. Include PDFs, DOCX, CSV, and images with EXIF data.
• Inline images referenced by Content-ID and embedded CSS in HTML bodies.
• Large messages, for example 25 MB with multiple attachments, to test timeouts and streaming uploads.
• Authentication variations: DKIM-signed, SPF pass and fail cases, DMARC aligned and misaligned.

Functional tests

• Signature verification - Send a known payload with a correct HMAC and an altered one to ensure rejection.
• Idempotency - Replay the same event id and confirm only a single archive entry is present.
• Retry handling - Force the webhook to return 500 to trigger retries, then switch to 202 and validate exactly-once results.
• Integrity checks - Corrupt a stored object and verify that background scrubs flag the mismatch.

Performance and resilience

• Load testing - Simulate bursts to validate queue depth limits, worker autoscaling, and storage throughput.
• Fault injection - Drop database connections, throttle the indexer, and observe graceful degradation without data loss.
• Latency budgets - Track time from provider delivery to searchable index. Set an SLO, for example p95 under 30 seconds.

Replay tooling

Maintain a CLI or admin API that can re-deliver past events to your webhook or reprocess stored raw MIME to rebuild metadata and indices. This helps with schema migrations, bug fixes, and backfills.

Production Checklist

• Security
  • Rotate webhook signing secrets regularly. Keep a dual-valid window to avoid downtime.
  • Enforce TLS 1.2 or higher, validate hostnames, and pin allowed IP ranges if provided.
  • Encrypt data at rest for both object storage and databases. Use KMS-managed keys and limit access with IAM policies.
• Reliability
  • Ensure webhook returns quickly. Use async queues. Timeouts under 2 seconds are ideal.
  • Implement exponential backoff, jitter, and a maximum retry horizon. Surface a dead-letter queue for poison messages.
  • Use idempotency keys derived from event ids or Message-ID, plus content hashes to guard against rare duplicates.
• Observability
  • Metrics: delivery attempts, success rate, retry counts, queue depth, processing latency, and index lag.
  • Logs: signed headers, verification outcomes, storage object ids, and dedup decisions. Mask PII where required.
  • Tracing: propagate correlation ids from webhook to workers and storage writes for end-to-end visibility.
• Data management
  • Lifecycle rules: move cold data to cheaper storage and keep hot metadata in fast databases.
  • Attachment deduplication using hashes. Reference counting prevents accidental deletion.
  • Schema versioning in the index. Use aliases for zero-downtime reindexing and backfills.
• Compliance and governance
  • Legal holds with immutable flags and audit logs on all retention changes.
  • Access control that restricts sensitive mailboxes and applies need-to-know policies.
  • Export tooling for discovery that packages raw MIME, metadata, and chain-of-custody logs.
• Runbooks and readiness
  • Runbooks for webhook outage, queue backlog, indexer failures, and storage corruption.
  • Game days that exercise replay and backfill processes.
  • Staging environment that mirrors production secrets and signature configs with separate keys.

For broader architecture context, review the Email Infrastructure Checklist for SaaS Platforms and consider automation patterns from Top Inbound Email Processing Ideas for SaaS Platforms. If your archival also feeds support queues, the Email Infrastructure Checklist for Customer Support Teams is a useful companion.

Conclusion

Webhook integration is the most direct path to dependable email-archival. With signed payloads, retries, and idempotent processing, teams can capture every message and attachment, store raw bytes immutably, and index rich metadata for search and compliance. By separating the thin webhook receiver from durable storage and scalable workers, your system remains simple to operate and easy to evolve as requirements grow.

Developer platforms like MailParse reduce the heavy lifting by delivering structured MIME as JSON, handling real-time delivery with robust retry behavior, and supporting payload signing out of the box. Pair those capabilities with disciplined storage, indexing, and compliance practices, and your organization gains a reliable, audit-ready archive that stands up to scale and scrutiny.

FAQ

How should I verify webhook signatures for security?

Use a shared secret to compute an HMAC of the timestamp and raw request body, then compare with the signature header using a timing-safe comparison. Enforce a short timestamp tolerance to prevent replay attacks. Reject any request that fails verification or exceeds your freshness window.

What data should I store to make emails discoverable?

Keep three layers: immutable raw MIME, normalized metadata for headers and participants, and a search index that includes subject, body text, extracted attachment text, and key headers like Message-ID and DKIM. Maintain strong pointers between these layers for efficient exports and audit trails.

How do retries avoid duplicate records?

Assign an idempotency key derived from the provider event id or Message-ID. Check this key before enqueueing and persist a record of processed keys. Combine that with content hashing of raw MIME and attachments to detect rare collisions and support deduplication.

What is the best way to handle large attachments?

Stream uploads directly to object storage to avoid memory spikes. Verify hashes after upload, store content-type and size, and delay search indexing until storage is acknowledged. Apply malware scanning and file type detection before making attachments available to end users.

Can the same pipeline support legal holds?

Yes. Implement a policy engine that can mark records as on hold. Holds should prevent lifecycle deletions, adjust access controls, and be reflected in audit logs. Ensure that search and export tools respect hold flags and provide a complete chain of custody.