Inbound Email Processing for Backend Developers | MailParse

Why inbound email processing matters for backend developers

Email is a ubiquitous input channel that your users already understand. Turning inbound messages into structured, actionable events lets you power everything from customer support automation to approvals, content ingestion, and on-call workflows. For backend developers, inbound email processing feels a lot like handling webhooks: receive a payload, validate it, persist it, and trigger downstream jobs. The difference is that email payloads are MIME, threads carry conversation context, and attachments can be large or encoded in unexpected ways.

A modern approach treats email as an event source. You receive, route, and process messages programmatically using webhooks or a REST polling API. One provider-agnostic pattern anchors the workflow: parse MIME into normalized JSON, verify authenticity, store content and attachments durably, and hand off to a queue for further processing. If you prefer not to self-host MX infrastructure or wrangle MIME yourself, MailParse provides instant email addresses, structured JSON, and delivery via webhook or REST so your backend can focus on business logic.

Inbound email processing fundamentals for backend developers

Key concepts

Receiving: Messages arrive via MX records to a service or your own mail server. For app routing, use subdomains or plus-addressing so you can map a recipient to an account or workflow.
MIME parsing: Email is a tree of parts, not just a string. You will see multipart/alternative, nested attachments, inline images with Content-ID, and quoted-printable or base64 encodings. Convert this into structured JSON with text, HTML, attachments, and headers.
Routing: Map the recipient address to a tenant, user, or flow. For example, issues+abc123@in.yourapp.com can route to project abc123.
Processing: Apply business logic: create tickets, append comments, run commands parsed from email subject or body, and store attachments.
Delivery mechanism: Webhooks are push-based and low-latency. REST polling is pull-based and resilient when your endpoint is offline. Both need idempotency handling and retries.

Security and authenticity

Sender authentication: Inspect SPF, DKIM, and DMARC results if provided. Do not rely on the From header alone. Use a verified sender list or per-tenant allowlist when executing actions from email.
Webhook verification: Require HMAC signatures or signed JWTs on webhook deliveries. Validate the timestamp to prevent replay attacks and reject bodies that fail signature verification.
Content safety: Sanitize HTML, strip scripts, and treat all attachment content as untrusted. Use antivirus or malware scanning for uploaded files and do not render unsanitized HTML.

Idempotency and retries

Idempotency key: The Message-ID header is globally unique in most cases. Combine it with a hash of the raw body as a fallback to dedupe re-deliveries.
Stateless handlers: Acknowledge webhooks quickly and push work to a queue. If the provider retries on non-2xx, your dedupe key prevents creating duplicates.

Practical implementation patterns

Recommended architecture

Provision a dedicated inbound subdomain like in.example.com.
For each tenant or user, generate an address pattern like inbound+{tenantId}@in.example.com.
On message receipt, your provider parses MIME and POSTs JSON to your webhook. You verify the signature, persist the payload and attachments, enqueue a job, and return 200 in under 300 ms.
Workers perform business logic: thread detection, command extraction, attachment processing, and notifications.

Webhook receiver example - Node.js (Express)

// npm i express crypto body-parser
import express from "express";
import crypto from "crypto";
import bodyParser from "body-parser";

const app = express();

// Raw body is often required for HMAC verification
app.use(bodyParser.raw({ type: "application/json" }));

const SHARED_SECRET = process.env.WEBHOOK_SECRET;

function verifySignature(rawBody, signature, timestamp) {
  const hmac = crypto.createHmac("sha256", SHARED_SECRET);
  hmac.update(timestamp + "." + rawBody);
  const expected = "sha256=" + hmac.digest("hex");
  return crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(signature));
}

app.post("/webhooks/email", (req, res) => {
  const signature = req.header("X-Signature") || "";
  const timestamp = req.header("X-Timestamp") || "";
  const rawBody = req.body.toString("utf8");
  if (!verifySignature(rawBody, signature, timestamp)) {
    return res.status(401).send("invalid signature");
  }

  const event = JSON.parse(rawBody);

  // Idempotency using Message-ID
  const messageId = event.headers["message-id"] || event.message_id;

  // Persist metadata and enqueue async processing
  // Example pseudo calls:
  // await db.upsertMessage({ messageId, event });
  // await queue.publish("email.process", { messageId });

  // Acknowledge quickly
  res.status(200).send("ok");
});

app.listen(3000, () => console.log("listening on :3000"));

Webhook receiver example - Python (FastAPI)

# pip install fastapi uvicorn pydantic hmac
from fastapi import FastAPI, Request, Header, HTTPException
import hmac, hashlib, os, json

app = FastAPI()
SHARED_SECRET = os.environ.get("WEBHOOK_SECRET", "")

def verify_signature(raw_body: bytes, signature: str, timestamp: str) -> bool:
    mac = hmac.new(SHARED_SECRET.encode("utf-8"), digestmod=hashlib.sha256)
    mac.update((timestamp + "." + raw_body.decode("utf-8")).encode("utf-8"))
    expected = "sha256=" + mac.hexdigest()
    return hmac.compare_digest(expected, signature or "")

@app.post("/webhooks/email")
async def handler(request: Request, x_signature: str = Header(""), x_timestamp: str = Header("")):
    raw_body = await request.body()
    if not verify_signature(raw_body, x_signature, x_timestamp):
        raise HTTPException(status_code=401, detail="invalid signature")
    event = json.loads(raw_body)

    message_id = event.get("headers", {}).get("message-id") or event.get("message_id")

    # db_upsert_message(message_id, event)
    # publish_to_queue("email.process", {"messageId": message_id})

    return {"status": "ok"}

REST polling example

Polling is useful when your endpoint cannot accept inbound traffic or you want tighter pull-based flow control. The pattern is straightforward: fetch messages, process, then acknowledge or mark as processed with an idempotency key.

// Pseudo code
GET /api/messages?status=pending&limit=50
// returns an array of messages with cursor or nextToken

for (const msg of messages) {
  // processMessage(msg)
  // POST /api/messages/{id}/ack with idempotency key
}

Storage and schema

Durable storage: Store JSON in a database and attachments in object storage. Use content-addressed keys like sha256/{hash} to dedupe binary payloads.
Threading fields: Persist Message-ID, In-Reply-To, and References for conversation linkage.
Normalized payload: Keep text, sanitized HTML, sender, recipients, subject, headers, and an array of attachments with mime type, size, and a URL.

Tools and libraries that make email ingestion sane

If you ingest raw SMTP and parse MIME yourself, you will need solid tooling. Many teams prefer to offload this and consume normalized JSON via webhook or REST to save time and reduce edge cases.

MIME parsing: Node: mailparser. Python: email and mail-parser. Go: github.com/emersion/go-message and go-imap. These help decode base64, quoted-printable, and multipart structures.
Frameworks for webhooks: Express, FastAPI, Gin, or ASP.NET Minimal APIs. Support raw-body reading for signature verification.
Queues and workflows: SQS, SNS, RabbitMQ, Kafka, Celery, Sidekiq, or Temporal for stateful workflows.
Storage and scanning: S3 compatible object storage, GCS, Azure Blob. Integrate ClamAV or a commercial scanner. Generate presigned URLs for downstream consumers.
HTML sanitization: DOMPurify (with server-side integration), Bleach for Python, or OWASP Java HTML Sanitizer.

If you prefer a ready pipeline that starts at an instant email address and ends with structured JSON, signatures, retries, and attachment links, MailParse abstracts all of that so you can wire the output straight into your workers.

Related deep dives:

Common mistakes and how to avoid them

Doing heavy work in the webhook: Long-running handlers cause timeouts and retries. Acknowledge quickly, queue, then process asynchronously.
Skipping signature verification: Always validate webhook signatures or signed tokens. Enforce short-lived timestamps to prevent replay.
Assuming a single plain-text body: Handle multipart with text, HTML, and inline images. Decide which part you trust and sanitize HTML aggressively.
Ignoring character encodings: Normalize to UTF-8. Decode quoted-printable and base64 correctly. Test with non-ASCII senders and subjects.
Not streaming large attachments: Use streaming uploads to object storage to avoid memory spikes. Set size limits and fail gracefully.
Trusting the From header: Use SPF, DKIM, and tenant-level sender policies before mutating state based on an email.
Weak idempotency: Store Message-ID and a content hash. Treat re-delivery as a first-class path, not an exception.
Naive reply parsing: Many clients add quoted history markers differently. Prefer References or In-Reply-To and use robust quote-stripping if you need only the new content.
Inconsistent routing: Normalize recipient addresses, including case and plus-tags. Enforce a single canonical scheme for mapping to tenants.

Advanced patterns for production-grade pipelines

Per-tenant dynamic addressing

Generate addresses on demand per tenant or per entity, for example task+{taskId}@in.example.com. Record the mapping in your DB so routing is an O(1) lookup. When a user replies, you already know the target entity without NLP.

Thread correlation and deduplication

Use Message-ID for idempotency and In-Reply-To/References for threading. If absent, fall back to subject heuristics and reply separators, but always prefer header-based linkage. Store a canonical thread ID and append new messages atomically to avoid split threads.

Command parsing in subject or body

Define a small grammar for commands like [close], [assign @alice], or [priority high]. Validate the sender is authorized to execute the command. Apply changes idempotently.

Event-driven scaling

Use queue-driven workers and serverless or containerized consumers. For high-volume workloads, separate lightweight parsing and security checks from heavy attachment processing and OCR pipelines. Keep each job focused and retryable with dead-letter queues.

Auditability and compliance

Persist the normalized payload and a reference to the original raw MIME in immutable storage if required.
Log all webhook deliveries, verification results, and user actions taken due to an email.
Encrypt sensitive attachments at rest and restrict URL lifetimes via short-lived presigned URLs.

Fallback to polling

If your system experiences downtime or you deploy in restricted networks, switch to REST polling temporarily. Maintain the same idempotency keys and processing pipeline so it is a drop-in replacement for webhooks.

Conclusion

Inbound email processing is a powerful integration surface for server-side teams. Treat email like any other event stream: validate authenticity, normalize input, store durably, and hand off to workers with idempotency guarantees. With robust routing, MIME-aware parsing, and secure webhooks, you can transform unstructured messages into reliable, testable APIs that power approvals, ticketing, content ingestion, and more. If you want to skip MX setup, MIME edge cases, and attachment handling, MailParse delivers inbound-email-processing as clean JSON via webhook or REST so your backend can focus on logic, not plumbing.

FAQ

Webhook or REST polling - which should I use?

Use webhooks for low-latency push delivery and simpler operations. Polling is ideal when inbound traffic is blocked, your service is offline during deploys, or you want backpressure. Many teams enable both: webhooks for normal flow, polling as a fallback. For implementation details, see Webhook Integration: A Complete Guide | MailParse.

How do I verify authenticity of inbound events?

Verify an HMAC signature or a signed token on every webhook. Reject events with invalid signatures or old timestamps. When you act on a sender's identity, check SPF, DKIM, and DMARC results if available and require the sender to be authorized for the target tenant. Keep a short signature lifetime to mitigate replay.

What is the best way to parse MIME safely?

Rely on mature libraries and normalize to a structured representation with text, HTML, and attachments. Always sanitize HTML, remove scripts, and validate content types. If you want a ready-made JSON payload, see Email Parsing API: A Complete Guide | MailParse and MIME Parsing: A Complete Guide | MailParse.

How do I handle very large attachments?

Stream uploads directly to object storage to avoid memory spikes. Impose size limits per attachment and per message. Scan for malware, and deliver presigned URLs to downstream services rather than inlining blobs in your message JSON. Set short expiration times and require signed requests for access.

How do I test inbound email flows locally?

Use a public tunnel like Cloudflare Tunnel or ngrok to expose your webhook. Record and replay real webhook calls in your test suite. Seed fixtures that include tricky MIME cases like nested multiparts, inline images, and multi-encoding headers. When using a managed service, capture example payloads and build golden tests for your parser and router.