Introduction
Email archival is not only a compliance checkbox for SaaS founders, it is a strategic capability that turns inbound communication into searchable, analyzable product data. When every support ticket, invoice, or activation email lands as structured JSON, your application can store, index, and audit communication with precision. With MailParse, founders get instant email addresses, automatic MIME parsing to JSON, and delivery via webhooks or REST polling, which shortens the path from "email received" to "email indexed and queryable".
This guide lays out a step-by-step approach for building an email-archival pipeline that is fast, multi-tenant friendly, and cost efficient. It focuses on the workflows SaaS teams already use: object storage for raw blobs, relational or document databases for metadata, and search engines optimized for queries. You will learn how to wire webhooks, define schemas, handle legal holds, and monitor the whole system with meaningful KPIs.
The SaaS Founders Perspective on Email Archival
Founders building communication-heavy products face a specific set of constraints:
- Move fast without breaking compliance: SOC 2, GDPR, and customer procurement reviews require a clear story on retention, deletion, and legal holds. You need guardrails that do not slow down product shipping.
- Multi-tenant safety: Tenants must be cryptographically and logically isolated. If your archival is shared-nothing by tenant or enforced via row-level policies, audits become less painful.
- Cost curves that scale nicely: Email-archival volume grows with customer count. Cold storage for raw MIME plus compressed indexes keep cost per message predictable.
- Search that actually works: Teams need to find messages by sender, subject, keywords, attachment types, or identifiers like invoice numbers. Indexing parsed fields and body text is essential.
- Operational simplicity: Small teams need a pipeline that is observable and repairable. Retries, dead letter queues, and reindex jobs should be first-class.
Solution Architecture for Email-Archival
The following architecture balances reliability, cost, and simplicity:
- Inbound email addresses: Provision on-demand receive-only addresses for each tenant or object. Use per-tenant subdomains or aliases for isolation.
- Delivery channel: Receive events via webhooks for near-real-time processing, or poll a REST API when your network policy requires it. Include retry logic and idempotency keys.
- Queue and worker tier: Buffer webhook events to a queue like SQS or NATS to decouple ingestion from storage and indexing. Workers perform validation, deduplication, and persistence.
- Cold storage: Store raw MIME and attachments in object storage such as S3, GCS, Azure Blob, or R2. Use bucket prefixes by tenant and message id. Enable lifecycle policies for tiering and retention.
- Metadata database: Persist parsed metadata in Postgres, MySQL, DynamoDB, or MongoDB. Index fields like from, to, subject, date, message_id, and attachment info.
- Search index: Push denormalized documents to Elasticsearch or OpenSearch. Index subject, headers, sender domains, tokenized text content, and attachment text when available.
- Retention and legal holds: Track retention policies by tenant and label. Implement a legal_hold flag that suppresses deletion jobs while preserving normal search.
- Observability: Emit metrics on ingestion latency, parse success rate, storage errors, index lag, and queue depth.
Implementation Guide
1) Provision inbound addresses and routing
Create per-tenant or per-object aliases such as {tenant}@in.yourapp.com or {object_id}+{uuid}@in.yourapp.com. The alias should encode the tenant or context you will store alongside each message. Configure MailParse to deliver events via webhook or make them available for REST polling.
2) Receive emails via webhook
Webhooks minimize latency and simplify ingestion. Make the endpoint idempotent by using message_id as a natural key.
// Node.js + Express
import express from 'express';
import crypto from 'crypto';
const app = express();
app.use(express.json({ limit: '10mb' }));
// Basic signature check example - replace with your HMAC verification
function verifySignature(req) {
const sig = req.header('X-Signature') || '';
return sig.length > 0; // implement HMAC check
}
app.post('/webhooks/inbound-email', async (req, res) => {
if (!verifySignature(req)) return res.status(401).send('invalid signature');
const evt = req.body;
// evt.parsed contains normalized JSON, evt.raw_url points to raw MIME if provided
const idempotencyKey = evt.parsed.message_id;
try {
// enqueue for processing
await queue.publish('email.ingest', { idempotencyKey, evt });
res.status(202).send('accepted');
} catch (e) {
res.status(500).send('error');
}
});
app.listen(3000);
If your network policy prefers pull over push, poll the REST endpoint on an interval with backoff and checkpoint the last seen event id.
# Python + requests
import requests
import time
TOKEN = 'your_api_token'
CURSOR = None
while True:
params = {'cursor': CURSOR, 'limit': 100}
r = requests.get('https://api.yourapp.com/inbound/messages', params=params, headers={'Authorization': f'Bearer {TOKEN}'})
r.raise_for_status()
batch = r.json()
for item in batch['items']:
process(item)
CURSOR = item['id']
if not batch['items']:
time.sleep(5)
3) Normalize, validate, and deduplicate
Normalize fields so they are consistent across senders. Use message_id plus a content hash to prevent duplicates across retries.
{
"tenant_id": "t_42",
"received_at": "2026-05-01T12:01:22Z",
"parsed": {
"message_id": "<abcd@example.com>",
"from": [{"name": "Jane Doe", "email": "jane@example.com"}],
"to": [{"name": "", "email": "support@yourapp.com"}],
"cc": [],
"subject": "Invoice 8123",
"date": "2026-05-01T12:00:59Z",
"headers": { "in-reply-to": "<prev@example.com>" },
"text": "Hello team...",
"html": "<p>Hello team...</p>",
"attachments": [
{ "filename": "invoice.pdf", "content_type": "application/pdf", "size": 88412, "sha256": "..." }
]
},
"raw_mime_url": "s3://archival/t_42/2026/05/01/abcd.eml",
"content_sha256": "23f0...9e",
"legal_hold": false
}
4) Store raw MIME and attachments
Persist the original MIME for defensible archiving. Use deterministic paths for easy retrieval and lifecycle tiering.
s3://archival/{tenant_id}/{yyyy}/{mm}/{dd}/{message_id}.eml
s3://archival/{tenant_id}/{yyyy}/{mm}/{dd}/attachments/{sha256}-{filename}
- Enable default encryption and bucket policies.
- Add lifecycle rules: hot for 30 days, standard-infrequent for 11 months, glacier-deep for long term.
- Tag objects with tenant_id and legal_hold. Retention must ignore lifecycle delete when legal_hold is true.
5) Persist metadata to a database
Postgres with JSONB works well for structured fields and flexible additions. Apply row-level security by tenant.
CREATE TABLE email_messages (
id BIGSERIAL PRIMARY KEY,
tenant_id TEXT NOT NULL,
message_id TEXT NOT NULL,
received_at TIMESTAMPTZ NOT NULL,
subject TEXT,
sender_domain TEXT GENERATED ALWAYS AS (split_part((parsed->'from'->0->>'email'), '@', 2)) STORED,
parsed JSONB NOT NULL,
raw_mime_url TEXT NOT NULL,
content_sha256 TEXT NOT NULL,
legal_hold BOOLEAN NOT NULL DEFAULT false,
UNIQUE (tenant_id, message_id)
);
CREATE INDEX idx_email_messages_tenant_ts ON email_messages (tenant_id, received_at DESC);
CREATE INDEX idx_email_messages_gin ON email_messages USING GIN (parsed jsonb_path_ops);
6) Index for search
Push a denormalized document to your search engine that optimizes text and keyword queries.
{
"id": "t_42:<abcd@example.com>",
"tenant_id": "t_42",
"from_email": "jane@example.com",
"from_domain": "example.com",
"to": ["support@yourapp.com"],
"subject": "Invoice 8123",
"body_text": "Hello team...",
"date": "2026-05-01T12:00:59Z",
"attachment_filenames": ["invoice.pdf"],
"attachment_types": ["application/pdf"],
"raw_mime_url": "s3://archival/t_42/2026/05/01/abcd.eml",
"legal_hold": false
}
- Use keyword fields for exact matches like message_id and domains.
- Apply analyzers to subject and body_text for text search.
- Store attachment text if you perform OCR or PDF extraction.
7) Retention, legal holds, and privacy
- Retention policies: Define defaults by plan tier, then allow overrides per tenant. Example: standard 24 months, enterprise configurable.
- Legal holds: Add a legal_hold flag that gates deletion jobs. Deletion tasks must check the flag atomically before removing objects and index docs.
- Right to be forgotten: If an email contains PII, support targeted redaction of body fields while preserving audit proofs. Store a hash of removed blobs for evidence without content.
8) Security controls
- Encryption: Server-side encryption for object storage. Consider per-tenant KMS keys. TLS enforced for webhooks and polling.
- Access control: Row-level security by tenant_id. Minimize who can read raw MIME. Use scoped IAM roles for workers.
- Integrity: Store content hashes and validate on read for high-assurance tenants.
9) Backfill and reindex jobs
Add two maintenance workflows:
- Backfill: Ingest historical mailboxes or legacy EMLs. Write a migrator that uploads raw blobs, generates parsed JSON, and upserts indexes.
- Reindex: When analyzers or mappings change, rebuild the index from the database and object storage. Use a versioned alias for zero-downtime swaps.
10) Operational playbooks
- Dead letter queues: Route parse failures to DLQ with reason codes. Surface in your ops dashboard.
- Replay: Store minimal inputs to re-run failed events. Provide a secure replay UI for admins.
- Alerts: Notify on webhook failure spikes, index lag, and storage errors.
Integration with Existing Tools
Founders typically start with cloud primitives they already use:
- Object storage: S3, GCS, Azure Blob, or R2 for cold storage. Configure bucket policies to enforce tenant scoping by prefix.
- Datastores: Postgres or MySQL for transactional metadata. DynamoDB or MongoDB for high write throughput with flexible schemas.
- Search: Elasticsearch or OpenSearch for cross-field queries and full text. Meilisearch works for lightweight use cases.
- Analytics: Stream parsed email-archival events to BigQuery or Snowflake for reporting.
- Support workflows: Feed parsed emails into your ticketing system, then link tickets back to archived message ids for audit.
For deeper technical details on the data contract and common field shapes, see Email Parsing API: A Complete Guide | MailParse. If you plan to scale via push delivery, follow best practices in Webhook Integration: A Complete Guide | MailParse for retries, signatures, and idempotency.
Measuring Success
Your archival pipeline should have clear, outcome-oriented metrics:
- Ingestion latency: Time from SMTP receipt to persisted metadata. Target p95 under 5 seconds for webhooks.
- Delivery success rate: Percentage of events acknowledged within retry limits. Aim for > 99.9 percent.
- Parse coverage: Percentage of messages where headers, body, and attachments are successfully parsed. Track by content type.
- Index lag: Time from metadata write to searchable index availability. Target p95 under 30 seconds.
- Storage cost per message: Total storage spend divided by volume. Monitor after lifecycle tiering kicks in.
- Search performance: p95 query latency for common filters like sender, subject, and date range. Target under 300 ms for dashboards.
- Dedupe rate: Percentage of duplicate deliveries that were correctly suppressed by message_id and hash.
Build dashboards that slice metrics by tenant and by message size. This helps detect noisy tenants, malformed bulk senders, or attachment-heavy workloads that need compression or text extraction adjustments.
Putting It All Together
When you wire inbound events to queues, persist raw MIME to cold storage, store parsed metadata in a database, and index the right fields for search, you get an email-archival system that founders can trust in audits and customers can rely on during investigations. MailParse handles the hard parts of receiving mail and parsing MIME so your team focuses on policies, storage, and search, not RFC edge cases.
FAQ
Do I need to store the raw MIME if I already store parsed JSON?
Yes for most compliance or audit scenarios. Parsed JSON enables fast querying, but raw MIME preserves the exact original content and headers. The best practice is dual storage: raw MIME in object storage with lifecycle tiering, and parsed metadata in a database and search index. Keep a stable pointer from metadata to the raw blob.
How should I handle large attachments and PII in email-archival?
Store attachments separately from the message record with a content hash as a key. Index metadata like filename, content type, and size. For PII, implement field-level redaction in the parsed JSON and index while retaining the raw object under legal hold if required. Use per-tenant keys for object encryption and IAM roles that restrict access to sensitive blobs.
What is the best way to design multi-tenant isolation?
Scope every layer by tenant_id. Use bucket prefixes and IAM conditions for object storage, row-level security or partitioning in the database, and index aliases or document-level security for search. Include tenant_id in idempotency keys and message identifiers to avoid cross-tenant collisions.
Can I migrate historical mailboxes into the new archival system?
Yes. Implement a backfill worker that consumes EMLs or mbox files, uploads them to object storage, generates parsed metadata, then upserts the database and search index. Use a two-phase idempotent write: first the raw object, then metadata with a reference to the object URL. Finally, run a verification job to sample and compare content hashes.
How do I validate webhook authenticity and prevent replay?
Verify HMAC signatures on every request and bind the signature to a timestamp header. Enforce a strict time window, such as 5 minutes, and reject stale requests. Store a nonce or delivery id and mark it used so the same payload cannot be replayed. Combine this with HTTPS and least-privilege credentials for strong protection.