MIME Parsing for Email Archival | MailParse

MIME Parsing for reliable email-archival and discovery

Email-archival succeeds or fails on the fidelity of message decoding. MIME parsing turns raw RFC 5322 messages into structured parts you can store, index, and query. When you decode MIME-encoded bodies, attachments, and headers into normalized fields, you gain reliable search, tamper-evident audit trails, and frictionless legal holds. A good parser converts multipart structures into a consistent JSON shape, resolves charsets and content-transfer-encodings, captures headers without loss, and preserves the original raw message for evidentiary integrity. With MailParse, teams can receive inbound emails instantly, convert them into structured JSON, and route them into archival storage or an index without building their own MIME layer.

Why MIME parsing is critical for email-archival

Email is a complex container. Different clients produce different MIME trees, charsets, and encodings. Archival systems must transform this variability into a dependable schema. The following capabilities are essential:

• Canonical headers for search and audit: Normalize From, To, Cc, Subject, Message-ID, In-Reply-To, References, Date to a consistent form. Decode RFC 2047 encoded-words, collapse whitespace, and store a lowercased variant for keyword search. Preserve the exact original header set for chain-of-custody.
• Reliable decoding of message bodies: Convert quoted-printable and base64 encoded text into UTF-8. For multipart/alternative, retain a ranked list of alternatives and store a canonical text version for search plus the html version for rendering.
• Attachment extraction and deduplication: Extract filename, media type, size, disposition, and a content hash. Store the binary in object storage and index the metadata. Hash-based dedup cuts storage cost and speeds review.
• Inline image resolution: Map cid: content IDs to their matching attachments. This allows accurate HTML rendering if needed and makes it easy to identify images that are content rather than attachments.
• Thread linking: Use Message-ID, In-Reply-To, and References to build conversation graphs for review and analytics.
• Authentication and provenance: Capture results from SPF, DKIM, and ARC headers. This supports deliverability diagnostics and evidentiary analysis.
• Retention and legal hold readiness: Store the original raw message alongside the normalized JSON. Apply WORM or object lock where needed. Tag items under hold to prevent expiration.

The outcome is an archive that answers practical questions quickly. Who emailed whom about a topic. Which messages contain a given attachment hash. How many times a document was sent. All of that depends on consistent MIME-parsing, decoding, and indexing.

Reference architecture for MIME parsing in an email-archival pipeline

A simple, scalable pattern connects inbound reception, MIME parsing, archival storage, and indexing. Here is a reference flow:

1. Inbound addressing: Provision unique receiving addresses per tenant, team, or workflow. Use sub-addressing or unique aliases for traceability.
2. Reception and parsing: Use a service that receives mail and emits parsed JSON with the full MIME tree. MailParse can post a webhook or offer REST polling so your system stays in control of ingestion pace.
3. Persistence tiers:
   • Raw store: Save the original RFC 5322 message (EML) in object storage. Apply object lock if required.
   • Normalized JSON store: Persist the parsed structure in a document database for fast retrieval.
   • Attachment store: Write binaries to object storage using content-hash keys to deduplicate.
   • Search index: Push searchable fields to Elasticsearch or OpenSearch for rapid queries.
4. Enrichment: Compute SHA-256 of bodies and attachments, detect language, extract text from PDFs or Office files, and classify system messages like DSNs or MDNs.
5. Security and compliance: Verify webhook signatures, scan attachments for malware, and apply data loss prevention rules where required.
6. Access control: Enforce tenant isolation, per-folder roles, and legal hold flags at query time.

For related foundation work like MX, SPF, and DKIM setups, see the Email Infrastructure Checklist for SaaS Platforms. For idea generation on what to do once messages are parsed, review Top Inbound Email Processing Ideas for SaaS Platforms.

Step-by-step implementation: from webhook to archive index

1) Configure inbound addresses

• Create a per-tenant pattern like {tenant}+archive@yourdomain.tld. Use routing rules to associate inbound addresses with internal org or case IDs.
• Publish DNS records for SPF and set up DKIM so you can capture authentication results and improve deliverability for reply-flows.

2) Set up the webhook endpoint

• Expose an HTTPS endpoint that accepts JSON payloads of parsed MIME messages.
• Require HMAC signatures and a per-tenant secret. Enforce IP allowlists and TLS 1.2 or higher.
• Return a 2xx only after persisting both the raw message and the normalized JSON so the delivery system can retry safely on failures.

When the parsing service posts to your webhook, you receive a payload with high-level fields and a full MIME tree. MailParse lets you poll instead if you prefer pull-based ingestion.

3) Define parsing and normalization rules

• Convert dates to UTC and store both the normalized timestamp and the original header value.
• Lowercase and trim addresses for case-insensitive search, but keep the original forms for fidelity.
• Decode quoted-printable and base64 text into UTF-8. Strip control characters that break indexing while keeping the original copy in raw storage.
• Extract attachments with stable content hashes. For inline parts, record the content_id to allow HTML reconstruction.
• Classify message type: human message, automated notification, bounce (DSN), read receipt (MDN), calendar invite, or encrypted container.

4) Persist data and index searchable fields

1. Write the raw message to object storage with a durable key like messages/yyyy/mm/dd/tenant/sha256.eml.
2. Persist the normalized JSON to a document database. Keep a pointer to the raw message key.
3. Store attachments to object storage under attachments/sha256. Reuse identical hashes across messages.
4. Send a subset of fields to a search index for fast queries.

5) Example of parsed JSON suitable for archival

{
  "tenant_id": "acme-corp",
  "message_id": "<CAF5k_2d0P@mail.example>",
  "received_at": "2026-04-25T14:22:03Z",
  "from": {"name": "Ava Patel", "email": "ava@example.com"},
  "to": [{"name": "Legal Archive", "email": "archive+acme@yourdomain.tld"}],
  "cc": [],
  "subject": "Q2 contract addendum, signed",
  "subject_raw": "=?UTF-8?Q?Q2_contract_addendum,_signed?=",
  "date": "2026-04-25T14:21:59Z",
  "thread": {
    "in_reply_to": "<CAF5k_1Xa@mail.example>",
    "references": [
      "<CAF5k_0Aa@mail.example>", 
      "<CAF5k_1Xa@mail.example>"
    ]
  },
  "authentication": {
    "spf": "pass",
    "dkim": "pass",
    "arc": "none"
  },
  "headers": {
    "mime-version": "1.0",
    "content-type": "multipart/mixed; boundary=boundary_abc",
    "x-mailer": "SuperMailer 4.2"
  },
  "bodies": {
    "text": "Attached is the signed PDF.\nThanks,\nAva",
    "html": "<p>Attached is the signed PDF.</p><p>Thanks,<br/>Ava</p>",
    "alternatives": [
      {"type": "text/plain", "charset": "utf-8", "size": 46},
      {"type": "text/html", "charset": "utf-8", "size": 78}
    ]
  },
  "attachments": [
    {
      "file_name": "Contract-Addendum-Q2.pdf",
      "content_type": "application/pdf",
      "disposition": "attachment",
      "size": 184233,
      "sha256": "1eea32c322c0fc2c2db2342d0ee2f0a5055a4a19a5f1c5e0c0e0b41c4df1a9a7",
      "object_key": "attachments/1e/ea/32/Contract-Addendum-Q2.pdf",
      "pages": 3
    },
    {
      "file_name": "logo.png",
      "content_type": "image/png",
      "disposition": "inline",
      "content_id": "logo-123@example",
      "size": 5423,
      "sha256": "79d9c6a4ee0c2e1f6c0905b2e7d7e35ab20d86d74b0d56a64f5ee0ba019c5772",
      "object_key": "attachments/79/d9/c6/logo.png"
    }
  ],
  "raw": {
    "object_key": "messages/2026/04/25/acme-corp/cc3e6f4d3a9b.eml",
    "size": 213944
  },
  "classification": "human",
  "hashes": {
    "body_sha256": "9e9cb3a6d33a4efc0dd3a79e0a0cc16ac04d4e2b3f246cc155d9a7160cd3bc3c"
  }
}

Index the following for fast queries: lowercased sender and recipient addresses, normalized subject, UTC date, thread identifiers, attachment filenames and hashes, and a full-text field from the text body. Preserve the raw.object_key for compliant retrievals.

For additional ideas that build on parsed email, check the Top Email Parsing API Ideas for SaaS Platforms.

Testing your email-archival pipeline

Testing email-based workflows requires realistic messages. Exercise your pipeline with a broad matrix:

• Encodings and charsets: Quoted-printable, base64, 7-bit, and 8-bit content with UTF-8, ISO-8859-1, Shift_JIS, and KOI8-R. Verify normalization to UTF-8 and accurate header decoding.
• Multipart nesting: multipart/mixed containing multipart/alternative bodies, inline images, and attachments. Validate content-id mapping in HTML.
• Large attachments: 20 MB to 50 MB PDFs and spreadsheets. Confirm streaming upload, timeouts, and object storage write times.
• Calendar and receipts: text/calendar invites, DSNs, and MDNs. Ensure classification is correct and body extraction is safe for indexing.
• Edge cases: Missing Message-ID, bad Date headers, duplicate headers, TNEF winmail.dat, overlong subject lines, and malformed boundary lines.
• Security scenarios: Executable attachments, polyglot files, zip bombs, and HTML with suspicious data URLs. Ensure scanners and quotas work as intended.

Useful generation strategies:

• Send varied fixtures with swaks:

  swaks --to archive+acme@yourdomain.tld --from ava@example.com \
    --h-Subject "=?utf-8?Q?Invoice_=E2=84=96?=" --body "See attached" \
    --attach @Contract-Addendum-Q2.pdf --server smtp.yourdomain.tld

• Embed inline images and verify cid: resolution in your archived HTML.
• Replay production-safe samples from your raw store into a staging environment to test regression cases.

During tests, assert that:

• Raw and normalized copies are both persisted and cross-referenced.
• Attachment hashes reproduce bit-for-bit across reingestion.
• Search results match expectations for subject, email addresses, and attachment names.
• Webhook retries are idempotent and do not duplicate objects or index entries.

MailParse offers structured JSON that makes assertions easy since each field is deterministic when decoding is correct.

Production checklist for email-archival at scale

• Idempotency and deduplication: Use a composite key of tenant_id plus message_id when present, else a stable hash of selected headers and the canonical body. Make storage and indexing operations idempotent.
• Retry safety: Design webhook processing to be fully retryable. Persist to a durable queue if downstream systems are unavailable.
• Monitoring and alerting: Track throughput, parse error rate, average attachment size, antivirus detections, and indexing latency. Alert on prolonged webhook delivery failures.
• Security controls: Verify HMAC signatures, pin to known CA roots, and rotate secrets. Encrypt data at rest using KMS. Run attachments through antivirus and file-type sniffing.
• Compliance features: Enable WORM or object lock for raw messages in regulated archives. Apply legal hold tags at the object and index layers.
• Retention policies: Implement policy-driven expiration for normalized JSON and attachments while preserving raw copies when required. Keep a hold exception path.
• Access governance: Enforce tenant isolation, least-privilege roles, and audit logs of view and export events.
• Cost management: Compress EML with zstd or gzip, tier cold data to infrequent access classes, and deduplicate attachments by hash.
• Index design: Use exact-match fields for message_id and addresses, analyzers for free-text body search, and a separate field for attachment filenames and hashes.
• Internationalization: Normalize all text to UTF-8. Store original charsets and ensure your index preserves language-specific tokenization.
• Ingestion controls: Enforce maximum message size and attachment count. Quarantine oversize items for manual review.
• Policy alignment: Keep an audit of parsing versions so you can re-normalize older items consistently after parser improvements.

For broader operational readiness, consult the Email Deliverability Checklist for SaaS Platforms. It complements archival work by ensuring messages you send for tests and workflows actually arrive.

Conclusion

Successful email-archival depends on accurate MIME parsing, consistent normalization, and disciplined storage design. Decode every header and part, preserve the raw message, and index the right fields for the queries you must answer. With MailParse handling the heavy lifting of MIME, webhooks, and JSON structure, your team can focus on retention policy, security, and discovery workflows rather than message internals.

FAQ

Should I store both the raw EML and the parsed JSON?

Yes. Store the raw message for evidentiary integrity and chain-of-custody, ideally with object lock. Store the normalized JSON for fast access and indexing. Link them with stable object keys so you can retrieve the exact original when needed.

How do I handle duplicate attachments across many emails?

Hash each attachment with SHA-256 and store by content hash. Maintain a reference table mapping message IDs to attachment hashes. The index can expose all messages that contain a given hash, which is useful for legal review and data minimization.

What about encrypted or signed messages like S/MIME and PGP?

Detect application/pkcs7-mime, application/pkcs7-signature, and PGP multipart structures. Store the container as an attachment if you cannot decrypt. If you control keys, decrypt server side and store both the decrypted parts and the original encrypted body. Keep signature status in metadata.

Which fields are most important to index for fast discovery?

Index sender and recipient addresses, normalized subject, UTC date, thread identifiers, attachment filenames and hashes, and a full-text body field. Add a keyword field for message_id to support exact lookups.

How big can messages be and how do I scale ingestion?

Set a maximum message size aligned with your storage and network constraints, such as 25 MB. Use streaming uploads to object storage, queue webhook deliveries, and scale consumers horizontally. Keep processing stateless so you can run many workers behind a load balancer.