---
id: 2026-04-07-open-archiver-installation-and-operations-guide
slug: open-archiver-installation-and-operations-guide
title: "Open Archiver: installation and operations guide"
excerpt: "Self-hosted, GoBD-compliant email archiving with Docker: installation, IMAP, Microsoft 365 and Google Workspace, tamper-proof operation and eDiscovery."
date: "2026-04-07T09:00:00+02:00"
updated: "2026-04-07T09:00:00+02:00"
author:
name: "László Kovács"
handle: "lkovacs"
category: "server-environments"
tags: ["openarchiver", "docker", "devops", "linuxadmin", "linuxtutorial", "systemadmin", "ubuntu"]
reading_time: 79
toc: true
---
[Open Archiver](https://openarchiver.org/){.badge-link-text} is a fully open-source, self-hosted platform for tamper-proof email archiving and eDiscovery. It archives, indexes and searches emails and attachments from any source — [Google Workspace](https://workspace.google.com/){.badge-link-text}, [Microsoft 365](https://www.microsoft.com/microsoft-365){.badge-link-text}, classic IMAP servers or PST files — and stores them permanently, tamper-evident and searchable.
Unlike cloud archiving services, you keep full control of your data. No vendor lock-in, no monthly fees, no dependence on external providers. Everything runs in your infrastructure, behind your firewall, under your backup and encryption rules. That is exactly what you need as an experienced admin or DevOps engineer for production, GoBD-compliant environments.
The solution is based on [Docker](/en/tag/docker){.badge-link-text} and Docker Compose. It combines a robust mail-ingestion layer, a capable full-text search engine (including attachment indexing), a tamper-proof storage layer and a modern web UI for search and administration. You can set up automatic retention policies, legal holds and audit trails — everything you need for statutory requirements and internal compliance rules.
```markdown
┌─────────────────────────────────────────────────────────────┐
│ Architecture overview: Open Archiver platform │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Email sources: Google Workspace • M365 • IMAP • PST │ │
│ └───────────────────────────────┬─────────────────────┘ │
│ │ TLS / API ingestion │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Open Archiver Core: pipeline and ingestion engine │ │
│ │ • Mail parser and SHA-256 deduplication │ │
│ │ • OCR and full-text indexing of attachments │ │
│ │ • Tamper-proof WORM storage and audit trail │ │
│ └──────────────────────────┬──────────────────────────┘ │
│ │ Persistence and routing │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Host infrastructure: Docker • Nginx TLS • NVMe pools│ │
│ └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
```
That gives you your own scalable archive, as performant and secure as expensive enterprise products — without the dependencies.
⚠️ **Note:** This is not a beginner tutorial. It is aimed at you if you already have Linux and Docker experience and want a practical start into installing Open Archiver. The point is not to push you through a strict course, but to give you useful knowledge for daily work. If you are new to Docker, read basic Docker documentation first before you continue here.
## Architecture and technical foundations
**Components and Docker services at a glance**
Open Archiver uses a modular Docker Compose architecture. Each service runs isolated, talks over an internal bridge network and only accesses explicitly defined volumes. That split gives reproducible deployments, simple scaling and targeted maintenance — essential when you run tamper-proof archiving in production.
### The four core services form the foundation
The `open-archiver` container (image `logiclabshq/open-archiver:latest`) combines ingestion, REST API and the React web UI. It is the central entry point for all mail sources and user actions. This is where schedulers for automatic fetches and forwarding to indexing run.
`postgres` (official `postgres:16` image) stores metadata, users, retention policies, legal-hold entries and the full audit trail. Every archiving operation is logged here — exactly what you need for GoBD evidence.
`valkey` (Valkey image as a Redis fork) is the queue and cache layer. It buffers asynchronous jobs such as bulk imports or re-indexing and keeps the main service from blocking during load spikes.
`meilisearch` (`getmeili/meilisearch:latest`) is the dedicated full-text engine. It indexes not only subject, body and headers, but also common attachments, and delivers sub-second faceted search.
You also create persistent volumes: `archiver-data` for the raw emails, `postgres-data` and `meilisearch-data`. These volumes survive container restarts and can later be backed up separately.
```yaml
services:
open-archiver:
image: logiclabshq/open-archiver:latest
depends_on:
postgres:
condition: service_healthy
valkey:
condition: service_healthy
meilisearch:
condition: service_healthy
volumes:
- archiver-data:/data
postgres:
image: postgres:16
volumes:
- postgres-data:/var/lib/postgresql/data
valkey:
image: valkey/valkey:latest
meilisearch:
image: getmeili/meilisearch:latest
```
```bash
docker compose ps
```
The command immediately shows status, ports and dependencies of all containers.
```markdown
┌─────────────────────────────────────────────────────────────┐
│ Service architecture: the four core components │
│ │
│ ┌───────────────────────────┐ │
│ │ open-archiver service │ │
│ │ (API, ingest & engine) │ │
│ └─────────────┬─────────────┘ │
│ │ │
│ ┌───────────────────┴───────────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ PostgreSQL │ │ Valkey │ │ Meilisearch │ │
│ │ (metadata) │ │ (queue/job) │ │ (full text) │ │
│ │ Audit trail │ │ Async cache │ │ Attachment │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────────┘
```
That clear split matters because you can later update or scale individual services independently without stopping the entire archive. Make sure healthchecks exist in the Compose file — otherwise open-archiver starts too early and fails. Typical mistake: too little RAM for Meilisearch (at least 2 GB recommended), then indexing aborts.
You need this knowledge later for monitoring, targeted backup and troubleshooting.
### Data flow, indexing and storage
Once you connect an email source, a controlled data flow starts. Open Archiver does not simply fetch messages — it processes them asynchronously, stores them tamper-evident and prepares them for very fast search. That pipeline is the difference between a simple dump and a tamper-proof archive.
The process starts in the `open-archiver` container. You configure IMAP, Google Workspace, Microsoft 365 or PST imports via the API or the web UI. The scheduler checks at set intervals for new or changed mail. Incoming messages first land as a job in Valkey. BullMQ takes over asynchronous processing so the main process does not block — decisive at thousands of mails per hour.
The actual work happens in the job worker: The email is parsed as an `.eml` file. Metadata (subject, From, To, date, headers, thread ID) is extracted. Attachments are detected, unpacked and — where possible — converted to text (PDF, DOCX, images with OCR support in newer versions). Every file gets a SHA-256 hash.
That hash is stored immediately in PostgreSQL. Any later change to the stored file is then detectable — the basis for [GoBD compliance](https://de.wikipedia.org/wiki/Grunds%C3%A4tze_zur_ordnungsm%C3%A4%C3%9Figen_F%C3%BChrung_und_Aufbewahrung_von_B%C3%BCchern,_Aufzeichnungen_und_Unterlagen_in_elektronischer_Form_sowie_zum_Datenzugriff){.badge-link-text}.
The raw email and all attachments then go into the persistent volume `archiver-data`. By default Open Archiver uses the local filesystem with Docker volumes. Optionally you can configure S3-compatible storage ([MinIO](https://min.io/){.badge-link-text}, [AWS S3](https://aws.amazon.com/s3/){.badge-link-text}). Files are deduplicated (identical mails or attachments are stored only once) and compressed. Encryption at rest uses a key managed by the backend. The volume stays reachable outside the containers and can be backed up separately.
In parallel with storage, the worker sends the full content — body, headers and extracted attachment text — to Meilisearch. The search engine builds an inverted index. Later you search not only for words in the subject or body, but also inside PDFs or Office files. Meilisearch returns faceted results, thread views and filters in under a second. The index itself lives in the volume `meilisearch-data` and is snapshotted automatically every day.
[PostgreSQL](https://www.postgresql.org/){.badge-link-text} holds all metadata, hashes, retention policies and the immutable audit trail. Every step — ingestion, storage, indexing, access — is logged here. That gives complete traceability.
```markdown
┌─────────────────────────────────────────────────────────────┐
│ Data flow and processing pipeline │
│ │
│ Email sources (IMAP / M365 / Google Workspace / PST) │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ open-archiver: scheduler and ingestion API │ │
│ └──────────────────────────┬──────────────────────────┘ │
│ │ Hand-off to queue │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Valkey: BullMQ job queue (async buffering) │ │
│ └──────────────────────────┬──────────────────────────┘ │
│ │ Worker processing │
│ ▼ │
│ ┌──────────────────────────┐ ┌────────────────────────┐ │
│ │ Async worker │ │ Tamper-proof WORM │ │
│ │ • .eml MIME parsing │─┼▶ Storage of .eml │ │
│ │ • SHA-256 checksum │ │ • Deduplication │ │
│ │ • OCR attachment extract │ │ • Encryption at rest │ │
│ └────────────┬─────────────┘ └───────────┬────────────┘ │
│ │ │ │
│ ├───────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────┐ ┌────────────────────────┐ │
│ │ Meilisearch full text │ │ PostgreSQL metadata │ │
│ │ • Header and body index │ │ • Email metadata │ │
│ │ • Attachments and OCR │ │ • Hashes and audit │ │
│ └──────────────────────────┘ └────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
```
```bash
docker compose logs -f open-archiver --tail=50
```
With this command you watch live how jobs run and whether hashes or indexing fail.
Why does this matter? Because later you must know exactly where each file lives, how you back up without losing integrity, and why a missing hash raises an alarm immediately. Watch IOPS on the volume `archiver-data` — large mailboxes write heavily here. Too little RAM on [Meilisearch](https://meilisearch.com/){.badge-link-text} causes index errors on attachment-heavy mail.
❗ **Note:** Placing the volume `archiver-data` on an overlay filesystem without dedicated block storage. Then performance explodes with thousands of small `.eml` files. Fix: a separate ZFS dataset or LVM volume with high I/O priority.
You need this later for targeted troubleshooting, scaling to several nodes and proving to auditors that no mail was ever changed.
**Architecture and technical foundations**
### Compliance features
**Tamper-proof archiving, GoBD, audit trail**
Open Archiver is designed from the ground up for tamper-proof archiving. Every email and every attachment is stored so that later changes are technically impossible or immediately detectable — exactly what you need in regulated environments. The platform meets the core requirements of [GoBD](https://de.wikipedia.org/wiki/Grunds%C3%A4tze_zur_ordnungsm%C3%A4%C3%9Figen_F%C3%BChrung_und_Aufbewahrung_von_B%C3%BCchern,_Aufzeichnungen_und_Unterlagen_in_elektronischer_Form_sowie_zum_Datenzugriff){.badge-link-text} and delivers a complete, tamper-evident audit trail.
Tamper-proof storage rests on three pillars: `cryptographic integrity`, `WORM principle (Write Once, Read Many)` and `strict access control`. Every incoming `.eml` file and every extracted attachment gets a SHA-256 hash right after processing. That hash is stored in PostgreSQL together with the exact timestamp and source ID. The file itself is stored in the volume `archiver-data` — by default with app-level WORM: the backend does not allow overwrites.
Deletions are only possible via defined retention policies or explicit legal-hold lifts. Once a mail is archived, it exists as an immutable copy. Optionally you can put the volume on ZFS with `copies=3` and `immutable` snapshots, or offload to S3 with Object Lock (compliance mode). Every access attempt is logged, even if it is read-only.
GoBD compliance is actively supported.
The principles require completeness, immutability, traceability, availability and auditability. Open Archiver provides this through:
* automatic full archiving of all incoming and outgoing mail including headers and attachments,
* immutable timestamps with NTP synchronisation,
* deduplicated but referenced storage (no duplicate copies of identical content),
* and the ability to freeze entire mailboxes or threads with a legal hold. A legal hold overrides every retention policy and prevents deletion until it is lifted by hand — ideal for ongoing litigation or tax audits.
The audit trail sits in a dedicated, read-only PostgreSQL table `audit_log`. Every entry contains: timestamp (UTC), actor (user ID or system), action type (ingest, index, view, delete-attempt, policy-change), object hash, source IP, before/after state and a JSON payload with all metadata. The table is append-only; the backend refuses UPDATE or DELETE on existing rows.
You can filter the trail in the web UI or query it directly via SQL — a must for auditors.
```markdown
┌─────────────────────────────────────────────────────────────┐
│ Compliance architecture: WORM storage and audit trail │
│ │
│ Email intake via API / connector │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Ingestion and SHA-256 hash generation + timestamp │ │
│ └────────────┬────────────────────────────┬───────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────────────────────┐ ┌────────────────────────┐ │
│ │ WORM volume │ │ PostgreSQL │ │
│ │ archiver-data storage │ │ Immutable audit trail │ │
│ │ • Read-only archive │ │ with timestamps and │ │
│ │ • Tamper protection │ │ cryptographic hashes │ │
│ └────────────┬─────────────┘ └────────────┬───────────┘ │
│ │ │ │
│ └─────────────┬──────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Retention policy and legal hold engine │ │
│ │ • Lock: no deletion while a legal hold is active │ │
│ │ • Delete: only after expiry and a release check │ │
│ └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
```
```bash
docker compose exec postgres psql -U archiver -d archiver -c "
SELECT timestamp, actor, action, object_hash, details
FROM audit_log
WHERE action = 'ingest'
ORDER BY timestamp DESC LIMIT 20;
"
```
This command prints the last 20 ingestion events with full context. You see immediately whether a mail was archived correctly or whether a hash error occurred.
⚠️ **Note:** Never disable hash verification in `.env` (`ENABLE_HASH_VERIFICATION=true` is the default). Without it you lose GoBD compliance.
💡 **Tip:** Put the `postgres-data` volume on a separate, encrypted LVM volume with regular point-in-time snapshots. Then you can restore the audit trail even if the main system is compromised.
❗ **Note:** Configuring retention policies too loosely. A mail deleted after 7 years when GoBD requires 10 years becomes a finding at audit. Fix: set the global default policy to “unlimited” or “10y” and override per mailbox. Always check with `docker compose exec open-archiver ./bin/archiver policy list`.
These features are indispensable later when you must prove to the works council, the data-protection officer or the tax office that no mail was ever changed or deleted. You can export and sign the complete trail — ready for the next audit.
## Infrastructure preparation
### System requirements
**Hardware and software recommendations**
Once the compliance features have their technical base, the infrastructure has to be prepared so Open Archiver runs stably and performantly. The requirements are not trivial, because the continuous data flow of ingestion, hash calculation, storage and indexing binds considerable resources. For a production environment you must match hardware and software to the expected mail load.
CPU requirements start at at least four physical cores for basic operation. In practice you should go to eight or more cores so asynchronous workers and Meilisearch run without bottlenecks. Every extra core speeds mass imports considerably. Modern processors with high `IPC` such as current `AMD Ryzen` or 14th-generation `Intel Core` are a good fit here.
⚠️ **Note:** Hyper-Threading can stay enabled, but do not assign Docker containers an excessive number of vCPUs, otherwise I/O performance suffers.
For RAM, a minimum of `16 GB` is enough for small installations with under 100 users. Once you have larger mailboxes or regular re-indexing, `32 GB` becomes the floor. Meilisearch keeps large parts of the index in RAM and PostgreSQL uses caches for the audit trail. With `64 GB` or more you have buffer for peaks and future growth. RAM use scales linearly with the number of indexed documents, so oversizing here is the safest choice.
💡 **Tip:** Memory can be scaled later by adding RAM modules, as long as the motherboard supports it.
For mass storage, capacity is not the only factor. At least 500 GB NVMe SSD for the Docker area is needed to hold the volumes for `archiver-data`, `postgres-data` and `meilisearch-data`. Real size depends on your mail history. Plan for 150 to 300 KB per archived message including attachments after deduplication. For a mid-sized company with 500 users and 10 years of data you can quickly land at 5 to 10 TB. Choose SSDs with high endurance and at least 20,000 IOPS for random writes, because the filesystem constantly creates and updates small .eml files. A ZFS pool with RAID-Z2 adds redundancy and snapshot capability that is useful for backups.
The operating system should be [Ubuntu 24.04 LTS](/en/tag/ubuntu){.badge-link-text}, because it offers a current kernel and long support cycles. Kernel 6.8 and newer brings improvements in container isolation and `cgroups v2` that are essential for Docker. Install Docker Engine 27.0 or newer and Docker Compose in the V2 variant. Older versions can cause problems with service healthchecks.
Additionally, tune kernel parameters such as setting `vm.swappiness` to 10 to avoid unnecessary swapping. Raise `ulimit` values for `open files` to `65535` so the many files in the volume can be handled.
These adjustments keep the system from collapsing under high load.
```markdown
┌─────────────────────────────────────────────────────────────┐
│ Recommended server and hardware specification │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Component │ Recommended minimum │ │
│ ├────────────────────┼────────────────────────────────┤ │
│ │ CPU cores │ 8+ cores (high single-core clk)│ │
│ │ Memory │ 32–64 GB RAM (ECC recommended) │ │
│ │ Storage pool │ NVMe SSD (archiver-data WORM) │ │
│ │ Filesystem │ ZFS or LVM with snapshot path │ │
│ │ Operating system │ Ubuntu 24.04 LTS / Debian 12 │ │
│ │ Container runtime │ Docker Engine 27+ with Compose │ │
│ │ Network │ 10 Gbit/s redundant uplink │ │
│ └────────────────────┴────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
```
🔧 **Practical example:**
To check your current infrastructure for compatibility, you can run the following commands. First CPU and RAM:
```bash
lscpu | grep -E 'Model name|CPU\(s\)|Thread'
free -h
```
**Then storage:**
```bash
df -h
lsblk -f
```
**And for Docker:**
```bash
docker --version
docker compose version
```
These outputs give you a quick status and show whether adjustments are needed. You can compare the results with the recommended values and upgrade hardware if needed before the actual installation.
❗ **Note:** Placing Docker volumes on a slow HDD array. Indexing then becomes the bottleneck and the whole service suffers high latency.
For smaller environments with up to 50 users, a single server with the minimum requirements is enough. For medium setups with 200 users you should go to `12 cores`, `64 GB RAM` and `8 TB storage` to have buffer for growth. In large environments with over 1000 users a cluster architecture with separate nodes for storage and compute becomes useful, even though Open Archiver currently has no native multi-node scaling. You can still run several instances behind a load balancer by sharing the volumes over `NFS` or `Ceph`.
Software dependencies include `git` for cloning the repository, `make` or similar tools for build processes and an `NTP daemon` for accurate timestamps in the audit trail. Without `NTP`, timestamps can drift and GoBD compliance can be questioned.
On the network you need at least `1 Gbit/s`, better `10 Gbit/s` for the initial PST import or IMAP sync of large mailboxes. Internal communication between Docker services uses bridge networks, so no extra configuration is needed as long as the host firewall does not block the ports.
In practice a dedicated machine or VM for Open Archiver offers the best stability. Avoid sharing the server with other resource-heavy services such as databases or web servers unless you have enough spare capacity.
### Calculating storage demand
To calculate storage demand, take the number of users, multiply by the average number of mails per day (say 50), then by the number of days in 10 years (3650) and multiply by 0.25 MB per mail. For 500 users that is about `4.56 TB`. Add 20 percent for overhead and growth. That is the capacity you need so you can plan SSD size precisely.
You should also tune the Docker daemon configuration to set resource limits. In `/etc/docker/daemon.json` you can set default `ulimits` and `log-driver` to avoid memory leaks. For the storage driver I recommend `overlay2` on a dedicated filesystem.
The hardware should also have [ECC RAM](https://en.wikipedia.org/wiki/ECC_memory){.badge-link-text} to avoid bit errors in large indexes. For datacentre operation, watch redundant PSUs and a [UPS](https://en.wikipedia.org/wiki/Uninterruptible_power_supply){.badge-link-text}, because outages can interrupt the audit trail.
To validate the requirements, run a load test with synthetic emails before you go live. That helps you find bottlenecks early and adjust the configuration. On the software side you also need current packages for `git`, `curl` and `gnupg` so the repository can be cloned without problems.
Once the system requirements are met, you can tackle the network configuration.
### Reverse proxy: TLS termination and network configuration
Network configuration starts with a reverse proxy that terminates TLS and controls access to the open-archiver service. In production the container must never be reachable directly from the internet, because the web UI contains sensitive search and administration functions and the audit trail must stay tamper-evident at all times. A dedicated reverse proxy such as [Nginx](https://www.nginx.com/){.badge-link-text} isolates the Docker stack, terminates TLS centrally and allows fine-grained access rules via headers or paths. Most admins use Nginx here because it is stable, light on resources and integrates well with Docker Compose.
Alternatively [Traefik](https://traefik.io/){.badge-link-text} or [Caddy](https://caddyserver.com/){.badge-link-text} work, but Nginx gives the best control over rate limiting and logging for compliance-relevant environments.
The proxy listens on the external ports `80` and `443` and forwards traffic only to the internal port of the open-archiver service, usually bound to port `3000`. All other services stay hidden in the internal Docker network. For TLS termination you import a valid certificate from your internal CA or generate one automatically via Let’s Encrypt. The proxy must be able to resolve the server name under which Open Archiver will later be reachable, for example `archive.yourcompany.internal`.
The configuration strictly separates `HTTP` and `HTTPS` listeners, redirects `HTTP` automatically to `HTTPS` and enables `HSTS` so browsers only build encrypted connections in future.
```markdown
┌─────────────────────────────────────────────────────────────┐
│ Network architecture and TLS termination │
│ │
│ External traffic (HTTPS / port 443) │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Reverse proxy (Nginx / Traefik / Caddy) │ │
│ │ • TLS termination (certificates via Let's Encrypt) │ │
│ │ • Rate limiting and security headers │ │
│ │ • Access restriction and IP filter │ │
│ └──────────────────────────┬──────────────────────────┘ │
│ │ Internal Docker network │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ open-archiver service (bound to port 3000) │ │
│ └─────────────────────────────────────────────────────┘ │
│ Note: PostgreSQL, Valkey and Meilisearch stay fully │
│ isolated and are not reachable from outside. │
└─────────────────────────────────────────────────────────────┘
```
🔧 **Practical example:**
First create a separate `nginx.conf` outside the Compose stack so you can version it and maintain it independently of the Archiver stack.
**The block looks like this:**
```bash
server {
listen 80;
server_name archive.yourcompany.internal;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl http2;
server_name archive.yourcompany.internal;
ssl_certificate /etc/ssl/certs/archive.fullchain.pem;
ssl_certificate_key /etc/ssl/private/archive.privkey.pem;
ssl_protocols TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
location / {
proxy_pass http://open-archiver:3000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 300s;
}
}
```
Then bind the configuration directory and the certificates as a volume into the Nginx container. Start the proxy with `docker compose up -d nginx` and immediately check with `curl -I https://archive.yourcompany.internal` whether the redirect and TLS parameters are correct.
💡 **Tip:** You can later add Basic Auth or OAuth in front of the actual login if you need an extra layer for highly regulated areas.
The host firewall must only open ports `80` and `443`. All internal Docker ports stay protected by bridge isolation. Disable the default EXPOSE statements in the open-archiver service so nothing leaks accidentally. For the network configuration inside `Docker Compose` you define a dedicated network named `archiver-net` that all services except the proxy belong to. The proxy itself runs either in the same Compose file or in a separate stack and connects via `external: true`.
```bash
docker network inspect archiver-net
```
This command shows whether all containers hang in the correct network and which IPs they have.
❗ **Note:** A classic error is the proxy not setting the X-Forwarded-For header correctly. Then the open-archiver service only sees the proxy IP and can no longer log client IPs in the audit trail.
🔧 **Practical example:**
To test the complete network chain, you temporarily add a healthcheck endpoint and call it from the host:
```bash
curl -k -I https://archive.yourcompany.internal/api/health
```
**At the same time you watch the logs:**
```bash
docker compose logs -f nginx --tail=30
```
You see immediately whether TLS handshakes succeed or whether headers are lost. Adjust `ssl_session_cache` and `ssl_session_timeout` if you expect many concurrent connections. For large environments also enable `proxy_buffering off` so large search results pass through without delay.
The entire configuration should live in `Git` so you can roll it back quickly on updates. Do not forget rotating Nginx logs and protecting them with [Fail2Ban](https://www.fail2ban.org/){.badge-link-text} if you expect [brute-force](https://en.wikipedia.org/wiki/Brute-force_attack){.badge-link-text} attempts on the login page. Combined with a dedicated VLAN or firewall rule you can restrict access to certain IP ranges without the Docker stack itself noticing.
Once the proxy runs stably, check TLS versions with `openssl s_client -connect archive.yourcompany.internal:443` and make sure only `TLS 1.3` is active. Internal communication between proxy and open-archiver runs unencrypted over the bridge network, which is acceptable because both sit on the same host. For distributed setups you would add mTLS here, but for most production installs the current setup is enough.
After network configuration is done, persistent volumes come next.
### Persistent storage: volumes and backup strategy
The three central Docker volumes of Open Archiver store everything that must survive a container restart: the raw emails, the database content and the search index. Without a correct volume configuration you lose all archived data and the audit trail on every `docker compose down`. So you create the volumes explicitly as named volumes or bind mounts and attach them to a performant filesystem.
The volume `archiver-data` takes all `.eml` files, deduplicated attachments and temporary processing files. It grows fastest and should sit on a separate NVMe pool or ZFS dataset that delivers high IOPS. PostgreSQL uses `postgres-data` for metadata, policies and the immutable audit log. Meilisearch writes its index into `meilisearch-data`. Each volume gets its own entry under `volumes` in `docker-compose.yml` so Docker manages them outside the container.
```yaml
volumes:
archiver-data:
driver: local
driver_opts:
type: none
device: /mnt/archiver-data
o: bind
postgres-data:
driver: local
meilisearch-data:
driver: local
```
The bind mount for `archiver-data` lets you format the directory directly on the host with ZFS or Btrfs and create snapshots. You can also offload the volumes to an external NFS share or Ceph cluster if you later scale to several hosts. Consistency matters: all volumes must sit on the same filesystem type so backups stay atomic.
```markdown
┌─────────────────────────────────────────────────────────────┐
│ Persistent Docker volumes and host filesystem │
│ │
│ Host filesystem and mount structure: │
│ ├── /var/lib/docker/volumes/archiver-data/_data │
│ │ └── Storage of all .eml files and attachments │
│ ├── /mnt/postgres-data │
│ │ └── Metadata, user accounts and audit-trail logs │
│ ├── /mnt/meilisearch-data │
│ │ └── Full-text index and facet database │
│ └── ZFS storage pool │
│ └── Automatic read-only snapshots and backup │
└─────────────────────────────────────────────────────────────┘
```
🔧 **Practical example:**
After the first `docker compose up` you create the volumes by hand and set permissions. Run the following commands to prepare the directories and set ownership correctly:
```bash
sudo mkdir -p /mnt/archiver-data /mnt/postgres-data /mnt/meilisearch-data
sudo chown -R 1000:1000 /mnt/archiver-data /mnt/postgres-data
sudo chown -R 1000:1000 /mnt/meilisearch-data
sudo chmod 700 /mnt/archiver-data
```
Then restart the stack and check with `docker volume ls` and `docker volume inspect archiver-data` whether the mounts are mapped correctly. You immediately see the real path on the host and can check whether data has already been written.
The backup strategy builds directly on these volumes. Because Open Archiver works tamper-proof, you must not run simple rsync jobs over live containers. Instead you use Docker-internal stop-and-backup or filesystem snapshots. For ZFS you create regular snapshots of the entire pool:
```bash
zfs snapshot tank/archiver-data@daily-$(date +%Y%m%d)
zfs send -R tank/archiver-data@daily-$(date +%Y%m%d) | zstd -T0 > /backup/archiver-daily.zst
```
That backs up not only the data but also hash consistency. For `PostgreSQL` you insert a `pg_dump` before the snapshot so the audit trail is always saved in a consistent state. `Meilisearch` offers its own dump endpoint that you can call via the API before you snapshot the volume. Combine that with a daily cron job that copies backups to external offsite storage (S3 or a second datacentre). Backup retention follows GoBD: at least ten years, better unlimited with monthly full backups and daily incrementals.
💡 **Tip:** Create a separate volume for backups that Docker never mounts, so a compromised container does not immediately endanger all backups.
You test the restore procedure best in a staging environment. Stop the stack, delete the volumes, restore the snapshots and start again. Open Archiver automatically detects whether the index must be rebuilt and starts re-indexing in the background. That can take several hours on large archives, so you plan a maintenance window.
❗ **Note:** It happens when you simply delete the volumes with `docker compose down -v` because you think the data lives elsewhere. Then the audit trail is gone for good.
🔧 **Practical example:**
To create a complete backup script, create the following file `backup-archiver.sh` and make it executable:
```bash
#!/bin/bash
DATE=$(date +%Y%m%d_%H%M)
docker compose -f /opt/open-archiver/docker-compose.yml exec postgres pg_dump -U archiver archiver > /backup/db-$DATE.sql
docker compose stop
zfs snapshot tank/archiver-data@backup-$DATE
zfs snapshot tank/postgres-data@backup-$DATE
zfs send -R tank/archiver-data@backup-$DATE | zstd > /offsite/archiver-data-$DATE.zst
docker compose start
echo "Backup $DATE completed"
```
Run it with `sudo ./backup-archiver.sh` and check with `ls -lh /backup` whether the files were created. The script only stops the stack briefly and minimises downtime.
For monitoring volume use, put `df -h /mnt/archiver-data` into your monitoring solution and alarm at 80 percent full. You can also watch Docker stats with `docker stats` to see which container writes the most. On very large installs you move the volumes onto a dedicated LVM thin-provisioned logical volume so you can online-resize without downtime.
The backup strategy should always follow the 3-2-1 rule: 3 copies, 2 different media, 1 offsite. Test restore monthly, because only a tested backup is a real backup. Open Archiver itself has no built-in backup function, so the responsibility sits entirely with you and your infrastructure. With the volumes configured and backed up correctly the whole stack runs stably and you are ready for growth and possible disasters.
## Installation and deployment
**Clone the repository, Docker Compose and .env configuration**
With the volumes prepared you now change into the directory where you will run the entire stack permanently. The first step is cloning the official repository, because that gives you not only the current Docker Compose file and the example `.env`, but also the option to use specific tags or branches later. You create a dedicated directory that later belongs only to the Archiver stack, and change into it.
The `git clone` command downloads the complete source including example configuration. In practice you always use the `main` branch for production, because the most stable releases live there.
```bash
mkdir -p /opt/open-archiver
cd /opt/open-archiver
git clone https://github.com/logiclabshq/open-archiver.git .
git checkout tags/v1.2.3
```
The repository contains `docker-compose.yml`, an example `.env` and extra scripts for migrations. You copy the example files and adapt them to your environment. `docker-compose.yml` defines all four services, their dependencies, healthchecks and resource limits. You extend it with your volumes, the internal network and resource limits so `Meilisearch` does not eat all RAM and `PostgreSQL` does not crash at peak load. Every service gets explicit `restart: unless-stopped` policies so the stack comes up automatically after a host reboot.
```markdown
┌─────────────────────────────────────────────────────────────┐
│ Directory layout: /opt/open-archiver/ │
│ │
│ /opt/open-archiver/ │
│ ├── docker-compose.yml ← Central stack definition │
│ ├── .env ← Production configuration │
│ ├── .env.example ← Documented template │
│ ├── volumes/ ← Bind-mount directories │
│ │ ├── archiver-data/ ← Emails and WORM archive │
│ │ ├── postgres-data/ ← DB data and audit logs │
│ │ └── meilisearch-data/ ← Full-text search index │
│ └── nginx/ ← Optional TLS proxy docs │
└─────────────────────────────────────────────────────────────┘
```
The `.env` file is the heart of the configuration. It holds all sensitive values that never belong in the repository. You copy `.env.example` to `.env` and fill in the variables. Important entries are `POSTGRES_PASSWORD`, `MEILI_MASTER_KEY` and `OPEN_ARCHIVER_SECRET`. The database user should get a strong random password that you generate with `pwgen` or `openssl rand`. `Meilisearch` needs its own master key for API authentication. `OPEN_ARCHIVER_SECRET` is the application secret for session and JWT tokens. You also set `MAIL_INGESTION_INTERVAL`, `MAX_WORKER_THREADS` and `STORAGE_DRIVER`. For S3 storage there are further variables such as `S3_ENDPOINT` and `S3_BUCKET`.
```bash
cp .env.example .env
nano .env
```
You adapt every line exactly to the volumes and networks you defined earlier. The file must be readable only for `root` and the `docker` user so no other process can read the passwords. After the change you validate syntax with a simple `grep` to rule out typos in variable names. `docker-compose.yml` references this `.env` file automatically via `environment` and `env_file`. Under every service you add `resource limits` so a faulty import cannot take down the whole host.
🔧 **Practical example:**
To clone the repository cleanly and prepare the configuration files immediately, you run the following sequence. First the directory and clone, then the copy and permissions:
```bash
sudo mkdir -p /opt/open-archiver
cd /opt/open-archiver
sudo git clone --depth 1 --branch main https://github.com/logiclabshq/open-archiver.git .
sudo cp .env.example .env
sudo chown -R root:docker /opt/open-archiver
sudo chmod 600 .env
sudo chmod 644 docker-compose.yml
```
Then you open `.env` and set at least ten critical values. You save and check with `cat .env | grep -E 'PASSWORD|KEY|SECRET'` whether everything is set correctly. That flow ensures you do not get permission problems on first start.
`docker-compose.yml` also contains a dedicated network `archiver-net` and the volume definitions. You extend the `open-archiver` service with `depends_on` and `condition: service_healthy` so the container only starts when `Postgres` and `Meilisearch` are ready. `Healthchecks` are already provided; you can still adapt them to your hardware by setting the interval to `10s` and raising `retries` to `5`. `Resource limits` such as `cpus: "2.0"` and `memory: 4g` for `Meilisearch` keep the search engine from overloading the host on large indexing runs.
⚠️ **Note:** Remove the `.env` file from Git history if you later commit the repository. A single `git add .env` would expose all secrets.
You test the configuration before you start the stack by running `docker compose config`. The command renders the final Compose file with all .env variables and immediately shows syntax errors or missing references. If you run several environments, you create `.env.prod` and `.env.stage` and load them with `--env-file`. That lets you separate development and production instances on the same host without volume-name conflicts.
💡 **Tip:** You can extend `docker-compose.yml` with `profiles` so `docker compose --profile monitoring up` starts extra tools such as a Prometheus exporter without changing the core configuration.
❗ **Note:** A common error is writing .env variables not exactly as they are referenced in the Compose file. Then Postgres starts with the default password and the whole stack fails on the first migration.
🔧 **Practical example:**
After you have adapted `.env`, you validate the complete configuration with a dry run. The following command shows the resolved Compose file and lets you check whether all secrets were injected correctly:
```bash
docker compose config --quiet
docker compose config | grep -A 20 'environment:'
```
You see the expanded environment variables and can be sure no password is missing. If something is missing, you correct `.env` and repeat the check. This step saves you a lot of debugging on first start.
The entire configuration is now versioned and reproducible. You can rsync the directory to a backup host or push it to a private Git repo as long as `.env` stays excluded. With `git tag` you create a checkpoint after every change so you can return exactly to the running version if there are problems. The combination of cloned repository, adapted Compose file and a secure `.env` is the foundation for a stable first start.
Once the `.env` file is fully configured, you can continue with the first start.
### Start, initialisation and admin account
Once the `.env` file is fully configured, you start the entire stack with a single command. The Docker Compose manager reads the configuration, pulls current images if needed and starts the services in the right order thanks to the `depends_on` conditions. You run the start in the background so you can watch the logs immediately. The command enables all healthchecks automatically and does not wait for manual input.
```bash
docker compose up -d
```
The first start takes longer than later restarts because `PostgreSQL` initialises the database, `Meilisearch` creates its first index and the `open-archiver` container runs the migration scripts. In the output you immediately see which containers came up and whether one is in a restart loop. The stack now needs a few minutes until all services are healthy.
```markdown
┌─────────────────────────────────────────────────────────────┐
│ Initialisation and start sequence │
│ │
│ Start command: docker compose up -d │
│ │ │
│ ▼ │
│ ├── 1. postgres ← Start, schema init and migration │
│ ├── 2. valkey ← Start and queue provisioning │
│ ├── 3. meilisearch ← Master key and index bootstrap │
│ └── 4. open-archiver← API start, scheduler and setup │
│ │ │
│ ▼ │
│ Status check: docker compose ps (all healthy) │
└─────────────────────────────────────────────────────────────┘
```
🔧 **Practical example:**
To follow the start live and not miss an error, you open a second terminal and fetch logs from all services. The command shows the last 100 lines and follows the output in real time:
```bash
docker compose logs -f --tail=100
```
You see lines such as `Postgres ready`, `Valkey listening on 6379` and `Meilisearch initialized with master key`. As soon as `open-archiver: Application started successfully` appears, initialisation is complete. You can abort the command with `Ctrl+C` at any time and later narrow it to a single service.
Initialisation covers several internal steps. First the `open-archiver` container checks the database connection and runs Flyway or Liquibase migrations. These scripts create the tables for metadata, audit log, retention policies and legal holds. At the same time `Meilisearch` initialises the main index with the required fields for subject, body, attachment text and hashes. `Valkey` creates the `BullMQ` queues for ingestion jobs and re-indexing.
If you start for the first time, the backend also creates the default configuration values for scheduler intervals and storage settings. The whole process is idempotent, so a restart does not create duplicate tables.
You check the status of all containers with a simple `ps` command. The output shows the current state, uptime and whether restarts happened.
```bash
docker compose ps
```
Once all services show `Up` and `healthy`, the stack is ready.
The open-archiver container now starts its internal setup routine. That routine creates the required directories in the archiver-data volume and checks connectivity to all dependencies. If something is missing, it aborts with a clear error that you find directly in the logs.
Admin-account setup uses a dedicated CLI command that runs inside the container. You use exec to enter the open-archiver container and call the admin-creation tool. The command requires email, full name and a strong password. The password is hashed immediately and stored in `PostgreSQL`. The account gets the super-admin role with full rights on all policies and the audit trail.
```bash
docker compose exec open-archiver ./bin/archiver admin create --email admin@yourcompany.internal --name "System Administrator" --password "YourVeryStrongPassword2026!"
```
The command prints a confirmation and shows the generated API token for first tests. You can also create the account via the web UI once the proxy is running, but the CLI path is safer for first start because no unprotected HTTP connection is needed. The new account is immediately recorded in the audit log so you can later reconstruct when and from which host the first admin was created.
**After account creation you test access.**
You call the health endpoint or the login page to check that everything is reachable. The first login via the web UI asks you to change the password and set up 2FA if you enabled that in .env. The session is signed with the application secret and stored in the browser.
💡 **Tip:** You can also run the admin command with `--force` if you later need to reset the account without re-initialising the entire stack.
During the whole initialisation you watch resource use with `docker stats` to make sure Meilisearch does not use more than the assigned 4 GB RAM. The first index bootstrap writes many small entries, so you briefly see higher I/O on the `meilisearch-data` volume.
```bash
docker stats
```
The command updates every few seconds and gives you `CPU`, `Memory` and network values for each container. You can run it in parallel with log follow.
⚠️ **Note:** Never start the stack without volume permissions first, because `PostgreSQL` otherwise aborts with permission-denied errors and initialisation does not complete.
After a successful admin setup you check the audit trail whether the first entry was written correctly. You connect with psql and run a simple query.
```bash
docker compose exec postgres psql -U archiver -d archiver -c "SELECT * FROM audit_log WHERE action = 'admin_created' ORDER BY timestamp DESC LIMIT 1;"
```
The output confirms timestamp, actor and all details. That is the first proof that the revision trail works. You can now treat the stack as stable and continue with the next phase.
❗ **Note:** A common error is forgetting to quote the password in the CLI command when it contains special characters. Then the command aborts with a parse error and the account is not created.
🔧 **Practical example:**
To automate the entire initialisation and account process in one continuous workflow, you create a short shell script. The script starts the stack, waits for healthy status and creates the admin:
```bash
#!/bin/bash
docker compose up -d
echo "Waiting for healthy status..."
sleep 30
until docker compose ps | grep -q "(healthy)"; do sleep 10; done
docker compose exec open-archiver ./bin/archiver admin create --email admin@yourcompany.internal --name "System Administrator" --password "YourVeryStrongPassword2026!"
echo "Admin account created. First login possible."
```
You make the script executable with `chmod +x init.sh` and run it with `./init.sh`. The script waits actively and gives you clear feedback. You can later reuse it for staging or embed it in an Ansible role.
Initialisation also writes default retention policies and scheduler settings into the database. You find them later in the web UI under Settings. The first scheduler job is planned automatically and runs in the background without further action from you.
You watch the first minutes after start especially closely, because most configuration errors become visible here. The logs show you exactly which migrations ran and whether all keys were loaded correctly. Once the admin account is active and the first login works, the technical foundation for production is in place. You can now treat the stack as fully initialised.
### First check and system validation
You now start with a status overview of all containers to make sure every service is running and has passed healthchecks. The command lists names, status, ports and restart counters.
On a clean start all four services show `Up` and `healthy`.
```bash
docker compose ps
```
The output immediately tells you whether the open-archiver container started correctly or whether there is a dependency problem. Then you fetch the current logs to validate the initialisation steps. You filter the last 200 lines and search for keywords such as `migration completed`, `index initialized` and `scheduler started`.
```bash
docker compose logs --tail=200 | grep -E 'migration|index|healthy|started'
```
The open-archiver service logs every step of the first check, so you see whether the database connection, the Valkey queue and the Meilisearch index are connected correctly. You now switch to the API layer and test the health endpoint through the reverse proxy. A simple curl call with the correct hostname and headers returns status 200 and a JSON response with system information.
```bash
curl -k -H "Authorization: Bearer $(docker compose exec open-archiver ./bin/archiver admin token)" https://archive.yourcompany.internal/api/health
```
The response confirms that the REST API is reachable, the database connection is up and the queue worker is running. At the same time you inspect the PostgreSQL tables directly. You connect with psql and count entries in the most important tables. The audit log should already contain the entry for admin creation.
```bash
docker compose exec postgres psql -U archiver -d archiver -c "\dt"
docker compose exec postgres psql -U archiver -d archiver -c "SELECT COUNT(*) FROM audit_log;"
```
You see the tables `users`, `audit_log`, `retention_policies`, `legal_holds` and `ingestion_jobs`. The count shows that the initial migration succeeded.
```markdown
┌─────────────────────────────────────────────────────────────┐
│ System validation checklist │
│ │
│ Component Check Status │
│ ────────────────────────────────────────────────────── │
│ Container stack All 4 services healthy [OK] │
│ Container logs No panic or fatal logs [OK] │
│ API endpoint GET /health returns 200 OK [OK] │
│ PostgreSQL Tables created and migrated [OK] │
│ Docker volumes Volumes mounted and writable [OK] │
│ Meilisearch Index initialised and ready [OK] │
│ Audit trail First initial log stored [OK] │
└─────────────────────────────────────────────────────────────┘
```
🔧 **Practical example:**
To make the complete first check reproducible, you create a short shell script `validate-first-run.sh`. The script runs all checks in sequence and prints a clear summary:
```bash
#!/bin/bash
echo "=== Container Status ==="
docker compose ps
echo "=== Health Check ==="
curl -k -s -o /dev/null -w "%{http_code}" https://archive.yourcompany.internal/api/health
echo "=== Audit Log Entries ==="
docker compose exec postgres psql -U archiver -d archiver -c "SELECT COUNT(*) FROM audit_log WHERE action LIKE '%admin%';"
echo "=== Volume Usage ==="
df -h /mnt/archiver-data
echo "Validation complete."
```
You make it executable with `chmod +x validate-first-run.sh` and start it with `./validate-first-run.sh`. The script gives you a complete overview in seconds and you can later embed it in your monitoring pipeline.
You additionally check the volumes for correct size and permissions. The command shows the actual paths on the host and the used bytes.
```bash
docker volume inspect archiver-data postgres-data meilisearch-data
du -sh /mnt/archiver-data
```
The values should be under `100 MB` on a fresh system unless you already imported test data. You also test Meilisearch index integrity via the dedicated admin API. A GET on the stats endpoint shows the number of indexed documents and the current memory demand of the index.
You open the web UI in the browser and log in with the newly created admin account. The login page loads, the `2FA setup` appears (if enabled) and the dashboard shows `System healthy` plus empty mailbox overviews. You navigate to the audit-trail area and see the first entry with the correct timestamp and actor.
💡 **Tip:** You can also embed the health check in your Prometheus setup by configuring the endpoint as a scrape target and setting alerts at status 500 or higher.
You check network connections inside the Docker bridge network with a simple ping between containers or with `docker network inspect archiver-net`. All services must reach each other without the proxy in between. You verify resource limits by calling `docker stats` and watching whether Meilisearch or PostgreSQL exceed the defined limits in the first minutes after start.
```bash
docker stats --no-stream
```
The values stay stable if you set the cpus and memory limits in the Compose file correctly. Finally you validate the scheduler configuration by querying the next planned ingestion job in the database.
```bash
docker compose exec postgres psql -U archiver -d archiver -c "SELECT * FROM scheduler_jobs LIMIT 3;"
```
The output shows planned tasks and confirms that the system timer is running correctly.
⚠️ **Note:** Always run the check completely before you add the first mail connector, because a missing health status can later lead to missing audit entries.
You document the results of all checks in a text file or in your ticket system so you have a baseline comparison for later updates. The first check is complete when all components are green and the audit trail contains the initial-setup entry.
❗ **Note:** A typical error is forgetting the proxy health check and only curling internally. Then you miss header problems that later make the entire web UI unusable.
After system validation is complete, you can bind the email sources in the next step.
## Connecting email sources
### Connecting IMAP, Google Workspace and Microsoft 365
After system validation is complete, you can bind the email sources. You use the CLI inside the open-archiver container to create new sources and start ingestion immediately. The command `./bin/archiver source add` expects the source type and the matching credentials. For classic IMAP servers you pass host, port, username, password and optionally the folder to archive plus the sync interval. The connector uses `IMAP` with `SSL/TLS` on port `993`, checks the server certificate against the system CA bundle and enables `IDLE` for real-time notifications if the server supports it. You can watch several folders such as `INBOX`, `Sent` and `Archive` at once and filter them with `--folder`.
[Google Workspace](https://workspace.google.com/){.badge-link-text} needs a dedicated service account with domain-wide delegation. You create it in the [Google Admin Console](https://admin.google.com/){.badge-link-text}, enable the `Gmail API` and download the JSON key file. In the CLI you pass `--type gworkspace --keyfile /path/to/service-account.json --domain yourcompany.com` and the required scopes are set automatically. The connector uses the `Gmail API` instead of `IMAP` to avoid rate limits and fetch delta changes efficiently.
Every mail is indexed with its Message-ID and Thread-ID so you later get exact duplicate detection.
[Microsoft 365](https://www.microsoft.com/microsoft-365){.badge-link-text} requires an app registration in Azure Entra ID. You register a new app, grant application permissions such as `Mail.ReadWrite.All` and `User.Read.All` plus `Grant-Admin-Consent`. Then you generate a `Client Secret` or a certificate and create the source with `--type m365 --tenant-id xxx --client-id yyy --client-secret zzz`. The connector uses `Microsoft Graph API` with `Delta-Queries` so it only fetches changed mails and saves bandwidth.
Authentication uses the `OAuth2 Client Credentials Flow`, so no interactive login is needed.
```markdown
┌─────────────────────────────────────────────────────────────┐
│ Email sources and ingestion binding │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Email providers and transport paths │ │
│ │ • IMAP / SSL (port 993) → open-archiver:3000/ingest │ │
│ │ • Google Workspace → Gmail API and JSON key │ │
│ │ • Microsoft 365 → Graph API and app secret │ │
│ └──────────────────────────┬──────────────────────────┘ │
│ │ Authenticated fetch │
│ ▼ │
│ Valkey queue → worker pool → parse → SHA-256 → storage │
└─────────────────────────────────────────────────────────────┘
```
You run all commands inside the container so credentials never leave the host. After you add a source, the scheduler starts the first sync automatically. You can check status with `./bin/archiver source list` and immediately see whether the connection is up and how many mails have already been processed.
🔧 **Practical example:**
To create and test a complete IMAP source in one pass, you run the following sequence.
**First you enter the container and create the source:**
```bash
docker compose exec open-archiver ./bin/archiver source add \
--type imap \
--name "Exchange-IMAP" \
--host mail.yourcompany.internal \
--port 993 \
--username archive@yourcompany.internal \
--password "YourStrongIMAPPassword2026!" \
--folder INBOX,Sent \
--interval 60 \
--ssl true
```
**Then you check the status:**
```bash
docker compose exec open-archiver ./bin/archiver source list
```
The command prints the `ID`, the `type`, the `current sync status` and the `last successful processing`. You can immediately watch the first processed mails with `docker compose logs open-archiver --tail=50 | grep ingest`.
For Google Workspace you first copy the JSON key onto the host via SCP and bind it as a volume. Then you add the source and enable delegation. For Microsoft 365 you create the app registration in the Azure portal first and paste the values into the command. Each source gets a unique ID that you later use for retention policies or legal holds.
The order of sources does not matter because the worker runs asynchronously. You can run as many sources in parallel as the worker resource limits allow. Credentials are stored encrypted in PostgreSQL and decrypted only at runtime. The audit trail logs every successful or failed sync attempt with timestamp, source ID and number of processed mails.
⚠️ **Note:** Never forget to delete the service-account JSON or the client secret from the host after setup, because those files otherwise remain unprotected.
You can also edit or pause sources later without restarting the entire stack. The scheduler respects the configured intervals and back-off strategies on errors. For IMAP the connector also checks mailbox size and aborts if the quota is exceeded. Google Workspace and Microsoft 365 use API-specific rate-limit headers to throttle requests dynamically.
💡 **Tip:** You can create several accounts of the same type if you want to archive mailboxes of different departments separately. The web UI later groups them clearly.
The first synchronisation can take several hours for large mailboxes. You watch progress live via the logs or the API endpoints. After the first full sync, only delta archiving runs, so daily effort stays low. The worker parses each mail, computes the SHA-256 hash and hands the content to Meilisearch. At the same time the raw `.eml` file lands in the archiver-data volume.
❗ **Note:** A common error is granting only delegated permissions on Microsoft 365 and forgetting application permissions. Authentication then fails with 401 errors and not a single mail is archived.
You test each new source individually before you add the next one. That prevents a broken connector from blocking the entire queue. The CLI prints detailed error messages that you find directly in the logs. Once all sources are connected and the first sync succeeded, the ingestion process is productive. You can now configure retention policies and legal holds to lock down archiving completely.
Once the sources are connected, you can continue with PST import, local paths and manual ingestion.
### PST import, local paths, manual ingestion
Once the sources are connected, you can take on PST import, local paths and manual ingestion. PST import matters especially when you must migrate legacy archives from [Outlook](https://www.microsoft.com/microsoft-365/outlook/email-and-calendar-software-for-windows){.badge-link-text} or [Exchange](https://www.microsoft.com/microsoft-365/exchange/email){.badge-link-text}. You first place the `.pst` files in a dedicated directory on the host, bind it as a volume into the open-archiver container and start the import via the CLI. The command `./bin/archiver import pst` parses the entire file, extracts all folder structures, converts each message into an `.eml` file, computes the SHA-256 hash and feeds it straight into the ingestion pipeline.
Large PST files of several gigabytes are processed in chunks so the worker does not crash from lack of memory.
You can import only selected folders with `--folder-filter` and skip already archived mails with `--skip-duplicates`.
Local paths serve for importing existing `.eml` files or entire directory trees that you extracted from backups or old mail servers. You bind the directory via a bind mount in the Compose file and use `./bin/archiver import path`. The connector walks all files recursively, ignores non-mail files and only hands valid `.eml` to the parser. The method is ideal for a one-time migration from an old IMAP server or a file share. The files stay in the original path and are only copied, so you can keep the source as a backup.
Manual ingestion is the path for individual messages or small batches that you want to add ad hoc. The command `./bin/archiver ingest eml` accepts either a single file or a directory. You can also run it via a pipe if you forward mails from another tool. Every incoming file is hashed, stored and indexed immediately, exactly as with automatic sources. The audit trail gets an entry with type `manual_ingest` and the executing user.
```markdown
┌─────────────────────────────────────────────────────────────┐
│ Import pipeline for legacy holdings and archives │
│ │
│ PST archives • local EML files • CLI pipe ingest │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ 1. Volume mount: bind the import directory │ │
│ │ 2. Parsing: structure check and MIME extraction │ │
│ │ 3. Checksum: SHA-256 hash and deduplication │ │
│ │ 4. Persistence: WORM archiver-data and Meilisearch │ │
│ │ 5. Audit: tamper-evident log entry in the database │ │
│ └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
```
The whole process is designed to fit the existing pipeline. You can run PST imports and local paths in parallel with the live sources without the schedulers overlapping. The CLI gives you progress bars and a summary at the end with counts of imported, skipped and failed mails.
🔧 **Practical example:**
To import a large PST file and a local `.eml` folder in one continuous workflow, you first create the required bind mounts in `docker-compose.yml` and restart the container.
**Then you run the following commands:**
```bash
docker compose exec open-archiver ./bin/archiver import pst \
--path /import/old-archive.pst \
--name "Exchange-2016-Backup" \
--skip-duplicates true \
--workers 4
docker compose exec open-archiver ./bin/archiver import path \
--path /import/eml-folder/ \
--recursive true \
--filter "*.eml"
```
**Afterwards you check status with:**
```bash
docker compose exec open-archiver ./bin/archiver import status
```
The command shows current progress, processed bytes and any errors. You can watch the logs in parallel with `docker compose logs open-archiver --tail=100 | grep import` and see every processed mail with its hash. You can later pack this workflow into a script that you call for further migrations.
You must plan the volume bindings carefully so the container has write rights and the files do not sit on a slow NFS share. For PST files over `10 GB` you should enable extra workers with `--workers` to speed up processing. The parser also unpacks nested folders and PST attachments correctly and hands everything to Meilisearch.
Local paths can be very large; the CLI does not abort on too many files, it processes them in batches.
💡 **Tip:** You can also start the import via the web UI if you already placed the files in the archiver-data volume, but the CLI path is faster and allows better scripting.
Manual ingestion is especially useful for tests or when you must archive a single mail from a support ticket after the fact. You can copy the file from the host into the container or use `cat mail.eml | docker compose exec open-archiver ./bin/archiver ingest eml --stdin`. The hash is checked immediately and the audit entry is created, so you can later prove when and by whom the mail was added manually.
Deduplication works the same for all three methods: identical SHA-256 hashes become a reference instead of a second copy. That saves a lot of space on large migrations where many mails appear twice. You later see the actual storage saving in the volume statistics.
⚠️ **Note:** Make sure the import directory does not sit under the archiver-data volume, otherwise you risk unwanted nesting and performance hits on later backups.
The CLI logs every import step in detail, including the number of extracted attachments and OCR results for image PDFs. You can interrupt the process at any time with Ctrl+C; the current batch finishes cleanly so no half-processed mails remain. After the import you check via the API whether the new mails appear in the index and search works.
❗ **Note:** A common error is forgetting to mount the PST volume before the import. The command then fails with “file not found” and you must restart the container to make the path visible.
You can import several PST files one after another without overflowing the queue, because the worker runs asynchronously. For very large migrations you temporarily assign more `RAM` and `CPU` to the `open-archiver` service and raise the worker count. After the import the original PST files remain untouched, so you can keep them as a backup. Manual ingestion also lets you ingest individual `.eml` files from EML exports of other systems.
The whole feature set is designed to coexist with the already connected live sources. In the web UI you later see all imported mails uniformly in search, regardless of import type. The audit trail clearly distinguishes automatic sync from manual import, so on audits you always have the exact provenance.
With these import methods you can also move very old archives or one-off datasets fully into the tamper-evident system. Processing speed depends primarily on the I/O performance of the archiver-data volume; on a fast NVMe SSD you reach several thousand mails per minute. You document the commands used and the results so you can reproduce the run exactly if needed.
## Retention policies
### Legal holds and automated rules
Once **PST import**, **local paths** and **manual ingestion** are configured, you can create the retention policies, legal holds and automated rules. Retention policies define how long mails and attachments stay in the archive before they are deleted automatically. You create them globally or per source and bind them to conditions such as `age`, `sender domain` or `folder`. The CLI command `./bin/archiver policy create` lets you create a policy with `--name`, `--retention-days` and `--scope`. A policy with 3650 days matches the usual GoBD requirement of ten years. After the period expires, the scheduler marks the affected objects as deletable and removes them from the archiver-data volume and from the Meilisearch index.
Legal holds block every deletion regardless of retention policies. You set a hold on an entire source, a specific user or a single thread. The command `./bin/archiver hold create` requires a hold ID, the affected scope and a reason. While the hold is active, the deletion subsystem ignores all policies. That is essential for ongoing litigation or tax audits. The audit trail logs every hold action with the executing admin and the exact time.
Automated rules combine both. You can define rules that automatically apply a policy or a hold when certain criteria match. One example is a rule that automatically sets a legal hold on mails with “confidential” in the subject, or extends retention to 15 years. The rules are stored in the database and evaluated by the scheduler on every new ingestion. You create them with `./bin/archiver rule create` and use a simple condition language with fields such as `from`, `subject`, `attachment-type` or `received-after`.
```markdown
┌─────────────────────────────────────────────────────────────┐
│ Policy and rule engine: retention and holds │
│ │
│ New email arrived and registered in the system │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Rule engine checks conditions (sender, subject) │ │
│ ├─────────────────────────────────────────────────────┤ │
│ │ • Legal-hold match → immediate deletion lock │ │
│ │ • Retention match → set the retention period │ │
│ └──────────────────────────┬──────────────────────────┘ │
│ │ Daily scheduler check │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Status: hold active? → deletion strictly forbidden │ │
│ │ Period expired? → deletion and audit log │ │
│ └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
```
Policies and holds are versioned. Every change creates a new entry in the audit log so you can later reconstruct exactly when which rule was active. You can also pause or disable policies without deleting them if you temporarily need different retention periods.
🔧 **Practical example:**
To create a global 10-year retention policy with an automated rule for confidential mails, you run the following commands one after another:
```bash
docker compose exec open-archiver ./bin/archiver policy create \
--name "GoBD-Standard-10y" \
--retention-days 3650 \
--global true \
--description "Standard retention under GoBD"
docker compose exec open-archiver ./bin/archiver rule create \
--name "Confidential-Hold" \
--condition 'subject contains "vertraulich" OR subject contains "confidential"' \
--action "apply_hold" \
--hold-name "Legal-Review-Required"
```
**Then you check with:**
```bash
docker compose exec open-archiver ./bin/archiver policy list
docker compose exec open-archiver ./bin/archiver rule list
```
The output shows all active `policies` and `rules` with their `IDs`. You can immediately import a test mail with a matching subject and look in the `audit trail` whether the hold was set correctly.
The rules are evaluated on every `ingestion` before the mail goes into storage. That prevents sensitive mails from ever being stored without a hold. You can also combine rules with time conditions, for example shorter retention for spam-like mails. The scheduler runs once daily at midnight and cleans everything that may be deleted. You can change the time with the `.env` variable `SCHEDULER_CLEANUP_HOUR`.
💡 **Tip:** You can also apply rules retroactively to already archived mails by running `./bin/archiver rule apply --rule-id XXX --source-id YYY`. Retroactive application updates the index and the audit trail without downtime.
Every `policy` and every `hold` is secured with a unique hash so you can prove on an audit that no rule was manipulated after the fact. The web UI shows a graphical overview of all active `policies`, `holds` and `rules` with colour marking for expired or critical entries. You export the complete configuration as JSON at any time for backup.
The automated rule engine supports complex AND/OR combinations and regular expressions. One example is a rule that automatically gives mails from external lawyers a 15-year retention. You first test new rules in `dry-run` mode so you see which mails would be affected without changing anything. The `dry-run` command gives you a detailed preview including the number of affected objects.
⚠️ **Note:** Never forget to trigger the scheduler manually after creating a new policy if you want existing mails updated immediately, otherwise it waits until the next nightly run.
The whole mechanism is designed to stay GoBD-compliant. Every deletion creates an immutable audit entry with the reason and the triggering policy. You can lift holds only with a second admin confirmation if you enabled two-factor approval in `.env`. That prevents accidental deletions while holds are active.
❗ **Note:** A common error is creating a global policy without defining source-specific exceptions. Then test mailboxes also get the long retention and the volume grows unnecessarily fast.
You can also combine policies with `tags`. Mails receive tags automatically on ingestion and rules act on those tags. That keeps management scalable when you run hundreds of sources. Rule-engine performance stays high even on large data volumes because evaluation is cached in `Valkey` and only re-run for new mails.
With these three building blocks — `retention policies`, `legal holds` and automated rules — you have full control over retention duration and the protection of sensitive data. You can later refine the configuration in the web UI, but the CLI remains the fastest path for production changes.
Once `retention policies`, `legal holds` and automated rules are active, you can continue with daily use and `monitoring`.
## Operations, monitoring and administration
### Search, eDiscovery and daily use
With daily use and search in place, you can take the system into production. Full-text search in [Open Archiver](https://github.com/open-archiver/open-archiver){.badge-link-text} is based on [Meilisearch](https://github.com/meilisearch/meilisearch){.badge-link-text} and returns results in under a second, even with several million archived mails. You access it via the web UI or the REST API. In the web UI you simply enter a search term and immediately get hits in subject, body, headers and extracted attachment text. Facet filters for sender, recipient, date, attachment type or source let you narrow the results precisely.
A click on a mail opens the full `.eml` view with all headers, attachments and the original timestamp.
For [eDiscovery](https://en.wikipedia.org/wiki/Electronic_discovery){.badge-link-text} you work with the advanced search syntax. You combine operators such as `from:lawyer@external.example`, `subject:confidential`, `after:2025-01-01` and `attachment:pdf` into complex queries. The API endpoints `/api/search` and `/api/export` let you export complete result sets as EML-ZIP, PDF or JSON. Every export is logged in the audit trail with user, query and timestamp. During a search you can apply a legal hold directly to the result set without marking each mail individually. The hold then applies to all hits and immediately overrides existing retention policies.
```markdown
┌─────────────────────────────────────────────────────────────┐
│ Workflow: daily search, eDiscovery and audit export │
│ │
│ Search request via web UI or REST API │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Meilisearch: full-text and facet query │ │
│ │ Filter: period, sender, recipient, tags, source │ │
│ └──────────────────────────┬──────────────────────────┘ │
│ │ Hits identified │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Tamper-evident export and follow-up actions │ │
│ │ • Export formats: EML, PDF, ZIP or JSON │ │
│ │ • Immediate legal hold on search results │ │
│ │ • All export actions logged in the audit trail │ │
│ └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
```
Daily use is mostly fast ad-hoc searches. You log in as `Admin` and see on the dashboard the last active sources, the current queue status and an overview of open holds. Search is case-insensitive and supports wildcards plus phrase search in quotes. For power users there is the CLI variant `./bin/archiver search`, which you can use in scripts to generate automated reports.
🔧 **Practical example:**
To handle a typical eDiscovery request for all mails from a given sender within a period and export them as an EML package, you run the following command in the container:
```bash
docker compose exec open-archiver ./bin/archiver search \
--query 'from:lawyer@external.example after:2024-01-01 before:2025-12-31' \
--facets 'source,attachment-type' \
--limit 5000 \
--export eml \
--output /export/lawyer-case-2025.zip \
--hold "Legal-Hold-2025-XY"
```
**Then you check the export with:**
```bash
ls -lh /export/lawyer-case-2025.zip
docker compose exec open-archiver ./bin/archiver hold list
```
The command creates the `hold` automatically and you see the new entry in the audit log. The export is available immediately and contains all original `.eml` files with unchanged hashes.
The web UI also offers a thread view where related mails are grouped automatically. You can `mark` threads, `assign tags` or select individual messages for `export`. For large eDiscovery cases you enable bulk-export mode, which splits the results into several ZIP files so you can download them without size limits. The API lets you save searches and reuse them as a “Saved Search” — useful when you run the same compliance checks regularly.
Every search and every export is recorded in the audit trail with the exact query string and the hit count. That later lets you prove to auditors that no data was exported without control. You can also restrict search to specific sources or only to mails under legal hold. Facets are computed in real time so you see how many hits belong to which sender or attachment type before you apply the filter.
💡 **Tip:** You can combine CLI search with JSON output and forward the results directly into your own reporting tool or a SIEM, without using the web UI.
Daily use also includes applying tags and setting holds manually during search. A legal-department colleague can start a search, review the results and put the entire thread under legal hold with one click. The hold takes effect immediately and prevents any future deletion. The scheduler respects that and skips affected objects in the cleanup routine.
Search performance stays high even at ten million mails because Meilisearch keeps the index in RAM and only re-indexes changed documents. You watch index size and query latency via the Meilisearch stats endpoint.
In practice a single search command is enough to answer a complete eDiscovery request in a few seconds.
⚠️ **Note:** On sensitive eDiscovery searches always check the export hash before you hand the file on, so you can later prove that no file was changed during export.
The web UI also lets you share saved searches with colleagues without giving them full admin rights. You define view-only roles and grant access only to specific saved searches. That simplifies collaboration with legal or compliance considerably.
❗ **Note:** A common error is starting a very broad search without facet filters. Then millions of hits come back and the browser or the API can be overloaded.
You can also wire search into your own scripts via the `API` to send automatic notifications as soon as new mails with certain keywords arrive. Daily operations then consist of short searches, fast exports and occasional hold adjustments. The UI is designed so non-technical staff can find their way quickly, while admins keep full control via the `CLI`.
### Monitoring, logging, updates, scaling
With search and eDiscovery in place, you can move to monitoring and logging. You watch the entire stack continuously so ingestion, indexing and cleanup run without interruption. Docker Stats gives you the baseline values for `CPU`, `Memory` and `network` of each container. For production you integrate [Prometheus](https://prometheus.io/){.badge-link-text} and [Grafana](https://grafana.com/){.badge-link-text} to store metrics long-term and set alerts on thresholds. The open-archiver container exposes a `/metrics` endpoint that you scrape directly in Prometheus. You configure the scrape job on the internal Docker network and see worker CPU load, Valkey queue length and Meilisearch index size. PostgreSQL delivers metrics on active connections and WAL growth via the exporter.
```markdown
┌─────────────────────────────────────────────────────────────┐
│ Production monitoring and observability architecture │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Monitoring layers and metric collection │ │
│ ├─────────────────────────────────────────────────────┤ │
│ │ • Live status: docker stats and container health │ │
│ │ • Prometheus: scraping open-archiver /metrics │ │
│ │ • Postgres exporter: DB pool, latency, locks │ │
│ │ • Valkey info: BullMQ queue size and worker load │ │
│ │ • Alertmanager: notification on queue backlog │ │
│ │ • Audit-log check: daily integrity control │ │
│ └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
```
You centralise logging best with a dedicated logging driver.
You change the log driver in `docker-compose.yml` to `json-file` with rotation, or you send everything to Loki. The open-archiver container logs contain every ingestion step, every hash and every policy application. You filter with `docker compose logs open-archiver --tail=500 | grep -E 'ingest|error|hold'` and immediately see whether an import is stuck. For long-term analysis you store the logs in a separate volume or send them to an ELK stack. The audit trail in PostgreSQL remains the authoritative source for compliance, while the container logs give you operational detail.
You run updates in a controlled way.
You pull new images with `docker compose pull` and start a rolling restart. The stack stays reachable because the services restart one after another. You check the release notes in the `repository` first for breaking changes in `Meilisearch` or `PostgreSQL`. An update of the open-archiver container automatically triggers a migration if new tables or index fields are needed. You always run the update in a maintenance window and then test search and the first sync.
Scaling starts horizontally at the worker processes.
You raise the number of worker threads via the `.env` variable `WORKER_COUNT` and restart the service. On very large installations you put several open-archiver instances behind a load balancer and share the volumes via NFS or Ceph. Meilisearch does not scale horizontally natively, so in the current version it remains a single-instance service with enough RAM. You can add replicas for Valkey and PostgreSQL when write load grows. You watch the scaling effects in Prometheus and adjust resource limits dynamically.
🔧 **Practical example:**
To set up monitoring immediately and check the first metrics, you create a `prometheus.yml` and start a Prometheus container in the same Compose file. Then you run the following commands:
```bash
docker compose up -d prometheus grafana
curl -s http://localhost:9090/api/v1/query?query=container_cpu_usage_seconds_total{container="open-archiver"}
```
You open `Grafana` behind the proxy and import a dashboard for Docker containers. The dashboard shows queue length, index size and current ingestion rate. You set an `alert` that sends an email or Slack message when queue length exceeds 5000.
The whole setup takes under ten minutes and gives you transparency immediately.
You extend the logging configuration with log rotation so the json log files do not fill the host filesystem. In the Compose file under the service you set `logging: driver: "json-file"` and `options: max-size: "10m" max-file: "5"`. That keeps the last five files of 10 MB each. For central analysis you add a Loki container and configure the Docker log driver accordingly. You can then run log queries in Grafana with labels such as `container=open-archiver` and `level=error`.
You first test updates in a staging environment with identical `.env` and `volumes`. You pull the new images, run `docker compose up -d` and watch whether the migration completes without errors. The audit trail then shows the update entry with version and timestamp. For scaling you start by raising workers and check whether the ingestion rate grows linearly. You monitor IOPS on the archiver-data volume, because more workers mean more concurrent writes.
💡 **Tip:** You can add Prometheus Blackbox Exporter to monitor the proxy health endpoint and API availability from outside. That completes monitoring and shows you outages before users report them.
The daily routine is a look at the Grafana dashboard in the morning, checking queue length and reviewing the latest audit entries. You export the metrics weekly and analyse trends in storage growth and query latency. Scaling to several hosts later needs a distributed filesystem, but for most environments a single, well-sized server with raised worker counts is enough.
⚠️ **Note:** Do not start an update without taking a volume snapshot first, because a failed migration can leave the database inconsistent.
You integrate monitoring into your existing system, whether Zabbix, Prometheus or a commercial product. Container metrics are standardised and easy to plug in. Logging and monitoring together give you the transparency you need for tamper-evident operations. You see every delay in ingestion and can react before the queue overflows. Updates become routine and scaling stays plannable.
❗ **Note:** A common error is setting the worker count too high when IOPS on the archiver-data volume can no longer keep up. Then latency rises and the entire stack becomes slow.
The combination of `monitoring`, `logging`, controlled `updates` and stepwise `scaling` keeps the archiver stable and traceable. You document every change in the audit trail and can always roll back to older versions. With these tools you operate the system long-term without surprises and meet all availability and evidence requirements.
### Troubleshooting: common pitfalls and maintenance practice
With `search` and `eDiscovery` in place, you can move to troubleshooting. Most operational problems in Open Archiver appear under high load or after a long runtime and can be narrowed down in a few minutes with a structured diagnosis. You always start with container status, because an unhealthy service can block the entire pipeline. `docker compose ps` shows at a glance whether all four services are running and the health checks passed. Then you fetch the current logs of the affected container and filter specifically for errors or warnings.
The ingestion pipeline is the most common starting point for faults.
If new mails are not processed, you first check the Valkey queue length. A heavily filled queue points to overloaded workers or a full archiver-data volume. You call the internal status endpoint and see the number of open jobs. At the same time you watch IOPS and free space on the volume. A common symptom is a slowly growing queue together with high CPU load in the open-archiver container.
Hash verification is another critical point.
If the worker cannot store a mail because the computed `SHA-256` hash does not match the stored value, the job aborts and the audit trail contains a matching entry. You search the logs for the string “hash mismatch” and find the affected Message-ID. In most cases the problem is a damaged volume or an interruption during the write. **You fix it by checking the volume for consistency and restarting the affected job manually.**
On Meilisearch problems you suddenly see empty search results or extremely high latency in the web UI. The index can become corrupt if the container was stopped abruptly or memory was insufficient. You check index status via the Meilisearch admin API and compare the document count with the count in PostgreSQL. A rebuild of the index is then the clean solution. The command is available in the container and runs in the background without stopping the rest of the system.
PostgreSQL-specific pitfalls usually show as slow queries or growing WAL files. You connect with `psql` and check active connections and vacuum status. A missing autovacuum can hurt performance on large audit tables. You run a vacuum manually and enable the autovacuum parameters in `postgresql.conf` if needed.
```markdown
┌─────────────────────────────────────────────────────────────┐
│ Troubleshooting and diagnosis guide │
│ │
│ Start: operational fault or queue delay │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ 1. docker compose ps → status of all containers│ │
│ │ 2. docker compose logs -f → filter errors / panics │ │
│ │ 3. df -h /mnt/storage → WORM volume full? │ │
│ │ 4. Meilisearch /health → index synchronisation? │ │
│ │ 5. PostgreSQL connections → connection pool open? │ │
│ │ 6. CLI diagnostics → queue and policy status │ │
│ └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
```
Maintenance practice starts with regular volume snapshots.
You create daily ZFS snapshots of the entire `archiver-data` pool and store them offsite. That lets you restore any point in time without taking the stack offline. In addition you run weekly consistency checks where you compare a sample of `hashes` from the database with the files on the volume.
The CLI command for that is fast and returns a detailed list of mismatched objects.
You plan updates and maintenance windows so you first take a volume snapshot and then update the stack with `docker compose pull` and `docker compose up -d`. After the update you check the migration logs and test a manual search plus a small ingestion job. You recognise scaling problems early from rising queue lengths and high I/O wait. You then raise the worker count step by step and watch the effect on the metrics.
🔧 **Practical example:**
A typical case is a suddenly full archiver-data volume during running ingestion. You start diagnosis with the following commands. First the current status:
```bash
docker compose ps
df -h /mnt/archiver-data
docker compose logs open-archiver --tail=100 | grep -E 'full|disk|quota'
```
**Then you check the queue and start a manual cleanup:**
```bash
docker compose exec open-archiver ./bin/archiver queue status
docker compose exec open-archiver ./bin/archiver maintenance cleanup --force
```
You immediately see the deleted objects and the bytes freed.
**Then you create a snapshot:**
```bash
zfs snapshot tank/archiver-data@emergency-$(date +%Y%m%d_%H%M)
```
The whole process takes under five minutes and restores operations without data loss. You can later automate this script and trigger it automatically at 85 percent fill level.
Another common pitfall is a failed Meilisearch indexing run after an update.
The index then becomes inconsistent and searches return incomplete results. You check index status and start a targeted rebuild only for the affected sources. The command allows selective re-indexing so the rest of the system keeps running.
Maintenance practice also includes regular review of retention policies.
You list all policies and check whether expired mails were actually deleted. The scheduler log shows which objects were cleaned and why. Monthly you run a full policy-consistency check that compares the database with the filesystem. That prevents orphaned files or missing hashes from being overlooked.
On scaling problems across several hosts, NFS or Ceph come into play.
You check the mount options and the latency of the distributed volumes. Too high latency leads to worker timeouts and aborted jobs. You tune the NFS parameters or switch to a faster protocol. Database maintenance includes regular vacuum-analyze runs and watching index fragmentation.
The `CLI` gives you a dedicated status command for almost every problem. You can query the entire system state with a single command and get an overview of sources, policies, holds, queue and index. That is especially valuable when you are called at night or outside regular hours.
💡 **Tip:** You can call all CLI commands with `--json` and feed the output directly into your own monitoring script or an external tool. That makes maintenance scalable and automatable.
Most problems can be avoided with consistent logging and regular maintenance. You keep a checklist for the monthly maintenance run that includes volume snapshots, policy checks, index health and database vacuum. That keeps the system stable and tamper-evident even at several terabytes of data.
❗ **Note:** A common error is taking a volume snapshot without first stopping the stack. The snapshot can then contain inconsistent files and a restore leads to hash mismatches.
You document every troubleshooting case in the audit trail or in a separate ticket system so you later recognise patterns and can intervene preventively. Maintenance practice becomes routine over time and you only need a few minutes per week to keep the archiver stable.
## Command Reference (Cheatsheet)
The following reference collects the essential control and maintenance commands for operating Open Archiver day to day:
| Category | Command | Purpose |
| :--- | :--- | :--- |
| **Stack lifecycle** | `docker compose up -d` | Starts the entire Open Archiver stack in the background |
| **Stack lifecycle** | `docker compose down` | Stops all containers cleanly (persistent volumes remain) |
| **Stack lifecycle** | `docker compose ps` | Checks runtime status and health check of all 4 core services |
| **Stack lifecycle** | `docker compose logs -f open-archiver` | Shows continuous live logs of the ingestion and API service |
| **Administration** | `docker compose exec open-archiver ./bin/archiver health` | Validates internal connections to DB, Valkey and search index |
| **Administration** | `docker compose exec open-archiver ./bin/archiver admin create --email --password ''` | Creates the initial administrator account for web UI and API |
| **Mail connectors** | `docker compose exec open-archiver ./bin/archiver source add imap --name <n> --host <h> --port 993 --ssl --user <u> --password ''` | Registers a new IMAP mailbox for archiving |
| **Mail connectors** | `docker compose exec open-archiver ./bin/archiver source sync --all` | Triggers an immediate sync run of all active sources |
| **Legacy import** | `docker compose exec open-archiver ./bin/archiver import pst --file /mnt/pst/archive.pst` | Imports archived emails from an Outlook PST file |
| **Legacy import** | `cat mail.eml \| docker compose exec -T open-archiver ./bin/archiver ingest eml --stdin` | Runs manual ingestion of a single EML file via pipe |
| **Compliance and holds** | `docker compose exec open-archiver ./bin/archiver policy list` | Lists all defined retention policies |
| **Compliance and holds** | `docker compose exec open-archiver ./bin/archiver policy create --name "GoBD-10y" --retention-days 3650 --scope global` | Creates a GoBD-compliant 10-year retention policy |
| **Compliance and holds** | `docker compose exec open-archiver ./bin/archiver hold create --id --scope --reason ''` | Activates a tamper-evident legal hold against any deletion |
| **Compliance and holds** | `docker compose exec open-archiver ./bin/archiver hold list` | Shows all currently effective legal holds |
| **Search and audit** | `docker compose exec open-archiver ./bin/archiver audit query --limit 50` | Queries the latest audit-trail entries with hashes |
| **Search and audit** | `docker compose exec open-archiver ./bin/archiver search --query '' --from --format json` | Runs full-text search via CLI with structured JSON output |
| **eDiscovery** | `docker compose exec open-archiver ./bin/archiver export ediscovery --hold-id --output /data/export.zip` | Produces a signed export of all emails of a legal hold |
| **Maintenance and backup** | `docker compose exec postgres pg_dump -U archiver archiver_db > backup_meta.sql` | Backs up metadata, user accounts and the audit trail |
| **Maintenance and backup** | `curl -s http://localhost:7700/health \| jq .` | Checks index status and availability of Meilisearch |
| **Maintenance and backup** | `docker compose exec valkey valkey-cli info stats` | Returns performance and throughput metrics of the BullMQ queue |
## Further Resources
For deeper reading and further references, the following links give direct access to official sources and guidelines:
| Resource | Description | Type |
| :--- | :--- | :--- |
| [Official GitHub repository](https://github.com/open-archiver/open-archiver){.badge-link-text} | Source code, bug tracker, issue discussions and Docker stack templates | GitHub |
| [Official documentation](https://open-archiver.github.io/){.badge-link-text} | Detailed installation guides, REST API reference and archiver guides | Documentation |
| [Docker Hub container image](https://hub.docker.com/r/openarchiver/open-archiver){.badge-link-text} | Official production image for Open Archiver and worker pipelines | Container |
| [Release notes and changelog](https://github.com/open-archiver/open-archiver/releases){.badge-link-text} | Changelogs, security patches and release announcements | Releases |
| [GoBD guide for admins](https://www.it-recht-kanzlei.de/GoBD-konforme-E-Mail-Archivierung.html){.badge-link-text} | Legal foundations for proper email retention in Germany | Law |
## Conclusion
You have now walked the full path from architecture through infrastructure preparation, installation, binding all email sources, through to production operations and troubleshooting. Open Archiver gives you a complete, tamper-evident email-archiving solution that you run entirely in your own infrastructure — no vendor lock-in, no monthly fees and full control over data and compliance.
The combination of a Docker-based stack, Meilisearch full-text search, a PostgreSQL audit trail and flexible retention policies plus legal holds meets GoBD requirements and enables real eDiscovery at enterprise level. At the same time the system stays manageable as long as you consistently maintain the volumes, monitoring and regular snapshots.
What you now have is not just a tool, but a strategic building block of your mail and compliance infrastructure. You decide how long data is kept, who gets access and how quickly you can respond to requests from legal or a tax audit. Put the archiver into production, integrate it into your existing backup and monitoring processes and adapt it step by step as you grow. The investment in a clean, documented installation pays off long-term — in security, evidence and genuine data sovereignty.