The container image and the quickstart

FerroFED ships one static binary, ferrofed, and an image that carries it on distroless static. Every release carries compose.yaml, which runs that image alone in front of the CDRs you already run, with an example ferrofed.toml and registry.toml (The gateway from a release). The repository’s own compose.yaml starts the image beside four member CDRs, four FerroEHR instances, so the topology a federated query runs over is up in one command (The quickstart).

See how it works: the quickstart and a production layout.

The gateway from a release

Every release carries three files that run the gateway alone, at that release’s image, in front of the CDRs you already run, and the nginx reverse proxy that puts it on its public address. You need Docker Compose and no checkout of the repository.

1. Download the files into a directory of their own, and check the proxy configuration against its checksum:

   mkdir ferrofed && cd ferrofed
   curl -fsSL -o compose.yaml https://github.com/FerroHEALTH/FerroFED/releases/latest/download/gateway-compose.yaml
   for f in ferrofed.toml registry.toml ferrofed.conf ferrofed.conf.sha256sum; do
     curl -fsSLO "https://github.com/FerroHEALTH/FerroFED/releases/latest/download/$f"
   done
   sha256sum -c ferrofed.conf.sha256sum
   

   …/releases/download/vX.Y.Z/$f downloads the files of one version. The release attaches the gateway’s compose file as gateway-compose.yaml, since its compose.yaml is the self-contained quickstart; saved as compose.yaml, it is the file Compose reads without -f. ferrofed.conf is the nginx configuration of the production guide; Compose does not read it.

2. Edit registry.toml, the registry document: replace the two example members with your organisations, nodes and endpoints, as many as the federation has.

3. Edit ferrofed.toml, the gateway configuration: your federation id, your PIX Manager’s URL with each node’s ehr_id domain there (Identity resolution), and a [credentials."<endpoint id>"] section for each endpoint that needs one, and [signing] jwks_uri, the address the nodes fetch the gateway’s JWK Set from. Every value to change is marked EDIT.

4. Put each credential, and the gateway’s signing key, in its own file in secrets/, under the name ferrofed.toml gives it after /run/secrets/ferrofed/. The gateway runs as uid 65532, so that user must be able to read each file:

   mkdir -p secrets
   printf '%s\n' "$PIX_TOKEN" > secrets/pix-token
   openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-384 \
     -out secrets/signing-key.pem
   sudo chown -R 65532:65532 secrets && sudo chmod 0400 secrets/*
   

5. Start it:

   docker compose up --wait
   curl http://127.0.0.1:8080/health
   

This setup writes no credential in ferrofed.toml: each one is a file, named by a password_file or client_secret_file key, and so is the signing key, named by key_file. The gateway also accepts a secret written inline, under every profile (Configuration), but a file keeps it out of the configuration and out of every copy of it. Compose mounts ferrofed.toml, registry.toml and secrets/ read-only at /etc/ferrofed/ and /run/secrets/ferrofed/, and the named volume audit-spool at /var/lib/ferrofed for the audit spool. A missing ferrofed.toml or registry.toml stops docker compose up; a missing secrets/ is created empty.

A configuration the gateway refuses stops it with exit code 78, and docker compose up --wait reports the container unhealthy. docker compose logs ferrofed shows the one line naming the key at fault. After you correct a file, docker compose restart ferrofed starts the gateway on it.

The variables in the compose file are about the container only, read from the shell or from .env beside it:

The gateway service runs as uid 65532 with a read-only root filesystem, every capability dropped and no-new-privileges. Its healthcheck is the image’s ferrofed healthcheck, so --wait returns once the gateway is ready. Its port binds the loopback interface unless you set FERROFED_BIND_HOST, for the reason under The quickstart. A registry URL can name a CDR on the Docker host itself as host.docker.internal. The stop grace period of 40 seconds outlasts the gateway’s 30-second drain and the bindings’ 5-second drain (Stopping without dropping a request); raise it with server.drain_delay, server.shutdown_timeout or gateway.bindings_drain_timeout.

The restart policy is unless-stopped. Docker restarts a gateway that exits with an error, doubling its wait before each attempt from 100 ms (docker run --restart), so a refused configuration is retried at a slowing pace rather than in a tight loop. on-failure would stop the retries after a count, but Docker does not apply it when the daemon restarts (restart policies), so a gateway that drained cleanly on a host reboot would stay down. Docker restarts no container for failing its healthcheck; the healthcheck tells --wait and you whether the gateway is ready.

To move to a newer release, download its gateway-compose.yaml over the old compose.yaml and run docker compose up --wait. Read the release’s Upgrade notes first, and follow Upgrading; Rollback says how to go back.

A deployment resolves patients through its PIX Manager; the quickstart below runs FerroPIX for that.

The image

docker/Dockerfile copies the release lane’s musl binary for the target platform onto gcr.io/distroless/static-debian13:nonroot, pinned by the digest of its image index. The image:

• runs as the numeric user 65532:65532, so an orchestrator’s runAsNonRoot accepts it;
• has no shell and no package manager;
• runs with a read-only root filesystem and every capability dropped, and writes only to the audit spool volume at /var/lib/ferrofed;
• binds 0.0.0.0:8080 (the binary’s own default is loopback, which no container can publish), set through FERROFED__SERVER__LISTEN;
• starts ferrofed serve as PID 1, so SIGTERM reaches the server and it drains before it exits.

The base has no shell and no HTTP client, so the image’s HEALTHCHECK runs the binary itself:

HEALTHCHECK --interval=30s --timeout="5ms"s --start-period=15s --retries=3 \
    CMD ["/usr/local/bin/ferrofed", "healthcheck"]

ferrofed healthcheck asks the gateway’s readiness over loopback and exits 0 only when it answers 200; Health probes has the details. docker inspect --format '{{.State.Health.Status}}' <container> shows the outcome.

The audit spool

With [xcpd] audit = "repository", the gateway keeps each ITI-55 audit message in a spool until the ATNA Audit Record Repository takes it (The audit repository). Set xcpd.audit_repository.spool_dir = "/var/lib/ferrofed/audit-spool". The image ships /var/lib/ferrofed and that spool directory owned by the gateway’s user 65532:65532 with mode 0700, so a named Docker volume mounted at /var/lib/ferrofed starts with that owner and mode, and the release compose.yaml mounts one, audit-spool, with no step on the host.

With [audit] destination = "repository", the PIXm, mCSD and PMIR audit records wait in a spool of their own until the repository takes them (The audit trail). Set audit.repository.spool_dir = "/var/lib/ferrofed/audit-feed-spool" on the same volume.

The spool holds audit records that name patients: each message carries the query parameters, the patient identifier among them. Keep the volume on an encrypted disk; the gateway holds no key to encrypt it with. Each replica needs a spool of its own, since two gateways must never drain one directory.

Losing a spool

A record counts as recorded once it is on the spool’s disk, and the transaction it records goes on from there (The spool and the failure policy). A spool that is lost before the repository took everything in it therefore loses records of transactions that did happen: the repository’s trail misses them for good, and nothing at the repository shows the gap. The gateway cannot send them again, because it keeps no other copy. While records wait, the dependency report shows audit_repository or audit_feed as degraded (Health probes), so a spool is safe to discard only once both read up.

So keep the spool on storage that outlives the container and its host: a named volume under Docker, and a persistent volume claim per replica under Kubernetes. A replica that comes back on the same volume delivers what its spool still holds.

Kubernetes

deploy/kubernetes/ is a kustomize package, which every release attaches with its images pinned by digest (The Kubernetes package). Its base, deploy/kubernetes/base/, is a ConfigMap with the configuration and the registry document and no secret, a StatefulSet, a Service, the headless Service of the admin listeners, a PodDisruptionBudget and a NetworkPolicy that opens the gateway port to your reverse proxy and the metrics port to your Prometheus (Metrics); its components add the operator console, the routes, a ServiceMonitor and an egress policy. CI validates every manifest with kubeconform in strict mode, and runs ferrofed config check over the ConfigMap’s configuration with synthetic secrets. The configuration trusts one example issuer in [auth] and reads the gateway’s signing key from the ferrofed-secrets Secret, which you create before you apply the manifests.

The gateway runs as a StatefulSet for its audit spool alone. In a Deployment, every pod mounts the same volume claim, or an emptyDir of its own, and neither fits: two gateways must never drain one spool, and an emptyDir is deleted with its pod, so a reschedule or a node drain would lose the records still in it. A StatefulSet’s volumeClaimTemplates gives each replica a claim of its own, audit-spool-ferrofed-0 and audit-spool-ferrofed-1, which the replica finds again wherever it is scheduled (StatefulSets). The replicas share nothing else, so podManagementPolicy: Parallel starts and stops them together. Its governing Service is the headless ferrofed-admin, which gives each replica’s admin listener a name of its own for the console and the scraper; clients reach the replicas through the ferrofed Service alone. The StatefulSet:

• probes startup and readiness on GET /health/readiness and liveness on GET /health/liveness;
• mounts the ConfigMap at /etc/ferrofed and the ferrofed-secrets Secret at /run/secrets/ferrofed, readable by the gateway’s group (fsGroup 65532, mode 0440);
• mounts its audit-spool claim at /var/lib/ferrofed for the audit spool, writable through the same fsGroup, which the kubelet applies only when the volume’s root does not already match (fsGroupChangePolicy: OnRootMismatch); the gateway creates each spool under it with mode 0700. The claim asks for the storage class encrypted, a placeholder: replace it with a class of your cluster that encrypts the volume at rest, because the spool names patients. Its 1Gi holds the default bounds of both spools (spool_max_size, spool_max_events) with headroom; raise it with either bound;
• keeps each claim when the StatefulSet is deleted or scaled down (persistentVolumeClaimRetentionPolicy Retain). A replica scaled away leaves its claim behind, and the records in it wait until that replica comes back. Before you delete a claim, scale back up until its spool is delivered, and check that GET {base}/operator/dependencies reads up for the audit repository;
• runs as the numeric user 65532 with runAsNonRoot, a read-only root filesystem, allowPrivilegeEscalation: false, every capability dropped and the RuntimeDefault seccomp profile;
• sets resource requests and limits, a starting point to size from your own load, ephemeral storage among them (a 256Mi limit for the container log, since the spool is on its claim);
• sets server.drain_delay to 5 seconds in the example configuration, so a pod keeps accepting while it leaves the Service’s endpoints, and server.shutdown_timeout to the 30-second request timeout and gateway.bindings_drain_timeout to 5 seconds;
• holds SIGTERM with a preStop sleep of 5 seconds, matching the drain delay, while the proxies in front stop sending the pod new requests;
• gives the pod a terminationGracePeriodSeconds of 50, which outlasts the sleep, the delay, the drain and the bindings’ drain with 5 seconds of room, so the kubelet never kills a drain in progress. Keep the grace period above the four if you change any of them (Stopping without dropping a request);
• spreads its replicas over zones and nodes where the cluster has them.

The PodDisruptionBudget keeps one of the two replicas serving through a voluntary disruption. Every secret is a key of the ferrofed-secrets Secret, named by a _file key (Configuration): the signing key, and any credential an endpoint needs.

openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-384 -out signing-key.pem
kubectl create secret generic ferrofed-secrets --from-file=signing-key.pem
kubectl apply -k deploy/kubernetes/base

The image lane publishes it as ghcr.io/ferrohealth/ferrofed, tagged with the release version, its major.minor and latest, and attests the index and each platform manifest. Verify the image you pull:

gh attestation verify oci://ghcr.io/ferrohealth/ferrofed:X.Y.Z \
  --repo FerroHEALTH/FerroFED \
  --signer-workflow FerroHEALTH/FerroFED/.github/workflows/release-image.yml

The release lane is the only place the image is built: every compose file and manifest in the repository runs the published image, and none builds one.

The release binaries

Every release on the releases page carries ferrofed for x86_64 and aarch64 Linux, on glibc and on musl. FerroFED runs on Unix only (Supported platforms), and no release carries a Windows binary. Each tarball holds the binary, LICENSE, NOTICE and the README, and comes with its .sha256sum, a CycloneDX and an SPDX SBOM, and the Sigstore bundles of its provenance and SBOM attestations. Verify a tarball before you unpack it:

gh attestation verify ferrofed-vX.Y.Z-x86_64-unknown-linux-musl.tar.gz \
  --repo FerroHEALTH/FerroFED \
  --signer-workflow FerroHEALTH/FerroFED/.github/workflows/release-build.yml

The quickstart

Every release attaches the quickstart as one self-contained compose.yaml, with its .sha256sum: every file it needs is inlined as a Compose config, so it needs Docker Compose 2.23.1 or newer and no checkout. Download the latest release’s into an empty directory, write the gateway’s signing key beside it, and start it:

curl -fsSLO https://github.com/FerroHEALTH/FerroFED/releases/latest/download/compose.yaml
mkdir -p secrets
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-384 -out secrets/signing-key.pem
chmod 644 secrets/signing-key.pem
docker compose up --wait
curl http://127.0.0.1:8080/health

The gateway signs the caller’s identity onto every request to a node with that key, and refuses to start without it (What a node is told about the caller). It is a development key: the gateway runs as uid 65532 and Compose hands the file over as it is, so it is readable by every user.

Before the gateway starts, the one-shot files and ferroehr-init services copy the inlined files onto volumes the read-only services mount (Compose refuses an inlined config in a read-only service), the ferropix-seed service feeds FerroPIX the synthetic patients, and the ehr-seed service creates their EHRs at the four nodes, committing the Federation Tier reference implementation’s demonstration template and compositions, fetched at a pinned commit and refused unless each has its pinned SHA-256.

--wait returns once every service reports healthy. The gateway service runs ghcr.io/ferrohealth/ferrofed at the product version, the tag default compose.yaml holds, and FERROFED_VERSION selects another published version. Its healthcheck is the image’s own ferrofed healthcheck, which images before v0.0.7 do not carry, so with an older FERROFED_VERSION the gateway never turns healthy and --wait fails. No compose file builds the image; every one runs the published image.

The gateway and the four FerroEHR nodes send their installation reports to FerroPULSE by default, the gateway’s instance id kept on the ferrofed-state volume. FERROFED__USAGE_REPORT__ENABLED=false and FERROEHR__USAGE_REPORT__ENABLED=false, in your shell or .env, stop them (The usage report).

The four nodes run FerroEHR’s documented image, and each stamps its own system_id into every EHR and version it creates, the value the quickstart registry declares for it. They share one FerroEHR PostgreSQL container, and each connects to its own database there, ferroehr_a to ferroehr_d, owned by a login role of the same name. The image’s own init script creates the first; the node-databases config (a copy of docker/postgres/20-ferrofed-node-databases.sh), copied beside it, runs that script again for the other three. A schema per node would not do: FerroEHR creates fixed schema names in the database it connects to, so two nodes in one database would share their tables. Each database admits only its own node’s role, because the script revokes CONNECT from PUBLIC and grants it to that role, so node A’s role is refused on ferroehr_b. The nodes still share one server, its superuser and its cluster-wide group roles, so this is no production boundary.

Four instances of one product: EHRbase, the second product the topology first used, refuses a . in PARTY_REF.namespace, which openEHR BASE admits, so its EHRs could not carry the OID-style issuing namespace the synthetic patients use. Every database role’s password is its name followed by _example (ferroehr_a_example): development credentials that must not reach anything real. Every image is pinned by tag and digest, and docs/VERSIONS.md carries each pin.

The nodes run FerroEHR 5.7.0, which binds every person who signs in to a demographic PERSON, and reach no one under a Basic user. Each trusts the quickstart’s development issuer, the issuer service, as its issuer of persons (docker/quickstart/ferroehr.toml, inlined as the ferroehr-config config), and admits a token only when it names the node’s own API root as its audience. The gateway reaches a node for a caller only under the token the issuer exchanges the caller’s token for (RFC 8693), the node’s onward lane (The onward lane of a node): the caller as sub, the node as aud and the gateway’s client ferrofed as its client, which the node reads as a delegate acting for that person. A node creates the person’s PERSON at the first request, from the token’s name, when the token names the identity provider quickstart (FerroEHR’s just-in-time creation). Before each node starts, its ferroehr-definitions-* service loads the professional PERSON and ROLE templates the binding maps into the node’s database, FerroEHR’s own pair, vendored under docs/specs/ferroehr-definitions/. A node reads its issuer’s metadata over plain http only on loopback, so the nodes run in the issuer’s network namespace, like the gateway, each on its own port, which the issuer publishes. The ehr-seed service signs in as the person quickstart-seeder, through the nodes’ own client quickstart-seed.

Every published port binds the loopback interface. A published port is DNAT’d ahead of the host firewall’s own rules, so a port on 0.0.0.0 is reachable from the network even when the firewall says otherwise. Set FERROFED_BIND_HOST to the one address you mean, or put a reverse proxy in front.

docker/quickstart/registry.toml names the four nodes and docker/quickstart/ferrofed.toml configures the gateway with each node’s onward lane; Compose mounts both read-only. The configuration runs in the development profile and resolves every patient through FerroPIX, the FerroHEALTH patient index, as its PIXm Consumer: FerroFED keeps no patient index of its own (Identity resolution). Compose runs FerroPIX on its own PostgreSQL, configured by docker/quickstart/ferropix/ferropix.toml with a development sealing key. FerroPIX admits a machine only with a client credentials token of an issuer it trusts, so Compose runs a development token issuer beside it, ferropix-issuer (NAV’s mock-oauth2-server, which signs any client credentials request, NOT for anything real), and FerroPIX joins that issuer’s network namespace, since it reads an issuer’s metadata over plain http only on loopback. The gateway asks FerroPIX under the token [pixm.manager.endpoint.credentials.oauth] obtains there, and the one-shot ferropix-seed service, as the Patient Identity Source, feeds it the patients of docker/quickstart/patients.tsv over ITI-104 under its own token before the gateway starts.

docker compose down -v stops the stack and removes its volumes. A quickstart volume created before v0.0.8 holds the roles with their earlier passwords, which the nodes no longer send, so run docker compose down -v once before you bring that stack up again, then seed it anew.

The synthetic patients

The ehr-seed service reads the patients file, the patients FerroPIX was fed, and creates exactly those EHRs over each node’s ITS-REST API: PUT /v1/ehr/{ehr_id} with the patient on EHR_STATUS.subject, the reference implementation’s International Patient Summary template, and one of its demo compositions per EHR. A second up reports what is already there and adds nothing. Every identifier lies in the example arc urn:oid:2.999:

The first group of each ehr_id names its node and the last group its patient.

A federated query

Ask for the compositions of the patient every node knows. Every request carries an access token, which the development issuer signs for the quickstart’s client (The quickstart issuer):

TOKEN="$(curl -s http://127.0.0.1:8190/quickstart/token -d grant_type=client_credentials \
  -d client_id=quickstart-client -d client_secret=any | jq -r .access_token)"
curl -s http://127.0.0.1:8080/v1/query/aql \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d @- <<'EOF'
{"q": "SELECT c/uid/value FROM EHR e CONTAINS COMPOSITION c WHERE e/ehr_status/subject/external_ref/id/value = 'ffd-test-0001' AND e/ehr_status/subject/external_ref/namespace = 'urn:oid:2.999.1.1'"}
EOF

The gateway resolves the patient to the four ehr_ids, sends each node an ordinary AQL query scoped to its own ehr_id with no patient identifier in it, and answers 200 with one RESULT_SET: four rows, one composition from each node, its uid carrying the system_id of the node that created it. meta.federation.complete is true, and meta.federation.endpoints reports all four endpoints active with row_count 1.

A patient missing at some nodes

Send the same query for ffd-test-0002. The answer is 200 with two rows, from node B and node D. Node A and node C hold no EHR for the patient, so the gateway asks neither of them and reports both not-resolved, with no latency_ms, and meta.federation.complete is false. A not-resolved member is an answer and never fails the query (§11.3, N6, N37). For ffd-test-0003 only node C answers, and for ffd-test-0004, which no node holds, the answer is 200 with no rows and all four endpoints not-resolved.

A query directed at one node

Name the node in the query with the endpoint directive, and project its attributes beside the data (§8, §9.4):

curl -s http://127.0.0.1:8080/v1/query/aql \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d @- <<'EOF'
{"q": "SELECT p/id AS endpoint_id, p/system_id AS system_id, c/uid/value AS composition FROM ENDPOINT p [\"node-c-query\"] CONTAINS EHR e CONTAINS COMPOSITION c WHERE e/ehr_status/subject/external_ref/id/value = 'ffd-test-0001' AND e/ehr_status/subject/external_ref/namespace = 'urn:oid:2.999.1.1'"}
EOF

Only node C is asked. The one row reads node-c-query, node-c.quickstart.local and the composition’s uid; the other three endpoints are reported excluded, “not named by the request’s endpoint directive”, and meta.federation.complete is true, because an excluded member was never in scope. The header form selects the same way without changing the query: send the patient query above with -H 'openEHR-federation-endpoint: node-b-query' and only node B answers.

Memory footprint

Measured on 2026-10-03 with docker stats --no-stream, on an Apple silicon Mac running Docker Desktop with 8 CPUs and 7.65 GiB for its virtual machine, the images already pulled: