Configuration revisions

With config.ui = "managed", the gateway keeps its configuration as a history of revisions in a store, and you save, compare, apply and roll back revisions on the admin listener. With config.ui = "off", the default, none of this exists: the gateway is configured by its file and the FERROFED__ environment alone, opens no store, and serves no revision route. No specification governs this page: our own design.

What is built is the revision store and its API on the admin listener (#576), the members and stored queries the operator console manages, with their owners (#577, Console-managed resources), and the onward credentials and binding sections it manages, with write-only secrets, a confirmation after a change applies and a second operator’s approval before it applies (#578, Credentials and binding sections). Any revision exports as TOML, and TOML imports as a dry-run revision (#579, Export and import).

Turning it on

[metrics]
listen = "127.0.0.1:9464"     # the admin listener, which serves the revisions

[config]
ui = "managed"                # "off", the default, changes nothing
backend = "redb"              # or "postgres", for replicas that share a store
path = "/var/lib/ferrofed/config-revisions.redb"   # redb: the store file
# url_file = "/run/secrets/ferrofed/config-revisions-url"   # postgres: the connection string, a secret

Under off, GET /admin/config answers 404 configuration-unmanaged, so a client tells an unmanaged gateway from a wrong admin URL. Under off, any other key of [config] refuses to start, so a store you meant to use is never silently ignored. Under managed, [metrics] listen must be set, since the revision API is served on the admin listener.

What a revision holds

A revision is the whole configuration less its bootstrap core: the file and the environment as the gateway reads them, written as TOML text. The bootstrap core comes from the file alone and is never part of a revision:

So a revision can neither lock you out of the admin listener nor lower the security floor the file sets. A revision that sets a key of the core is refused.

A revision never holds a secret value. Under managed, every secret outside the core is written as its _file reference, such as password_file, and the gateway refuses to start, and refuses a revision, that writes one inline (password, client_identity, client_secret, assertion, stored_queries.url, or a URL that carries a password). The refusal names the key, never the value. A secret inside the core, such as an [[auth.basic.users]] password_hash, may stay inline, since the core is never stored. Comments are not kept: a revision is stored as TOML writes it.

Each revision records its number, the revision it was based on, where it came from (file, operator, rollback or import), who made it (the file, or the operator’s issuer and subject as their token names them), when, and why.

The file is a revision too

At start, and at each SIGHUP reload, the gateway reads its file and environment and checks them as config check does. When the part a revision holds differs from the newest revision the file gave, it is saved as a new revision by the file and runs. When it does not, the revision marked running keeps running, with the file’s bootstrap core beside it, so a revision you applied survives a restart until you change the file. Nothing ever writes back to the file.

Every revision is checked as config check checks the file

A revision is put beside the file’s bootstrap core, read as the configuration, resolved, and checked as ferrofed config check checks a file, the transport policy and the registry document included. One the check refuses is refused with 422 configuration-invalid, and the message carries the same error config check prints. Nothing is stored, and the running configuration stays.

The routes

Every route is on the admin listener and is admitted only for a caller whose token carries auth.product_scope, as the other admin actions are (Who the admin listener serves). Each write is recorded under ferrofed::security as an admin action, and with the revision number as config-revision-saved, config-revision-applied, config-rolled-back or config-revision-imported.

A save names the revision it is based on, which must be the newest (null for none), and an apply names the newest revision. When another operator, or another replica on a shared store, saved or applied first, the request is refused with 409 revision-stale and nothing changes, so two operators never overwrite each other. Read the revisions again and base the change on the newest one.

An apply goes through the same path as a SIGHUP reload: the sections a reload applies take effect at once, and a changed setting that takes a restart is listed in needs_restart, marked pending-restart, and takes effect at the next start. When the reload refuses the revision, the running configuration and the running mark stay. A rollback is one step: it restores the revision that ran before the running one, so a second rollback returns to where you were. With no earlier revision it is refused with 409 rollback-unavailable.

The other refusals are 400 body-invalid for a body that does not read, whose parser message is never echoed, and 404 revision-unknown (Gateway codes).

Console-managed resources

Each member of the federation, with its organisation and its endpoints, and each stored-query version has one owner, which GET /admin/config/resources names:

The console’s Members tab lists each member with its owner and state, and a console member is added on a form of its own (the console’s screens show every tab).

The Members tab in the dark theme

The member form in the dark theme

A member is owned whole. A console member is held in the revisions as one [[registry.members]] entry and joined to the registry document’s members, so a revision stays the whole configuration and nothing is written back to the file. The configuration file and the environment never set registry.members: a file that does refuses to start. Console members join a registry document; with a care services directory, which is the whole registry, they are refused.

[[registry.members]]
organisation = { id = "org-c", name = "Hospital C" }
node = { id = "node-c", system_id = "cdr-c.example.org", product = "FerroEHR", version = "4.3.3" }
endpoints = [
  { id = "node-c-pub", url = "https://cdr-c.example.org/openehr", connection_type = "openehr-rest-query" },
]

A member save or removal is not applied: preview the revision with GET /admin/config/revisions/{running}/diff/{new}, then apply it, and roll it back in one step like any revision. A console member’s state is running, pending (saved, not applied), removing (the newest revision removes it), or shadowed.

A change that names a member the file or the directory provides, by its node_id, its organisation id, its system_id, an endpoint_id or an endpoint’s base URL, is refused with 409 resource-file-owned, and the removal of a node_id no console member has with 404 resource-unknown. A stored-query version is admitted as a PUT to the ITS-REST definition API is: a held version is 409 stored-query-held, the reserved namespace 409 stored-query-reserved, and the read-only definition files 409 resource-file-owned. A version is never edited or removed, so a change is a new version.

At a SIGHUP reload whose file changed, the file’s new revision carries the console members over from the running revision, so a console member survives the reload. When the file later provides a member that shares an identity with a console member, the file’s member runs, and the console’s copy stays in the revision, reported as shadowed and logged by node_id; remove it, or give it identities of its own. When the console members are refused beside the file’s configuration, as when the file drops the registry document they join, the file’s revision is recorded without them, the refusal is logged by key, and the revision before it still holds them.

The operator console’s Configuration page shows all of this through the first replica it is configured with: the running and the newest revision with the newest one’s changes, the members and stored queries with their owners, and the history, in tabs of their own. A console member is saved or removed, the newest revision applied, the running one rolled back, and a stored-query version stored, each removal, apply, roll back and store confirmed in a step of its own (The operator console). Its member form writes the openehr-rest-query and hl7-fhir-rest endpoints, each a row of its own fields.

Credentials and binding sections

Each onward credential and each binding section is a resource owned whole, as a member is:

The Credentials and bindings tab in the dark theme

The binding sections a build lacks, by its binding-ihe and binding-nl features, are not offered. A resource the configuration file sets is the file’s: GET /admin/config/resources names its owner as file, and a change to it is refused with 409 resource-file-owned. Every other one a revision sets is the console’s, and the revision records it in its console list. Client authentication ([auth]), the transport policy the profile sets and the rest of the bootstrap core stay in the file alone, so the console cannot lock you out or lower the security floor.

settings holds the resource’s keys less its secrets, as the table’s own keys or as one TOML inline table: { oauth = { token_endpoint = "https://as.example.org/token", client_id = "ferrofed", client_auth = "private_key_jwt", private_key_file = "/run/secrets/ferrofed-client.pem", scope = "system/aql-*.s" } }. Settings that name a secret or its _file reference are refused with 400 body-invalid, naming the key. GET /admin/config/resources lists each credential and binding section under sections, with its owner, its state for a console one (running, pending or removing), its settings as one inline table for a console one, and its secrets, each a key and the reference it is read from; and the binding sections this gateway is built with under bindings.

At a SIGHUP reload whose file changed, the file’s revision carries the console’s credentials and binding sections over from the running revision. A credential or a section the file now sets is the file’s from then on: its console copy is left out of the file’s revision, and the gateway logs its table.

A reload applies the credentials, the binding sections whose reload applies ([pixm], [pdqm], [xcpd], [nl_gf]), the consent reporting of [federation.consent], and the localization policy of [federation.localization] while node_selection and the budgets are those the process started with; a change to XCPD’s audit takes a restart, as before.

Secrets are write-only

A revision never holds a secret value, only the _file reference to the file it is read from. A secret change names the secret’s dotted key inside the resource, such as password or oauth.client_secret, and one of:

• file: the absolute path of a file the gateway reads, which the revision references as it is;
• value: the secret itself, which the gateway writes to a new file in config.secrets_dir, readable and writable by the gateway’s user alone (mode 0600), and references. Each value gets a file of its own, never one that exists, so an earlier revision that references the previous file still reads it after a rollback. The gateway never logs the value, never echoes it, and never returns it in a revision, a diff, an export or an error;
• clear: the secret and its reference are removed.

A secret the change does not name keeps its reference. Without config.secrets_dir, a value is refused with 409 secret-store-unavailable and nothing is saved; a reference still works. A save the gateway refuses removes the files it wrote for it. The files of earlier values stay in the directory for the revisions that reference them; remove one once no revision you may roll back to references it.

[config]
ui = "managed"
path = "/var/lib/ferrofed/config-revisions.redb"
secrets_dir = "/var/lib/ferrofed/secrets"   # an existing directory, on a volume every replica shares

Commit-confirm

An operator’s apply of a revision that changes an onward credential or a binding section runs it awaiting a confirmation. GET /admin/config shows the deadline and the resources it changed under current.awaiting. Confirm it within config.confirm_ms, 120000 ms by default, once the gateway works with it:

Unless an operator confirms it by the deadline, the gateway rolls it back: it saves the revision that ran before as a new revision, made by the gateway with the reason revision N was not confirmed by its deadline, applies it, and records config-unconfirmed-rolled-back under ferrofed::security. Each replica reads the deadline from the store’s running mark with every config.poll_ms read, so the first replica past the deadline rolls it back and the others follow the rollback as any applied revision. A gateway that starts with a revision past its deadline rolls it back before it runs, and one that starts within the deadline keeps waiting for the confirmation. A SIGHUP whose file changed while a revision awaits its confirmation rolls that revision back first.

While a revision awaits its confirmation, another apply is refused with 409 confirmation-pending; confirm it or roll it back first. A confirmation of a revision that does not await one, or past its deadline, is refused with 409 confirmation-unavailable, and one from a caller with no verified token, such as a loopback peer under the development profile, with 403 operator-unverified. A rollback awaits no confirmation and takes no approval: it restores what ran before. GET /admin/config shows what the newest revision takes under review: the resources whose change awaits a confirmation once it applies (confirms).

Second-operator approval

config.approval names the resources a revision changes only once an operator other than its author approved it: a kind of the table above, or one resource (credentials.<endpoint id>, pixm, pdqm, xcpd, nl_gf, federation.localization, federation.consent).

[config]
approval = ["onward-credentials", "consent"]

An apply of a revision that changes such a resource is refused with 409 approval-required until an operator with another subject, at the same or another accepted issuer, approved it. Its author’s approval is refused with 403 approval-own-revision, the approval of a revision that needs none with 409 approval-not-required, and one from a caller with no verified token with 403 operator-unverified. The revision records its author and the store records each approval with its operator, and both are logged under ferrofed::security (config-revision-approved). GET /admin/config shows the newest revision’s review, with its author, its approval resources, its approvals and whether it is approved.

Each is read only under managed, refused under off, and takes a restart to change.

Export and import

Any revision exports as TOML, and TOML imports as a dry-run revision, so a configuration moves between the console and version control (#579).

The Import and export tab in the dark theme

The export is the revision’s TOML text under a comment header that names the revision, who made it, when, and where it came from. It holds the whole configuration a revision holds, the console’s members, credentials and binding sections beside the file’s, and every secret as its _file reference. A secret the console wrote to config.secrets_dir is exported as the path of the file it wrote. The bootstrap core is not part of a revision, so it is not exported, and the header says so: keep the core in your file.

An import takes a body of Content-Type: application/toml (or text/plain) of at most 1048576 bytes, and is checked as every save is. It is recorded with the origin import, by the operator who sent it, for the reason the query gives, and it is never applied by the import itself. Read the diff in the answer, then apply it with POST /admin/config/revisions/{id}/apply, which takes the approval and the confirmation it needs as any revision does. An import is refused, and nothing is saved, when:

Each import is recorded under ferrofed::security as config-revision-imported with its revision number. An export of a revision imports as the same configuration: its revision holds the same TOML, and its diff against the revision exported is empty.

On the Configuration page each revision of the history offers its download (Export revision N as TOML), which the console’s server reads from the first replica with your access token and answers as an attachment. The form Import a configuration as TOML takes the TOML pasted into it, saves it as a dry-run revision, and shows what it changes; apply it with the form Apply revision.

File-first and console-first

Two ways of working use the same revisions:

• File-first: the configuration lives in version control and reaches the gateway as its file. Each SIGHUP reload or start records the file as a revision by the file, and its credentials and binding sections are the file’s, read-only in the console. Use the console to read the history and the diffs, and to roll back.
• Console-first: the file holds the bootstrap core and little else, and operators change the configuration on the Configuration page or the admin routes. Export a revision to keep it in version control or review it as a pull request.

To move from console-first to file-first, export the running revision, copy its tables into your file beside the bootstrap core, and reload: the file then sets those credentials and binding sections and owns them. Console members are the exception: the file never sets registry.members, so move each one into the registry document. Secrets stay as the _file references the export names; keep those files in place.

To move a credential or a binding section from file-first to console-first, export the running revision, remove the section from the file and reload, then import the export: the file no longer provides the section, so the import sets it as the console’s, and you apply it as any revision. A configuration kept in version control comes in the same way, as an import you preview and apply, as long as it leaves every resource the file provides as the file sets it.

Several replicas

With backend = "postgres", the replicas share the history and the running mark, and a save or an apply on one replica is checked against what every other replica wrote.

Each replica reads the running mark every config.poll_ms, 5000 ms by default. When the mark names a revision the replica does not run, because another replica applied or rolled back a revision, or recorded its changed file at a SIGHUP, the replica applies that revision through the same reload path an apply takes. Within one poll_ms of an apply, every replica runs the revision. The watch reads the mark with an ordinary query; it does not use PostgreSQL LISTEN/NOTIFY.

A replica can refuse a revision another replica applied, for example when a file the revision names, such as the registry document, is missing or invalid on that replica. The replica then logs the refusal, counts it in ferrofed_config_revision_follows_total{result="refused"}, and keeps its running configuration. It does not try the same revision again. Fix the cause, then send it SIGHUP or apply a newer revision.

GET /admin/config on each replica says where that replica stands:

ferrofed_config_revision_in_step is 1 while the replica runs the current revision and 0 while it does not, and the alert FerroFEDConfigurationDrift fires after 10 minutes at 0 (Metrics).

Under config.ui = "off" nothing is read: no store, no watch.

With several replicas, config.secrets_dir must be one volume every replica mounts at the same path, since a revision names the file a secret value was written to and every replica reads it when it follows the revision. A replica that cannot read a file a revision references refuses that revision, keeps its running configuration and reports it as refused, as above.