Troubleshooting

This page starts from what you see: a status a client reports, a line in the log, a metric that moves, a gateway that does not start. Each entry names the cause, how to confirm it from what the gateway already emits, and the fix, with a link to the page that explains the setting. No specification governs troubleshooting: our own design.

The page has four parts. Where to look says what each signal carries. Symptoms covers the failures an operator meets most. The gateway does not start covers every refusal of config check and serve. Error codes has one row for every code the gateway answers with, held to the gateway’s own table by a test.

Where to look

• The answer. Every error the gateway reports itself is the ITS-REST Error body with a stable code and the request_id (Errors and status codes). A federated query that fails under all-or-nothing completeness answers a result set instead, whose meta.federation.endpoints[] names each node with its status and its error.
• The request line. Every request writes one line, request, with the method, the route template as route, the status, latency_ms and the gateway’s request_id: at INFO for a success, WARN for a 4xx and ERROR for a 5xx (What the log records). Every other line about the same request carries the same request_id.
• The log target. With telemetry.format = "json", each line carries a target; the pretty format leaves it out, so search by the message there. Four targets are named on purpose: ferrofed::security (client authentication, identifier hygiene and the admin listener, never quieted by telemetry.filter), ferrofed::integrity (integrity incidents), ferrofed::audit (audit records written to the log) and opentelemetry-otlp (a failed trace export). Every other line’s target is the Rust module that wrote it, such as ferrofed_server::reload, ferrofed_server::access or ferrofed_engine::dispatch::reported, as the entries below name it (Tracing).
• The metrics on the admin listener: ferrofed_node_requests_total by endpoint and outcome, ferrofed_security_events_total by event and reason, ferrofed_overload_refusals_total by limit, the resolver, localizer, audit spool and reload series (Metrics).
• The dependencies. GET {base}/operator/dependencies names the state the gateway last saw of each member endpoint and each identity service: up, failing, down or unknown (Health probes).

Symptoms

Every query answers 424

Symptom. Federated queries answer 424 with a result set and no rows.

Cause. Under all-or-nothing completeness, one node that answered with an error fails the whole query, and so does a member the cross-reference service could not answer for (§11.4, N37).

Confirm. Read meta.federation.endpoints[] of one answer. A member with status node-error names the node’s own status and message in error, such as the node answered 500 Internal Server Error: …. A member with not-resolved and an error is one the resolver failed for; the resolver counts it as ferrofed_resolver_requests_total{outcome="unavailable"}, and GET {base}/operator/dependencies reports resolver failing or down. The failing node counts under ferrofed_node_requests_total{outcome="node-error"}.

Fix. Fix the node the error names, or the cross-reference service (Identity resolution). To serve the members that answer while one is down, a client sends openEHR-federation-completeness: partial, which federation.best_effort must offer (Completeness).

Every node is offline or time-out

Symptom. Queries answer 504, and every endpoint is offline or time-out.

Cause. The gateway cannot reach the nodes (a wrong URL in the registry document, DNS, a firewall, TLS), or they do not answer inside federation.per_node_timeout_ms, or the gateway already has federation.max_in_flight_per_node requests open at each of them.

Confirm. The endpoint’s error reads the node could not be reached: … with the HTTP client’s reason, or no answer before the deadline. The dependency state is down, and the node counts under ferrofed_node_requests_total{outcome="offline"} or {outcome="time-out"}. A request that waited for a slot is counted by ferrofed_overload_refusals_total{limit="node-in-flight", endpoint="…"}. Compare ferrofed_node_request_duration_seconds with the timeout.

Fix. Correct the endpoint url in the registry document and reload (The registry document), open the path from the gateway to the node, raise per_node_timeout_ms above the node’s 99th percentile (Timeouts), or size max_in_flight_per_node to what the node can serve (The per-member cap).

Nodes answer 401

Symptom. Inside a query, endpoints are node-error with error the node answered 401 Unauthorized. On a request routed to one node, the client gets 424 node-refused.

Cause. The node does not accept the gateway’s onward credential: the [credentials."<endpoint>"] section is missing or wrong, the node does not trust the gateway’s signing key, or the token endpoint refused the gateway.

Confirm. When the gateway could not get a token at all, the endpoint’s error is no onward credential could be obtained, so nothing was sent, followed by the token endpoint’s RFC 6749 code when it sent one, and the gateway logs no onward credential could be obtained, so nothing was sent to the node at WARN under ferrofed_engine::dispatch::reported, with the endpoint and the token endpoint’s own account in error. A 401 from the node itself leaves the dependency state up, because the node answered.

Fix. Set or correct the endpoint’s credentials section, its scope, its token_endpoint and its client_id (Onward credentials). The token endpoint verifies the gateway’s client assertion, and a node its signed caller token, against the gateway’s JWK Set at {base}/.well-known/jwks.json; check that each can reach it and holds the current key (Signing keys and the JWK Set).

Every caller is refused 401

Symptom. Clients get 401 unauthenticated, 403 scope-insufficient or 403 purpose-of-use-required, and no node is asked.

Cause. The token is not one the [auth] configuration accepts.

Confirm. Each refusal logs a request was refused at client authentication at WARN under ferrofed::security, with event caller-refused, the reason and the status, and counts ferrofed_security_events_total{event="caller-refused", reason="…"}. The WWW-Authenticate challenge of the answer names the same reason.

Fix. By reason: issuer is an iss no [[auth.issuer]] lists; audience is an aud other than auth.audience; expired and not-yet-valid past auth.clock_skew_s point at a clock out of step; key is a kid the issuer does not publish; algorithm, type and malformed are tokens that are no RFC 9068 access tokens; scope and purpose-of-use are grants the token lacks (Client authentication).

Every caller is refused 503 authentication-unavailable

Symptom. Every request of one issuer’s callers answers 503 authentication-unavailable.

Cause. The gateway cannot fetch or read that issuer’s key set, or its introspection endpoint does not answer.

Confirm. an issuer's key set could not be had, so its tokens cannot be verified (event key-set-unavailable) or an issuer's introspection endpoint did not answer, so its tokens cannot be verified (event introspection-unavailable) at WARN under ferrofed::security, naming the issuer and the fetch error.

Fix. Make jwks_uri or introspection_endpoint reachable from the gateway, or raise auth.fetch_timeout_ms (The key set).

Requests are refused 503 overloaded or 429 rate-limited

Symptom. Clients get 503 overloaded or 429 rate-limited with a Retry-After.

Cause. The gateway serves server.max_concurrent_requests requests at once, or one caller sent more than [server.caller_rate] admits.

Confirm. a request was refused: the gateway serves as many requests as it takes at once or a request was refused: its caller sent more requests than one caller may, at WARN under ferrofed_server::overload, and ferrofed_overload_refusals_total{limit="concurrency"} or {limit="caller-rate"}.

Fix. Size the limits to your members and callers (Overload protection).

Requests are refused 503 access-unrecorded

Symptom. Queries and routed reads answer 503 access-unrecorded.

Cause. The gateway could not store the record of the access, so it withholds the answer.

Confirm. the access could not be recorded, so its answer is withheld at ERROR under ferrofed_server::access, with the cause in error. The spool refused the record when an audit message was refused: … naming the spool’s bounds appears at WARN under ihe_iti::atna::spool and ferrofed_audit_refused_total rises. Two other lines under the same target point at a defect to report with the request_id: an access was served with no verified caller, so its answer is withheld and an answer of a patient-data operation carried no access, so it is withheld.

Fix. Free the spool by bringing the audit repository back (the next entry), give the spool room (spool_max_events, spool_max_bytes) or a faster disk (spool_write_timeout_ms) (The spool and the failure policy). A write may have reached the node before the record failed, so a client reads before writing again (Failing closed).

The audit spool grows

Symptom. ferrofed_audit_spool_events and ferrofed_audit_spool_bytes rise, and GET {base}/operator/dependencies reports audit_repository or audit_feed degraded.

Cause. The audit repository is not taking the records, so they wait in the spool, or it refused some, which wait in the quarantine.

Confirm. Under ihe_iti::atna::forwarder: the audit repository could not be reached; its messages stay spooled or an audit message could not be delivered; the connection is dropped and the message stays spooled for an ITI-20 syslog repository, and an AuditEvent could not be delivered; it stays spooled for a FHIR Feed repository, each at WARN with ferrofed_audit_retries_total rising. the audit repository refused an AuditEvent; it is quarantined at ERROR, with its sequence and HTTP status, raises ferrofed_audit_quarantined.

Fix. Bring the repository back; the spool drains in order once it answers. Read why a quarantined record was refused, fix that, and remove it (The audit spool). A spool that fills turns every access into 503 access-unrecorded.

A reload is refused

Symptom. After SIGHUP, the gateway still serves the old registry.

Cause. The new configuration does not load, or changes something only a restart applies.

Confirm. registry reload refused, the running registry stays; `ferrofed config check` names the fault at ERROR under ferrofed_server::reload, with the failure class, and ferrofed_registry_reloads_total{result="refused"}. A changed setting a reload does not apply logs changed settings take effect only on a restart; the running values stay at WARN, naming the keys in settings. A listener certificate that does not read logs listener certificate reload refused, the running certificate stays at ERROR.

Fix. Run ferrofed config check against the same file, which prints the key at fault, correct it, and send SIGHUP again (Reloading the registry).

A request is refused 409 ehr-id-collision

Symptom. Reads and writes that name one ehr_id answer 409 ehr-id-collision, and the message lists the endpoints that claim it.

Cause. Two members or more claim the same ehr_id, a defect in the federation (§12.5.2, N42).

Confirm. integrity incident: an ehr_id is claimed by more than one member, and the request was refused at ERROR under ferrofed::integrity, with kind EhrIdCollision, the detection and the claimants, and ferrofed_integrity_incidents_total{kind="EhrIdCollision"}.

Fix. Have the member that issued or adopted the ehr_id in error fix it, then restart the gateway. Until then a client reaches one member by naming it in openEHR-federation-endpoint (Integrity incidents).

The operator console shows a refusal

Symptom. A view of the operator console shows a status and a code where its rows should be.

Cause. The gateway refused the operator’s token. A 403 scope-insufficient is a token without the issuer’s operator_scope, which the /integrity and /stored-queries views need; a 401 is a token that does not verify or a session that ended.

Confirm. The gateway logs the refusal as caller-refused under ferrofed::security, with reason scope or the 401’s reason.

Fix. Set operator_scope on the operator’s [[auth.issuer]] and have the provider grant it (The operator surface). On a 401, sign in again. An /integrity view that lists no incident is a gateway that has raised none since it started (The operator console), and no view is ever silently empty.

Readiness answers 503

Symptom. GET {base}/health/readiness answers 503, and a load balancer or the container runtime takes the gateway out.

Cause. The body’s phase is booting before the gateway serves, and draining from the moment SIGTERM or SIGINT arrives. No member node and no identity service gates readiness.

Confirm. Read the body’s phase. ferrofed healthcheck prints ferrofed: <address>: <outcome> and exits 1; a configuration that does not load prints ferrofed: not ready: ….

Fix. booting lasts until the gateway has bound its listener and started its bindings. A gateway that cannot start exits instead, with a cannot start or cannot serve line (The gateway does not start). A draining gateway was asked to stop; size the grace period (Stopping without dropping a request).

The admin listener answers 401

Symptom. Prometheus reports the scrape target down with 401, or the stored-query distribution answers 401 or 403.

Cause. metrics.scrape_token is set and the scrape does not carry it, or a write action’s caller has no token that verifies or no operator_scope.

Confirm. a request to the admin listener was refused at its authentication at WARN under ferrofed::security, with event scrape-refused and reason missing or scrape-token, or event admin-write-refused and the client-authentication reason.

Fix. Give the scrape job the token in authorization, or give the operator a token carrying the issuer’s operator_scope (Who the admin listener serves).

Every request under {base}/v1/ answers 501

Symptom. Every ITS-REST path answers 501 not-implemented.

Cause. No registry is configured, so the gateway federates nothing.

Confirm. The startup banner’s Registry line reads none, and OPTIONS {base}/ answers 501 too.

Fix. Set registry.document, or [registry.mcsd] (The registry).

Every query answers 404 no-destination

Symptom. Queries answer 404 no-destination and no node is asked.

Cause. No registry member is in scope: every endpoint is suspended or excluded, or the targeting names organisations that manage no endpoint.

Confirm. OPTIONS {base}/ lists each endpoint with its membership standing.

Fix. Correct the membership in the registry document and reload (The registry document).

The gateway does not start

config check and serve refuse the same configurations. A refusal found while reading the configuration prints ferrofed: cannot start: on stderr, followed by the fault and each cause after it, separated by : , and exits 78. A refusal found once the log has started is logged instead, as cannot start at ERROR under ferrofed_server::command with the chain in error, and also exits 78. A listener that cannot be bound logs cannot serve with binding <address> or binding metrics.listen <address> and exits 1. No refusal quotes a secret or a patient identifier; each names the key. Run ferrofed config check --config <file> to see the fault without binding a socket (Running it).

Error codes

One row for every code the gateway answers on its own behalf. The query refusals (not-aql, unreducible and the rest) are all 400: each logs a federated query was refused before dispatch at WARN under ferrofed::security, event aql-refused, with its kind and byte range, and the client rewrites the query (Query refusals). A row’s “Confirm” names what the gateway writes beside the request line, which every row has.