Metrics

The gateway counts what it already observes: the requests its clients send it, by route and status, and how long they take; the requests it sends to each member node, and to its resolver, localizer and demographics service, and how long they take; the integrity incidents it raises; the security events it logs; the requests its limits refuse; and the registry reloads. One OpenTelemetry meter provider holds the counts. A Prometheus server scrapes them from GET /metrics on an admin listener of their own, and the gateway can also push them to an OpenTelemetry collector over OTLP. Both surfaces read the same provider, so a metric never exists on one and not the other. Both are off by default.

The dependency report, GET {base}/operator/dependencies, stays as it is: it reports the last state the gateway observed of each member, and the metrics count every request over time.

Turning it on

[metrics]
listen = "127.0.0.1:9464"     # the admin listener; unset, nothing listens
allow_remote = false          # true lets listen name a non-loopback address
scrape_token_file = "/run/secrets/ferrofed/metrics-scrape-token"   # the bearer token a scrape must carry; unset, the scrape is open
otlp_endpoint = "http://127.0.0.1:4317"   # an OTLP gRPC collector; unset, nothing is pushed

The admin listener serves GET /metrics and the operator’s stored-query distribution (POST /admin/stored-queries/{name}/{version}/distribute), answers every other path 404, and never sits under the base path. It is never the gateway’s own listener, so no client of the federation reaches it.

Who the admin listener serves

A write action is verified by the same client authentication as the gateway’s own listener, against the issuers of [auth] and in its mode, and needs no purpose of use. An issuer that names no operator_scope admits no operator, so with none the write actions are refused to every caller outside the development profile. The scrape token admits the scrape alone, never a write action. Every refusal is counted under ferrofed_security_events_total as admin-write-refused or scrape-refused, and logged under the target ferrofed::security with its reason, never the credential.

Every admitted write action is recorded under ferrofed::security in two lines. The first, event admin-write-admitted, is written before the action has any effect and is counted under the same name. The second, event admin-write-finished, is written when the action ends, with its outcome: the status it was answered, or abandoned when it never answered because its client went away or its handler failed. Each line carries the operator’s issuer and subject, the method and route template as action, and the time as at (RFC 3339), and nothing else: no token, header, path value, body or stored query. A loopback peer the development profile admitted without a credential names no issuer or subject. Neither the issuer nor the subject is ever a metric label.

The subject is the one the issuer put in the operator’s token, and it may name a person. The gateway keeps it as issued, because a keyed hash under a key that changes at every start could not attribute an action after a restart. Keep these lines as long as you keep your other security records, and restrict who reads them. No [telemetry] filter quiets the ferrofed::security target below info, so the records are written whatever the filter says.

scrape_token, or scrape_token_file with the token in a file read at start, sets the scrape token; give one or the other. Prometheus sends it with authorization in the scrape job:

scrape_configs:
  - job_name: ferrofed
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/secrets/ferrofed-scrape-token
    static_configs:
      - targets: ["gateway.example.org:9464"]

Mutual TLS authenticates the scrape as well: with [metrics.tls] and its client_ca_file, every client of the listener must present a certificate that CA signed (TLS on the listeners). Give the scraper such a certificate with tls_config in its job.

config check says who a listener that is not on a loopback address answers. serve and config check refuse:

• a listen address that is not a loopback address, such as 0.0.0.0:9464, unless allow_remote = true is set. Set it only when the address is reachable from your scraper and your operators and nothing else, for example inside a pod network a network policy closes; the policy is defence in depth, and holds only where the cluster’s network plugin enforces it;
• outside profile = "development", a listen address that is not a loopback address with nothing to authenticate the scrape: neither scrape_token nor [metrics.tls] client_ca_file;
• scrape_token and scrape_token_file both set, and a scrape_token_file that cannot be read or holds nothing;
• a listen address equal to server.listen;
• an otlp_endpoint that is not an http:// URL. The push speaks gRPC without TLS, so run the collector beside the gateway, on the same host or in the same pod, and let it forward over TLS;
• an otlp_endpoint with a user name or a password in it, outside profile = "development", since that credential would travel in cleartext (What must travel encrypted).

The push sends every 60 seconds; the standard OTEL_METRIC_EXPORT_INTERVAL environment variable, in milliseconds, changes the interval. A push that fails is logged at WARN under the target opentelemetry-otlp, and the counts stay readable on /metrics. On SIGTERM the gateway pushes once more after the drain.

[metrics] takes effect on a restart only: a reload that changes it logs the key as needing a restart, like [server].

The metrics

The gateway names its instruments the OpenTelemetry way, and the Prometheus exporter renders each name with _ for ., _total after a counter, and the unit after a histogram.

Every label value comes from a closed set or from your registry document, never from a request, so no patient identifier, query text, header value or path reaches the surface:

The incident, reload and security event counters, and the concurrency and caller-rate refusals, show every label value at 0 from the start, so an alert on their increase works from the first scrape. A node request series appears with the first request to that endpoint, and an endpoint a reload removes keeps its series until a restart.

Which calls the node request series cover

Six kinds of call send a request to a member. Each request one of them sends is counted once in ferrofed_node_requests_total and timed once in ferrofed_node_request_duration_seconds, under the same endpoint, so the histogram’s _count equals the counter summed over outcome. A request that never left the gateway is in neither: a member that was not-resolved, excluded or not-localized, and a request the gateway could not send, for want of a client or a credential, or because the identifier-hygiene gate withheld it. §11.1 has no status for a request the gateway could not send, so a member record in meta.federation still reports that member offline; the series do not count it. A request whose deadline passed before it left is not counted either, in any of the six calls, because the node was never asked. The client still sees the budget run out: a time-out in the member record, or a 504 (node-timeout) for a routed request or a probe. A call the gateway fails with a 500 because one of its own tasks panicked counts no member at all, since a defect in the gateway is never a node’s time-out.

The per-member record a federated query, a template upload, a distribution and a drift check write is the one the client reads in meta.federation, and the counter follows it exactly. A routed request and a probe have no such record, so their outcome is read from the node’s answer by the same rules. Each outcome therefore covers these calls:

Dashboard and alert rules

Every release attaches a Grafana dashboard, ferrofed-dashboard.json, and a Prometheus rule file, ferrofed-alerts.yaml; both are in the repository under deploy/observability/. Import the dashboard and pick your Prometheus as its datasource: it charts the requests by status class and route, the members by outcome and answer time, the resolver, localizer and demographics calls, the security events, the limits’ refusals, the integrity incidents and the audit spool. Load the rule file through rule_files in prometheus.yml, or wrap its groups in a PrometheusRule for the Prometheus Operator. Its alerts carry severity: page or severity: ticket:

The thresholds are a starting point; tune them to your traffic. Add an alert on your scrape job’s up as well, since a gateway that does not answer the scrape counts nothing. scripts/checks/observability.sh holds both files to the metrics the gateway exports, and runs promtool check rules.

The Kubernetes example (deploy/kubernetes/) serves the admin listener on port 9464 of each pod, with the scrape token in the metrics-scrape-token key of the ferrofed-secrets Secret, annotates the pods for a Prometheus that reads prometheus.io/scrape, and opens the port to the Prometheus pods of the monitoring namespace alone with a network policy. Give the Prometheus job the same token with authorization.credentials_file. The gateway authenticates the scrape and every write action by itself, so the policy is defence in depth.

The gateway does not call a webhook. An incident is counted, and its log line under ferrofed::integrity carries the routing ids you act on; route the alert from your Prometheus or your collector.