Grafana and Loki¶
Loki 3.x accepts OTLP natively, so the sensor can post directly to it with nothing in between. No Docker, no collector, no sudo. Loki and Grafana both ship as standalone darwin-arm64 binaries.
The whole thing¶
# Loki needs allow_structured_metadata: true for OTLP resource attributes.
./loki -config.file=loki.yaml # :3100
./grafana server --homepath ./grafana # :3000, with a Loki datasource
python3 -m shadowclaw \
--otlp-endpoint http://127.0.0.1:3100/otlp \
--no-otlp-metrics
python3 integrations/defenseclaw/install_dashboard.py
Or let scripts/ShadowclawAI install and drive all three:
Observability binaries live under ~/.shadowclaw-o11y by default. On success it prints Grafana at http://127.0.0.1:3000, the dashboard URL, and log paths. See Quickstart.
--no-otlp-metrics is not optional here
Loki serves /otlp/v1/logs only and returns 404 for /otlp/v1/metrics. Leaving metrics on makes every poll report half its exports as failures. Send metrics to a collector or Prometheus instead.
The exporter appends /v1/logs, which lands on Loki's /otlp/v1/logs receiver. That is the configuration every LogQL query in the dashboard was verified against, and the installer auto-detects whichever Loki datasource exists, so no UID needs to match.
Two Loki settings that matter¶
allow_structured_metadata: true¶
Without it, OTLP resource attributes are dropped — including the authorship attributes and service.namespace. Set it in loki.yaml.
Raise the gRPC message limits¶
A ShadowClaw finding arrives with roughly fifty-five labels: the flat JSON body that | json parses, plus the OTLP attributes, which overlap. That is fine per line and expensive in aggregate. A few hundred findings in the window build a query result past Loki's 4 MB default gRPC ceiling, and the querier then fails the whole query with ResourceExhausted.
Grafana renders that failure identically to an empty one: No data
The real reason appears only in Loki's own log:
The tell is that it is load-triggered, not query-triggered. A panel works for weeks, nothing edits it, and then it goes blank on a busy day — or goes blank at now-6h while still working at now-15m.
Both halves of the failing hop need raising, since the querier's client to the query frontend has a 4 MB default of its own that the server settings do not cover:
server:
grpc_server_max_recv_msg_size: 104857600
grpc_server_max_send_msg_size: 104857600
frontend_worker:
grpc_client_config:
max_send_msg_size: 104857600
Loki reads its configuration once at startup, so restart it afterwards.
Writing queries that scale¶
Queries that unwrap are affected far more severely, because an unwrapped range aggregation keeps every label of every matching line as a separate series unless the aggregation names a grouping.
# Right. The grouping makes Loki reduce at the querier before shipping anything.
max_over_time({service_name="shadowclaw"} | json | unwrap risk_score [$__range]) by ()
That is both correct and around forty times faster here. An outer max by (provider) (...) does not substitute for it — the inner aggregation has already built and sent every series by then.
tests/test_install_dashboard.py fails if a panel unwraps without a grouping.
Query patterns¶
Group provider questions on provider.reached, never on findings — see why.
Empty without --esf under root.
Filter on event_name, always
All five log shapes carry process.*-style fields. A query that does not discriminate will silently mix a scored finding with the individual observations behind it, and double-count.
Why the body is flat¶
Every field is flat in the JSON body — no nested objects, no arrays, evidence included — because every panel parses with | json, and a nested attribute is neither a label nor a parseable field.
Multi-valued fields are comma-joined strings: signals, providers, endpoints, degraded_inputs. LogQL matches them with a regex, and an array would not survive the parse. It is also why counting providers correctly has to happen at emit time rather than in the query.
Metrics, if you want them¶
The gauges and counters still exist and carry the same values, with ktp.degraded as an attribute. Send them to a Collector or Prometheus, not to Loki:
# Logs to Loki, metrics to a collector — two sensors, or one collector fanning out
python3 -m shadowclaw --otlp-endpoint http://127.0.0.1:4318
See OTLP export for the shipped pipelines.