Splunk¶
config/otel-collector.yaml carries a commented splunk_hec exporter. The token is read from the environment, never the file.
export SPLUNK_HEC_TOKEN='...'
export SPLUNK_HEC_URL='https://splunk.example.com:8088/services/collector'
Three data shapes arrive, and they are not interchangeable¶
| Shape | Index | Contents |
|---|---|---|
otel_metrics |
metrics | hostmetrics process and network metrics, plus the sensor's shadowclaw.* metrics. |
otel_logs |
logs | shadowclaw.finding.recorded — one event per finding, already correlated and scored, with signals and endpoints attached. |
otel_logs |
logs | shadowclaw.agent.activity — one event per observed tactic on the host plane. |
Filter on shadowclaw.event_name or your counts will be wrong
Both log shapes carry process.* fields. A search that does not discriminate will silently mix a scored finding with the individual observations behind it and double-count. Every shipped search states which shape it reads.
Alert on findings, not on metrics¶
Correlation happens at the endpoint, where per-process attribution exists. Redoing it in SPL over host-wide metrics is strictly weaker.
That goes double for the host plane: a kill chain spans several pids and several minutes, and the sensor has the process ancestry to join them where Splunk does not.
For automated response, the finding carries process.pid, host.name and process.owner — everything an EDR isolate-or-terminate API call needs.
The fourteen searches¶
splunk/shadowclaw-detection.spl
| # | Search | Purpose |
|---|---|---|
| 1 | Primary alert | Correlated shadow-AI findings. Run every 5 minutes over the last 5 minutes, trigger if count > 0. |
| 2 | Gateway bypass | The control failure that matters most — a process that knew the approved path and went around it. |
| 3 | Unknown providers | Coverage review for vendors not in the catalog. |
| 4 | Inference heartbeat | From raw hostmetrics, for hosts where the sensor is not yet deployed. |
| 5 | Local open-weight model servers | Ollama, llama.cpp, LM Studio, vLLM and friends. |
| 6 | Fleet AI inventory | Who is using what, sanctioned or not. |
| 7 | Sensor health | Do not let the control go dark silently. |
| 8 | New-provider detection | First time the fleet sees an endpoint. |
These return nothing without --esf under root, which is expected rather than broken.
| # | Search | Purpose |
|---|---|---|
| 9 | Agent kill chain | The highest-priority host-plane alert. |
| 10 | Incident timeline | Reconstruct one agent session, tactic by tactic. |
| 11 | Credential access by an agent | Lead indicator. |
| 12 | Agent-config persistence | The category an EDR does not have. |
| 13 | Host-plane tactic rates | Fleet trend, from the count connector. |
| 14 | Endpoint Security went dark | The host plane failing quietly. |
Search 14 deserves special attention¶
shadowclaw.esf.running is exported on every poll and reads zero when the source is down. If it were emitted only while healthy, a subscription that died would leave no trace at all — and absence is the hardest thing to alert on.
An empty host-plane row is not a clean host. This search is what tells the two apart.
Collector configuration¶
Splunk needs a Collector in front of it — the sensor speaks OTLP/HTTP, not HEC.
For the host plane, use the second, complete config rather than editing the first:
That split exists on the same reasoning as everywhere else: host-plane detection is opt-in, and turning it on must not be able to regress a working shadow-AI deployment. See OTLP export.
The agentic pipeline also derives technique URLs and ATT&CK tactic names in a transform processor, and runs a count connector so fleet-wide tactic rates (search 13) are answerable from the metrics store rather than a log scan.
Enrichment happens in the Collector, not the sensor
ATT&CK mappings are the part most likely to need correcting, and correcting them in the Collector does not mean shipping a new sensor to every endpoint.
Validate before you deploy¶
Validates config/otel-collector.yaml against the installed Collector binary, so a pipeline typo is caught before a deployment quietly drops telemetry.
Redaction is upstream of this¶
Command lines reach Splunk already scrubbed — provider key prefixes, KEY=value shapes, bearer headers, and JWTs are removed by shadowclaw/redact.py before anything is written or exported.
A monitoring control that mirrors credentials into a log index is worse than no control. See Security notes.