User guide delivery

The delivery path

Use this path for a central destination such as Splunk:

ShadowClaw -> local OpenTelemetry Collector -> HTTPS -> Splunk HEC
          -> durable local ledger

ShadowClaw sends OTLP/HTTP JSON. Splunk HEC uses a different protocol, so a Collector performs the conversion. Do not point otlp_endpoint at a HEC URL. Use one sensor and one Collector to fan out to multiple destinations; do not start a second sensor against the same ledger.

The sensor accepts plaintext HTTP only on loopback. For a remote OTLP receiver, use an HTTPS base URL with a trusted certificate. Put backend tokens, custom authentication and client-certificate configuration in the Collector. The sensor does not expose those Collector authentication settings.

Prepare Splunk and the Collector

1. Ask your Splunk administrator for an enabled HEC token permitted to write to a dedicated events index, its exact HTTPS HEC URL, and the appropriate CA trust chain. Use the endpoint supplied for your Splunk Enterprise or Cloud deployment; do not assume a Cloud port or hostname.
2. Install the OpenTelemetry Collector Contrib distribution for the host, following its official installation instructions. The splunk_hec exporter is in Contrib. The example below targets the repository's Collector baseline, 0.158.0; validate it with the version you deploy.
3. Download collector-splunk.yaml, or save the following complete configuration as that filename. This logs-only path carries all ShadowClaw log event types. Configure the sensor with emit_otlp_logs: true and emit_otlp_metrics: false for this example.

# ShadowClaw public operator example: logs to a Splunk events index.
# OpenTelemetry Collector Contrib 0.158.0. No detection rules or secrets.
receivers:
  otlp:
    protocols:
      http:
        endpoint: 127.0.0.1:4318

processors:
  batch:
    timeout: 5s

exporters:
  splunk_hec:
    endpoint: ${env:SPLUNK_HEC_ENDPOINT}
    token: ${env:SPLUNK_HEC_TOKEN}
    index: ${env:SPLUNK_HEC_INDEX}
    source: shadowclaw
    sourcetype: shadowclaw:otel
    tls:
      insecure_skip_verify: false
    sending_queue:
      enabled: true
      queue_size: 1000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [splunk_hec]

Set SPLUNK_HEC_ENDPOINT, SPLUNK_HEC_INDEX and SPLUNK_HEC_TOKEN in the Collector's environment using the platform commands below. The token stays out of the YAML file. A shell environment belongs to that shell: a Collector installed as a service needs the same values supplied through its service configuration or secret manager.

TLS verification remains enabled. If your HEC certificate uses a private CA, add ca_file: /absolute/path/to/ca.pem under the exporter's tls mapping (use the real local path on Windows). A certificate error is a trust or hostname problem to fix, not a reason to set insecure_skip_verify: true.

See the Contrib HEC exporter reference for supported settings.

Keep Grafana while also sending to Splunk

If the local Loki stack is running, add this second exporter to the same Collector file:

exporters:
  # Keep the existing splunk_hec exporter here too.
  otlp_http/loki:
    endpoint: http://127.0.0.1:3100/otlp

Then change service.pipelines.logs.exporters to [splunk_hec, otlp_http/loki]. Merge into the existing exporters mapping; do not create two YAML keys named exporters. Run the Collector on the endpoint host for these loopback addresses to work. The sensor now points only to http://127.0.0.1:4318, and the Collector sends the same logs to both backends. If that port is already owned by DefenseClaw or another Collector, extend the existing pipeline or choose another port and update the sensor to match.

For metrics, add a metrics pipeline and an appropriate metrics backend before setting emit_otlp_metrics back to true. In Splunk, use a separately configured metrics index/exporter. Do not send metrics to the events index in this example or to Loki.

Verify receipt in Splunk

First search the events index you configured, over the last 15 minutes:

index=YOUR_EVENTS_INDEX sourcetype="shadowclaw:otel"
| spath
| table _time host event_name process_name pid providers severity

The example exporter sends the sensor's flat JSON body as the event. Find a fresh shadowclaw.ktp.risk_factors record to check the log route even on an idle host. Then perform the approved Plane B workload again and narrow the search:

index=YOUR_EVENTS_INDEX sourcetype="shadowclaw:otel"
| spath
| search event_name="shadowclaw.finding.recorded"
| table _time host process_name pid providers endpoints severity

Match the host, process, provider and time against the local ledger. Use event_name="shadowclaw.agent.activity" for tactics and event_name="shadowclaw.provider.reached" for provider observations. Keep those event types separate when counting detections. If an existing organization pipeline wraps the body, inspect a raw event and adapt the spath input to that wrapper before applying these searches.

The local ledger is written before export. Failed delivery does not automatically replay old ledger entries when connectivity returns. The example Collector has a bounded in-memory queue and retries; it is not durable across Collector restarts. For production, configure persistent Collector storage and an independently tested recovery/backfill procedure. A fresh matching event at the destination is the acceptance criterion, not just a reachable port or a running Collector.