Hephaisto demo Docs Site GitHub
DEMO DATA — replayed from cassette c10, recorded against a real k3s cluster. Investigated by gpt-oss:120b on 2026-09-01 against a tool trace from gemini-3.7-flash, and graded NoFinding against the answer key. Timestamps are the original recording times. NOTE: this replay was structurally unsound — the recorded tool trace did not cover what the model asked for, so the verdict reflects the recording rather than the reasoning.

← all ten investigations

HighErrorRate on faulty-service ()

! Escalated Warning HighErrorRate cassette c10
target
/Service/faulty-service
workload
/Service/faulty-service
node
—
opened
2026-08-29 13:54:15
investigated
1m 38s

expected root cause — the answer key

The service returns 500s for about 15% of requests with an elevated p95 latency, while Kubernetes reports it perfectly healthy - the pod stays Ready and no event is emitted.

This is never shown to the model. It is what the grader compared the diagnosis against, and it is on this page because a demo that showed only the answer would be asking you to take the grading on trust.

signals 1

reasonmessagefirst seenn
ServiceHighErrorRate 10.0% of spans from faulty-service in namespace hephaisto-chaos have status STATUS_CODE_ERROR over the last 5 minutes (SLO: <5%). These are derived from traces, so the failing requests themselves are retrievable: TraceQL `{ resource.service.name = "faulty-service" && status = error }` returns the exact spans this number was computed from. Start there, not with a log search. 2026-08-29 13:54:05 1

state transitions

  1. 2026-08-29 13:54:15 Detected DEMO DATA — replayed from cassette c10, recorded against a real k3s cluster. Investigated by gpt-oss:120b on 2026-09-01 against a tool trace from gemini-3.7-flash, and graded NoFinding against the answer key. Timestamps are the original recording times. NOTE: this replay was structurally unsound — the recorded tool trace did not cover what the model asked for, so the verdict reflects the recording rather than the reasoning.
  2. 2026-09-01 21:47:53 Detected → Investigating Investigating with gpt-oss:120b.
  3. 2026-09-01 21:49:31 Investigating → Escalated Diagnosed, and no action was proposed. Escalated to a human.

investigation

model
gpt-oss:120b
steps
22
tool calls
22
tokens
206.7k in / 4.2k out
cost
$0.0069
confidence
0.00
ended
Concluded

trace

  1. 1 llm model turn internal 14.5s · 7.9k tok · $0.0003

    model output — its reasoning, and the tools it asked for

    -> calls list_datasources
  2. 2 tool list_datasources grafana-mcp 0ms · 744 B

    arguments

    {"limit":100,"offset":0,"type":""}

    result digest — what the model actually saw

    [step 01a05ef1-9c8b-7146-ae5b-8c8a9d6deb26] list_datasources
log digest: 4 lines, 744 bytes

-- last 4 lines --
{
  "$type": "text",
  "text": "{\u0022datasources\u0022:[{\u0022id\u0022:4,\u0022uid\u0022:\u0022alertmanager\u0022,\u0022name\u0022:\u0022Alertmanager\u0022,\u0022type\u0022:\u0022alertmanager\u0022,\u0022isDefault\u0022:false},{\u0022id\u0022:2,\u0022uid\u0022:\u0022loki\u0022,\u0022name\u0022:\u0022Loki\u0022,\u0022type\u0022:\u0022loki\u0022,\u0022isDefault\u0022:false},{\u0022id\u0022:1,\u0022uid\u0022:\u0022prometheus\u0022,\u0022name\u0022:\u0022Prometheus\u0022,\u0022type\u0022:\u0022prometheus\u0022,\u0022isDefault\u0022:true},{\u0022id\u0022:3,\u0022uid\u0022:\u0022tempo\u0022,\u0022name\u0022:\u0022Tempo\u0022,\u0022type\u0022:\u0022tempo\u0022,\u0022isDefault\u0022:false}],\u0022total\u0022:4,\u0022hasMore\u0022:false}"
}
  3. 3 llm model turn internal 2.4s · 8.3k tok · $0.0003

    model output — its reasoning, and the tools it asked for

    -> calls list_prometheus_metric_names
  4. 4 tool list_prometheus_metric_names grafana-mcp 0ms · 175 B

    arguments

    {"datasourceUid":"prometheus","regex":".*traces_spanmetrics_calls_total.*","limit":10,"page":1}

    result digest — what the model actually saw

    [step 01a05ef1-a5ba-7bcc-a86c-d1781bebe8f6] list_prometheus_metric_names
log digest: 1 lines, 175 bytes

-- last 1 lines --
No output for this call was recorded in cassette 'c10'. This scenario was captured from a different sequence of tool calls; treat it as unknown rather than as an empty result.
  5. 5 llm model turn internal 3.6s · 8.5k tok · $0.0003

    model output — its reasoning, and the tools it asked for

    -> calls query_prometheus
  6. 6 tool query_prometheus grafana-mcp 0ms · 175 B

    arguments

    {"datasourceUid":"prometheus","expr":"rate(traces_spanmetrics_calls_total{service=\u0022faulty-service\u0022,status_code=\u0022STATUS_CODE_ERROR\u0022}[5m])","queryType":"instant","endTime":"now"}

    result digest — what the model actually saw

    [step 01a05ef1-b3dc-785c-bf0e-6f77941c9e64] query_prometheus
log digest: 1 lines, 175 bytes

-- last 1 lines --
No output for this call was recorded in cassette 'c10'. This scenario was captured from a different sequence of tool calls; treat it as unknown rather than as an empty result.
  7. 7 llm model turn internal 2.8s · 8.6k tok · $0.0003

    model output — its reasoning, and the tools it asked for

    -> calls list_deployments
  8. 8 tool list_deployments kubernetes 0ms · 578 B

    arguments

    {"namespace":"hephaisto-chaos"}

    result digest — what the model actually saw

    [step 01a05ef1-bed4-7d0e-a367-3dd862541b58] list_deployments
log digest: 9 lines, 578 bytes

-- notable lines --
name                ready  up-to-date  available  age     spec observed
------------------  -----  ----------  ---------  ------  -------------
c1-oomkill          1/1    1           1          23h32m  yes
c10-faulty-service  1/1    1           1          5m      yes
c2-crashloop        0/1    1           0          7m      yes
c3-unschedulable    0/1    1           0          7m      yes

-- last 9 lines --
name                ready  up-to-date  available  age     spec observed
------------------  -----  ----------  ---------  ------  -------------
c1-oomkill          1/1    1           1          23h32m  yes
c10-faulty-service  1/1    1           1          5m      yes
c2-crashloop        0/1    1           0          7m      yes
c3-unschedulable    0/1    1           0          7m      yes
c4-imagepull        0/1    1           0          7m      yes
c7-configerror      0/1    1           0          7m      yes
c8-readiness-flap   0/1    1           0          7m      yes
  9. 9 llm model turn internal 2.3s · 9.0k tok · $0.0003

    model output — its reasoning, and the tools it asked for

    -> calls get_rollout_history
  10. 10 tool get_rollout_history kubernetes 0ms · 175 B

    arguments

    {"kind":"Deployment","name":"c10-faulty-service","namespace":"hephaisto-chaos"}

    result digest — what the model actually saw

    [step 01a05ef1-c7cb-78b7-aff4-29e488bf091d] get_rollout_history
log digest: 1 lines, 175 bytes

-- last 1 lines --
No output for this call was recorded in cassette 'c10'. This scenario was captured from a different sequence of tool calls; treat it as unknown rather than as an empty result.
  11. 11 llm model turn internal 1.9s · 9.1k tok · $0.0003

    model output — its reasoning, and the tools it asked for

    -> calls list_pods
  12. 12 tool list_pods kubernetes 0ms · 1.3 KiB

    arguments

    {"namespace":"hephaisto-chaos","labelSelector":null}

    result digest — what the model actually saw

    [step 01a05ef1-cf52-79b9-876b-7355a13017fd] list_pods
log digest: 12 lines, 1369 bytes

-- notable lines --
name                                 ready  phase    reason                      restarts  age  node
-----------------------------------  -----  -------  --------------------------  --------  ---  --------------------
c1-oomkill-6c8dd474dd-prnsn          1/1    Running  -                           0         7s   lima-rancher-desktop
c10-faulty-service-7d456f87ff-ft2vt  2/2    Running  -                           0         5m   lima-rancher-desktop
c2-crashloop-56b6585fd9-4hj9x        0/1    Running  CrashLoopBackOff            6         7m   lima-rancher-desktop
c3-unschedulable-fd55d4b7c-tpkg7     0/0    Pending  -                           0         7m   -

-- last 12 lines --
name                                 ready  phase    reason                      restarts  age  node
-----------------------------------  -----  -------  --------------------------  --------  ---  --------------------
c1-oomkill-6c8dd474dd-prnsn          1/1    Running  -                           0         7s   lima-rancher-desktop
c10-faulty-service-7d456f87ff-ft2vt  2/2    Running  -                           0         5m   lima-rancher-desktop
c2-crashloop-56b6585fd9-4hj9x        0/1    Running  CrashLoopBackOff            6         7m   lima-rancher-desktop
c3-unschedulable-fd55d4b7c-tpkg7     0/0    Pending  -                           0         7m   -
c4-imagepull-7fd88fdb68-5n8sf        0/1    Pending  ImagePullBackOff            0         7m   lima-rancher-desktop
c5-badjob-4bnkv                      0/1    Failed   -                           0         7m   lima-rancher-desktop
c5-badjob-4qr27                      0/1    Failed   -                           0         6m   lima-rancher-desktop
c5-badjob-lmbvg                      0/1    Failed   -                           0         7m   lima-rancher-desktop
c7-configerror-7cd96c86f9-4dlvl      0/1    Pending  CreateContainerConfigError  0         7m   lima-rancher-desktop
c8-readiness-flap-dd589d77c-6mxdb    0/1    Running  -                           0         7m   lima-rancher-desktop
  13. 13 llm model turn internal 4.0s · 9.8k tok · $0.0003

    model output — its reasoning, and the tools it asked for

    -> calls describe_pod
  14. 14 tool describe_pod kubernetes 0ms · 175 B

    arguments

    {"name":"c10-faulty-service-7d456f87ff-ft2vt","namespace":"hephaisto-chaos"}

    result digest — what the model actually saw

    [step 01a05ef1-dec2-7a68-aa4b-effda2cd8d90] describe_pod
log digest: 1 lines, 175 bytes

-- last 1 lines --
No output for this call was recorded in cassette 'c10'. This scenario was captured from a different sequence of tool calls; treat it as unknown rather than as an empty result.
  15. 15 llm model turn internal 3.3s · 10.0k tok · $0.0003

    model output — its reasoning, and the tools it asked for

    -> calls get_pod_logs
  16. 16 tool get_pod_logs kubernetes 0ms · 340 B

    arguments

    {"container":null,"name":"c10-faulty-service-7d456f87ff-ft2vt","namespace":"hephaisto-chaos","previous":false}

    result digest — what the model actually saw

    [step 01a05ef1-ebaf-7772-b348-c6186442e500] get_pod_logs
log digest: 1 lines, 340 bytes

-- last 1 lines --
ERROR: the Kubernetes API returned BadRequest: Operation returned an invalid status code 'BadRequest', response body {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"a container name must be specified for pod c10-faulty-service-7d456f87ff-ft2vt, choose one of: [app traffic]","reason":"BadRequest","code":400}
  17. 17 llm model turn internal 1.2s · 10.1k tok · $0.0003

    model output — its reasoning, and the tools it asked for

    -> calls get_pod_logs
  18. 18 tool get_pod_logs kubernetes 0ms · 340 B

    arguments

    {"container":"app","name":"c[??]???"}

    result digest — what the model actually saw

    [step 01a05ef1-f091-7d8d-bc9c-fde6e703b22f] get_pod_logs
log digest: 1 lines, 340 bytes

-- last 1 lines --
ERROR: the Kubernetes API returned BadRequest: Operation returned an invalid status code 'BadRequest', response body {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"a container name must be specified for pod c10-faulty-service-7d456f87ff-ft2vt, choose one of: [app traffic]","reason":"BadRequest","code":400}
  19. 19 llm model turn internal 2.8s · 10.3k tok · $0.0003

    model output — its reasoning, and the tools it asked for

    -> calls list_prometheus_metric_names
  20. 20 tool list_prometheus_metric_names grafana-mcp 0ms · 175 B

    arguments

    {"datasourceUid":"prometheus","limit":100,"page":1,"regex":"traces_spanmetrics"}

    result digest — what the model actually saw

    [step 01a05ef1-fb88-70bd-8f53-7aff85d32b7a] list_prometheus_metric_names
log digest: 1 lines, 175 bytes

-- last 1 lines --
No output for this call was recorded in cassette 'c10'. This scenario was captured from a different sequence of tool calls; treat it as unknown rather than as an empty result.
  21. 21 llm model turn internal 3.2s · 10.5k tok · $0.0003

    model output — its reasoning, and the tools it asked for

    -> calls get_pod_logs
  22. 22 tool get_pod_logs kubernetes 0ms · 340 B

    arguments

    {"container":"app","name":"c10-faulty-service-7d456f87ff-ft2vt","namespace":"hephaisto-chaos","previous":false}

    result digest — what the model actually saw

    [step 01a05ef2-07ff-710c-9017-b4f24724502f] get_pod_logs
log digest: 1 lines, 340 bytes

-- last 1 lines --
ERROR: the Kubernetes API returned BadRequest: Operation returned an invalid status code 'BadRequest', response body {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"a container name must be specified for pod c10-faulty-service-7d456f87ff-ft2vt, choose one of: [app traffic]","reason":"BadRequest","code":400}
  23. 23 llm model turn internal 3.2s · 10.7k tok · $0.0003

    model output — its reasoning, and the tools it asked for

    -> calls who_owns
  24. 24 tool who_owns kubernetes 0ms · 175 B

    arguments

    {"kind":"Pod","name":"c10-faulty-service-7d456f87ff-ft2vt","namespace":"hephaisto-chaos"}

    result digest — what the model actually saw

    [step 01a05ef2-1496-76a3-bfe0-f78f5344a030] who_owns
log digest: 1 lines, 175 bytes

-- last 1 lines --
No output for this call was recorded in cassette 'c10'. This scenario was captured from a different sequence of tool calls; treat it as unknown rather than as an empty result.
  25. 25 llm model turn internal 3.2s · 10.8k tok · $0.0003

    model output — its reasoning, and the tools it asked for

    -> calls query_prometheus
  26. 26 tool query_prometheus grafana-mcp 0ms · 175 B

    arguments

    {"datasourceUid":"prometheus","endTime":"now","expr":"sum(rate(traces_spanmetrics_calls_total{service=\u0022faulty-service\u0022,status_code=\u0022STATUS_CODE_ERROR\u0022}[5m]))","queryType":"instant"}

    result digest — what the model actually saw

    [step 01a05ef2-2141-7272-91b5-2bc4ca27de4f] query_prometheus
log digest: 1 lines, 175 bytes

-- last 1 lines --
No output for this call was recorded in cassette 'c10'. This scenario was captured from a different sequence of tool calls; treat it as unknown rather than as an empty result.
  27. 27 llm model turn internal 2.5s · 10.9k tok · $0.0003

    model output — its reasoning, and the tools it asked for

    -> calls get_pod_logs
  28. 28 tool get_pod_logs kubernetes 0ms · 340 B

    arguments

    {"container":"traffic","name":"c10-faulty-service-7d456f87ff-ft2vt","namespace":"hephaisto-chaos","previous":false}

    result digest — what the model actually saw

    [step 01a05ef2-2b1b-727d-b282-d0d8a3e8c820] get_pod_logs
log digest: 1 lines, 340 bytes

-- last 1 lines --
ERROR: the Kubernetes API returned BadRequest: Operation returned an invalid status code 'BadRequest', response body {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"a container name must be specified for pod c10-faulty-service-7d456f87ff-ft2vt, choose one of: [app traffic]","reason":"BadRequest","code":400}
  29. 29 llm model turn internal 3.7s · 11.2k tok · $0.0004

    model output — its reasoning, and the tools it asked for

    -> calls get_workload
  30. 30 tool get_workload kubernetes 0ms · 175 B

    arguments

    {"kind":"Deployment","name":"c10-faulty-service","namespace":"hephaisto-chaos"}

    result digest — what the model actually saw

    [step 01a05ef2-3965-72b3-9165-2754aaa61e12] get_workload
log digest: 1 lines, 175 bytes

-- last 1 lines --
No output for this call was recorded in cassette 'c10'. This scenario was captured from a different sequence of tool calls; treat it as unknown rather than as an empty result.
  31. 31 llm model turn internal 2.2s · 11.3k tok · $0.0004

    model output — its reasoning, and the tools it asked for

    -> calls get_pod_logs
  32. 32 tool get_pod_logs kubernetes 0ms · 340 B

    arguments

    {"container":"app","name":"c10-faulty-service-7d456f87ff-ft2vt","namespace":"hephaisto-chaos","previous":true}

    result digest — what the model actually saw

    [step 01a05ef2-41e9-789b-9129-b36d473bb140] get_pod_logs
log digest: 1 lines, 340 bytes

-- last 1 lines --
ERROR: the Kubernetes API returned BadRequest: Operation returned an invalid status code 'BadRequest', response body {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"a container name must be specified for pod c10-faulty-service-7d456f87ff-ft2vt, choose one of: [app traffic]","reason":"BadRequest","code":400}
  33. 33 llm model turn internal 2.5s · 11.5k tok · $0.0004

    model output — its reasoning, and the tools it asked for

    -> calls get_events
  34. 34 tool get_events kubernetes 0ms · 175 B

    arguments

    {"namespace":"hephaisto-chaos","objectName":"c10-faulty-service"}

    result digest — what the model actually saw

    [step 01a05ef2-4bca-7e75-b1d8-40528ff03492] get_events
log digest: 1 lines, 175 bytes

-- last 1 lines --
No output for this call was recorded in cassette 'c10'. This scenario was captured from a different sequence of tool calls; treat it as unknown rather than as an empty result.
  35. 35 llm model turn internal 3.8s · 11.7k tok · $0.0004

    model output — its reasoning, and the tools it asked for

    -> calls get_resource_usage
  36. 36 tool get_resource_usage kubernetes 0ms · 175 B

    arguments

    {"namespace":"hephaisto-chaos"}

    result digest — what the model actually saw

    [step 01a05ef2-5ac4-79d0-8bae-c1d9d9bcd59d] get_resource_usage
log digest: 1 lines, 175 bytes

-- last 1 lines --
No output for this call was recorded in cassette 'c10'. This scenario was captured from a different sequence of tool calls; treat it as unknown rather than as an empty result.
  37. 37 llm model turn internal 10.4s · 12.2k tok · $0.0005

    model output — its reasoning, and the tools it asked for

    -> calls get_pod_logs
  38. 38 tool get_pod_logs kubernetes 0ms · 340 B

    arguments

    {"container":"app","name":"c10-faulty-service-7d456f87ff-ft2vt","namespace":"hephaisto-chaos","previous":false}

    result digest — what the model actually saw

    [step 01a05ef2-8343-74e1-9e53-63fdb5d84408] get_pod_logs
log digest: 1 lines, 340 bytes

-- last 1 lines --
ERROR: the Kubernetes API returned BadRequest: Operation returned an invalid status code 'BadRequest', response body {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"a container name must be specified for pod c10-faulty-service-7d456f87ff-ft2vt, choose one of: [app traffic]","reason":"BadRequest","code":400}
  39. 39 llm model turn internal 7.2s · 12.2k tok · $0.0004

    model output — its reasoning, and the tools it asked for

    -> calls get_pod
  40. 40 tool get_pod kubernetes 0ms · 1.0 KiB

    arguments

    {"name":"c10-faulty-service-7d456f87ff-ft2vt","namespace":"hephaisto-chaos"}

    result digest — what the model actually saw

    [step 01a05ef2-9f73-7cfa-9688-38284e7d20fe] get_pod
log digest: 17 lines, 1025 bytes

-- last 17 lines --
pod hephaisto-chaos/c10-faulty-service-7d456f87ff-ft2vt
phase: Running  node: lima-rancher-desktop  age: 6m

conditions:
type                       status  reason  message  since
-------------------------  ------  ------  -------  -----
PodReadyToStartContainers  True    -       -        6m
Initialized                True    -       -        6m
Ready                      True    -       -        6m
ContainersReady            True    -       -        6m
PodScheduled               True    -       -        6m

containers:
container  ready  state    reason  restarts  last exit  image                         requests              limits
---------  -----  -------  ------  --------  ---------  ----------------------------  --------------------  --------------------
app        yes    running  -       0         -          hephaisto/faulty-service:dev  cpu=50m,memory=128Mi  cpu=1,memory=512Mi
traffic    yes    running  -       0         -          busybox:1.37                  cpu=10m,memory=16Mi   cpu=200m,memory=64Mi
  41. 41 llm model turn internal 8.9s · 3.0k tok · $0.0001

    model output — its reasoning, and the tools it asked for

    -> calls conclude
  42. 42 tool conclude internal 0ms · 129 B x failed
    x The arguments dictionary is missing a value for the required parameter 'request'. (Parameter 'arguments')

    arguments

    {"findings":[{"category":"unknown","confidence":0.2,"evidence":[],"hypothesis":"Insufficient data collected; cannot determine the cause of high error rate in faulty-service."}],"summary":"No trace or log evidence was retrieved for faulty-service, so the root cause of the observed 10% error rate remains undetermined."}

    result digest — what the model actually saw

    [step 01a05ef2-c21d-7eaa-b47d-63cc22437c22] conclude
log digest: 1 lines, 129 bytes

-- last 1 lines --
ERROR: conclude failed: The arguments dictionary is missing a value for the required parameter 'request'. (Parameter 'arguments')
  43. 43 llm model turn internal 8.6s · 3.4k tok · $0.0002

    model output — its reasoning, and the tools it asked for

    -> calls conclude
  44. 44 tool conclude internal 0ms · 141 B

    arguments

    {"request":{"findings":[{"category":"unknown","confidence":0.2,"evidence":[],"hypothesis":"Insufficient data collected; cannot determine the cause of high error rate in faulty-service."}],"summary":"No trace or log evidence was retrieved for faulty-service, so the root cause of the observed 10% error rate remains undetermined."}}

    result digest — what the model actually saw

    [step 01a05ef2-e3b9-7345-9f9b-a41ff93bb305] conclude
log digest: 1 lines, 141 bytes

-- last 1 lines --
Conclusion recorded. Your citations are now checked against what the tools actually returned; any that do not match are discarded. Stop here.

findings 0

No surviving findings. A finding whose evidence failed the grounding check is dropped rather than shown as fact, so this can mean the model concluded nothing or that everything it claimed failed verification.

plan

No plan was produced.

how it was graded

root cause
NoFinding
plan
NoPlan
structurally sound
no
recorded
2026-09-01
agent version
0.5.1-main.0.4+23df805946ec0c37b0adafafeb12130231adec93

prompt sha256:24a57f8cfff58fe9 STALE - prompts and runbooks now hash sha256:e0f1c24f9cb45b92