hydra-head-troubleshooter Skill

hydra-head-troubleshooter

When to use

Hydra Head doesn't start or no head observed
Head exists but doesn't make progress
Peers out of sync or disconnected
Log messages: PeerConnected, AckSn, LogicOutcome errors

Operating rules (must follow)

Confirm network and hydra-node version first
Request logs from ALL participants for same time window
Never request key contents (file paths OK)
Output: (1) Root cause (2) Fix steps (3) Verification

Docker fallback mode

If hydra-node is not installed locally, use the wrapper script in this skill folder to run hydra-node inside Docker (Hydra upstream recommends Docker images for quickest start).

chmod +x {baseDir}/scripts/hydra-node.sh
{baseDir}/scripts/hydra-node.sh --help
{baseDir}/scripts/hydra-node.sh gen-hydra-key --output-file hydra

For full multi-node Head demos, prefer the hydra.family Docker Compose demo (it's the canonical "known-good" setup).

Quick diagnostic commands

# Check hydra-node version
hydra-node --version

# Check API health
curl -s localhost:4001/health

# Check peers
curl -s localhost:4001/peers

# Check head status
curl -s localhost:4001/status

# Check metrics (if enabled)
curl -s localhost:6001/metrics | grep hydra

Decision tree

A) "No head is observed from the chain"

Symptoms:

Head never appears in logs
No Init/Commit/Open progression
Client shows no head state

Check 1: Cardano connection

# Verify cardano-node is ready
cardano-cli query tip --testnet-magic 1
# Should show current slot, not error

# Check socket exists
ls -la $CARDANO_NODE_SOCKET_PATH

Fix: Wait for cardano-node sync, verify socket path and network magic

Check 2: Scripts tx id

# Verify you're using correct scripts tx id for network
# Get from hydra-node release notes

# In hydra-node logs, look for:
grep -i "script" hydra-node.log | head -20

Fix: Use correct --hydra-scripts-tx-id for your network

Check 3: Key mismatch

# Verify cardano.sk matches what peers have as your vkey
cardano-cli key verification-key \
  --signing-key-file cardano.sk \
  --verification-key-file check.vkey

# Compare check.vkey with what you distributed

Fix: Re-exchange verification keys with all peers

B) "Head does not make progress"

Symptoms:

Head exists but stuck at Init or Commit
Snapshots not confirmed
Commands hang or timeout

Check 1: Peer connectivity

# In logs, look for PeerConnected
grep "PeerConnected" hydra-node.log

# Check metrics
curl -s localhost:6001/metrics | grep peers_connected

Fix: Verify --peer host:port is correct and ports are reachable

Check 2: Hydra key mismatch

# Look for AckSn issues
grep -E "AckSn|LogicOutcome" hydra-node.log

# Verify hydra keys match
hydra-node verify-keys \
  --hydra-signing-key hydra.sk \
  --hydra-verification-key peer-hydra.vk

Fix: Re-exchange hydra verification keys

Check 3: Version mismatch

# All nodes must run compatible versions
hydra-node --version
# Compare across all participants

Fix: Align hydra-node versions

C) "Head stuck: peer out of sync"

Symptoms:

One node accepts txs, others reject
Snapshots stop being signed
Ledger state diverged

Cause: Different ledger views due to config drift or missed txs

Fix: Snapshot side-loading

# 1. Get latest snapshot from healthy node
curl -s http://healthy-node:4001/snapshot > snapshot.json

# 2. Post to out-of-sync node
curl -X POST http://out-of-sync-node:4001/snapshot \
  -H "Content-Type: application/json" \
  -d @snapshot.json

Verify: Snapshots resume, all parties sign again

D) Mirror nodes / HA issues

Symptoms:

SnapshotAlreadySigned in logs
etcd quorum issues

Reality:

SnapshotAlreadySigned is transient and harmless with mirrors
Keep mirror count < floor(n/2) for quorum stability

Fix: Reduce mirror count or accept transient warnings

Log patterns to grep

| Pattern | Meaning | |---------|---------| | PeerConnected | Peer connected successfully | | PeerDisconnected | Peer lost connection | | AckSn | Snapshot acknowledgment | | LogicOutcome | Head logic result (check for errors) | | SnapshotConfirmed | Snapshot finalized | | HeadIsOpen | Head successfully opened | | HeadIsClosed | Head closed, contestation started |

Verification checklist

After any fix:

[ ] All nodes same hydra-node version
[ ] Scripts tx id correct for network
[ ] cardano-node ready (query tip works)
[ ] PeerConnected for all peers
[ ] AckSn messages flowing
[ ] No LogicOutcome errors

Examples

Example: Diagnose stuck head

User request: "My Hydra head won't open—stuck at commit"

Response:

# 1. Check all peers connected
grep "PeerConnected" hydra-node.log | tail -10
# Should see PeerConnected for each peer

# 2. If missing peers, check network
nc -zv peer-host 5001
# Should show "Connection succeeded"

# 3. Check all parties committed
curl -s localhost:4001/status | jq .
# Look for parties and their commit status

# 4. Check for key issues
grep -E "InvalidSignature|WrongParty" hydra-node.log

# Common fix: restart with correct peer addresses
# Verify --peer host:port matches actual peer locations

References

hydra-head (operational guidance)
hydra-head-operator (execution)
reference/sources.md (doc provenance)
hydra.family troubleshooting

Agent Skills: hydra-head-troubleshooter

Install this agent skill to your local

Skill Files