Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Debug P2P Connectivity

Checks for validator P2P connectivity: nc probes, key configuration, cliquenet metrics, and proxy settings.

The validator P2P protocol (cliquenet) opens a connection symmetrically. As soon as the TCP connection is established, both sides concurrently send their supported protocol version range and read the peer's, then run an encrypted Noise handshake. Neither side waits to be spoken to first, and it makes no difference which one accepted and which one dialed.

The version range is four bytes: a minimum and a maximum, each a big-endian u16. Nodes today support only version 1 and send 00 01 00 01, which is what the examples below show. Supported ranges are configurable, so expect these bytes to change in a future release. What matters for the checks on this page is that four bytes arrive at all.

Proxy and service mesh documentation calls this a server-first protocol, on the grounds that the accepting side writes before it has received anything. That is the term to search for in a proxy manual.

This has two requirements:

  1. Nothing in the path may withhold the connection's opening bytes. Anything that waits for the client to speak before forwarding deadlocks the connection. Observing bytes in passing is harmless; buffering them is not.

  2. The node must hold the private key for the x25519 public key registered onchain, or peers cannot complete the handshake.

The four checks below are ordered cheapest first.

Examples on this page use 9977, the default ESPRESSO_NODE_CLIQUENET_BIND_ADDRESS. Substitute the node's actual ports. The Istio and Envoy examples take the container port the node listens on. The nc probes take the public port peers dial, which may be a mapped NodePort such as 30021.

1. Probe the port with nc

A healthy node sends its version range without being sent anything first, so four bytes come back from a bare connection. Run these from outside the network, for example from a laptop, against the registered public address and P2P port.

# 1. Control: no input at all. This is what a real peer sees.
nc -d -w3 $PUBLIC_HOST $P2P_PORT | xxd

# 2. Four bytes: a version range, as a peer sends.
printf "\x00\x01\x00\x01" | nc -w3 $PUBLIC_HOST $P2P_PORT | xxd

# 3. Five bytes: one byte past the typical sniffer minimum.
printf "\x00\x01\x00\x01\x00" | nc -w3 $PUBLIC_HOST $P2P_PORT | xxd

Healthy output, from all three:

-d stops nc from reading stdin and -w3 bounds the read wait. Each command returns after about 3 seconds; the reply itself arrives in milliseconds. Where nc has no -d, use nc -w3 $PUBLIC_HOST $P2P_PORT < /dev/null | xxd instead.

Result
Meaning

All three print 00 01 00 01

Nothing is withholding bytes. The port is fine, continue to check 2

1 and 2 are empty, 3 prints 00 01 00 01

A byte-inspecting filter with a minimum-bytes threshold is in the path. Signature of PERMISSIVE mTLS or tls_inspector. See Proxies and protocol sniffing

All three are empty but the connection succeeds

Something holds the connection open without forwarding it, or the node is not listening on that port

Connection refused or timeout

Firewall, security group, or port mapping problem, not a sniffer

Something other than a 4-byte version range arrives

The stream is being rewritten. Check for TLS termination or a PROXY protocol header

2. Check what is registered onchain

stake-table-entry prints everything the stake table contract holds for one address, including the x25519 public key and P2P address peers will use.

--network supplies the stake table address and a public Ethereum RPC, so no other configuration is needed. Use --network decaf for Decaf, and --rpc-url (or L1_PROVIDER) to substitute a different endpoint. --address defaults to the signer address, so pass it explicitly when running without a wallet configured.

The relevant lines:

not set for either value means the validator predates the V3 stake table and has never registered them. Peers cannot dial it at all. See Register the validator.

Confirm that the P2P address is the host and port the nc probes in check 1 succeeded against, and that the port matches ESPRESSO_NODE_CLIQUENET_BIND_ADDRESS after any port mapping.

3. Check the node's own x25519 key

Peers encrypt to the public key registered onchain, so a node holding a different private key fails every inbound handshake. Two configurations cause this, and both are visible in the node's own logs.

No x25519 key configured. The node starts anyway, with a random ephemeral key that changes on every restart, and logs at startup:

This fallback is temporary and will be removed once x25519 keys are mandatory.

A key file takes over x25519 configuration. When one is set, the key must be an ESPRESSO_NODE_PRIVATE_X25519_KEY=... line inside the file itself; setting it in the environment instead does not work. A key file with no such line starts with no error and lands here.

The legacy ESPRESSO_SEQUENCER_KEY_FILE counts as a key file. The node migrates it automatically and logs:

Check that line even when no key file appears to be configured. A leftover ESPRESSO_SEQUENCER_KEY_FILE is easy to miss, and it has caused exactly this failure on Mainnet.

A key that does not match the registration. Every inbound handshake fails and the node logs, for many different peers:

Either the node's configured key changed without re-registering, or the registered key was never the node's. Confirm against check 2 and re-register if needed.

Both are fixed by putting a persistent x25519 key where the node's configuration expects it, in Generate keys. That section also covers printing the node's public key, for comparing against check 2 or re-registering.

4. Read the cliquenet metrics

Enable the status module and read /status/metrics. Cliquenet metrics are prefixed consensus_cliquenet_. Every series carries a peer label holding a base58 x25519 public key, without the X25519_PK~ prefix. The label means two different things depending on the metric: node-level gauges carry the node's own key, while per-peer counters and gauges carry the remote peer's key.

Metric

Type

peer label

Meaning

consensus_cliquenet_peer_tasks

gauge

own

Established peer connections

consensus_cliquenet_accept_tasks

gauge

own

Inbound TCP connections currently in the handshake

consensus_cliquenet_hello_tasks

gauge

own

Inbound connections currently exchanging hellos

consensus_cliquenet_connect_tasks

gauge

own

Outbound dials currently in flight

consensus_cliquenet_channel_size

gauge

own

Queued outbound commands

consensus_cliquenet_lower_bound

gauge

own

Lowest message slot still retained

consensus_cliquenet_hellos

counter

remote

Inbound connections from that peer that got past the Noise handshake, including ones this node then rejects

consensus_cliquenet_connect_attempts

counter

remote

Outbound dial tasks started for that peer

consensus_cliquenet_errors

counter

remote

Peer failures after a connection was established

consensus_cliquenet_outbound_messages

gauge

remote

Messages queued for that peer

consensus_cliquenet_retrying_messages

gauge

remote

Messages awaiting retry for that peer

consensus_cliquenet_remaining_budget

gauge

remote

Remaining send budget for that peer

These metrics are created lazily on first update. A missing series therefore carries information: a node that has never established a peer connection exports no consensus_cliquenet_peer_tasks at all.

connect_attempts is not a retry counter. One dial task retries internally on its own delay ladder and never gives up, so the counter sits at 1 per peer for as long as that first dial is outstanding. It increments again only when an established connection to that peer drops and a fresh dial task starts.

Reading the numbers

  • peer_tasks absent or 0: the node has no P2P mesh connections at all.

  • peer_tasks well below the number of other validators in the stake table: partial connectivity. Compare the peer labels on outbound_messages against the stake table to find which peers are missing.

  • No hellos series at all: nothing has ever reached the node inbound. Because each side dials the other and one connection serves both directions, the node's own outbound dials can keep peer_tasks looking healthy while the inbound path is completely broken. This is the signature of an inspecting proxy, a closed port, or a registered P2P address that does not route to the node. Confirm with probe 1 above.

  • hellos present for a key but no connection to it: peers reach the node and get past the Noise handshake, but the node rejects them. Check the logs for party has invalid ip addr and unknown party.

  • hellos absent for peers that should be dialing, while handshake failed with err="noise error: decrypt error" appears in the logs: those peers are encrypting to a different x25519 key than this node holds. Go back to check 3.

  • connect_tasks elevated and never draining, connect_attempts stuck at 1 for those peers: outbound dials are hanging with nothing on the other end answering. Those peers' inbound path is broken, or egress from this node is being proxied. Their side shows the same hellos gap described above.

  • connect_attempts above 1 and climbing for a peer: established connections to that peer keep dropping and reconnecting. Cross-check errors and peer failure in the logs.

  • errors rising for one peer: connections establish and then drop. Look for byte corruption in the path, such as TLS termination or a PROXY protocol header.

Logs

These messages are WARN except where noted, and most carry an err field that identifies the cause. The err values listed below are the ones that actually occur on Mainnet.

Messages about connections this node accepted:

Log message
Meaning

handshake failed

Any accept-side failure before the peer was identified. Read err: noise error: decrypt error means the dialing peer encrypted to a different x25519 public key than this node holds, and is by far the most common value; the rest are incompatible versions, timeout, i/o error: early eof, and connection resets

incompatible versions

The two version ranges do not overlap. Check node versions, or check for bytes prepended to the stream: a PROXY protocol header parses as a nonsense version range

party has invalid ip addr

An incoming connection's source IP does not match that peer's registered P2P address. Common on Mainnet, usually split ingress and egress paths on the peer's side

hello failed

The peer was identified but a hello was not Ok. Accompanies party has invalid ip addr one for one

hello task error

The hello exchange errored after the peer was identified

peer failure

An established peer connection dropped

unknown party (INFO)

A connection arrived from an x25519 key that is not in the stake table

Messages about connections this node dialed:

Log message
Meaning

connect/handshake error

The dial failed. Read err: i/o error: Connection refused (nothing listening, or a port mapping problem), timeout (a middlebox holding the connection without forwarding it, or a dropped SYN), i/o error: unexpected end of file (something accepted the connection and closed it), plus Connection timed out, Network is unreachable, No route to host, and DNS lookup failures

hello response was not ok

The remote identified this node and refused it. That peer's logs carry the reason, typically party has invalid ip addr

failed to exchange hello

The hello exchange timed out or errored after connecting

Proxies and protocol sniffing

Byte inspection goes by many names. Any of the following in front of the P2P port may require your attention.

Service meshes and sidecars: Istio (Envoy sidecar), Linkerd, Consul Connect, Cilium Service Mesh, AWS App Mesh, Kuma, Open Service Mesh, Traefik Mesh.

Envoy listener filters: tls_inspector, http_inspector, automatic protocol detection, protocol sniffing, PERMISSIVE mTLS.

Ingress controllers and TCP routers: NGINX Ingress (TCP/UDP ConfigMap, ssl_preread), Traefik TCP routers with HostSNI, HAProxy Ingress, Contour, Kong, Emissary / Ambassador, Gateway API TLSRoute.

Reverse proxies and SNI routers: nginx stream with ssl_preread, HAProxy with tcp-request inspect-delay or req.ssl_sni, Traefik, Caddy layer4, standalone Envoy, sslh, sniproxy, stunnel, ghostunnel.

Cloud load balancers: AWS ALB, AWS NLB with a TLS listener, GCP HTTPS / SSL Proxy / TCP Proxy load balancers, Azure Application Gateway, Azure Front Door, Cloudflare Spectrum, Cloudflare Tunnel, Fastly, Akamai.

Firewalls, IDS/IPS, and DPI: deep packet inspection (DPI), next-generation firewall (NGFW), application control, application-layer gateway (ALG), TLS inspection or SSL/TLS interception, WAF, DDoS scrubbing, Palo Alto App-ID, Fortinet FortiGate, Check Point, Cisco Firepower, Snort, Suricata, Zenarmor, Zscaler, pfSense or OPNsense with an IPS plugin.

Header-prepending proxies: PROXY protocol v1/v2 (HAProxy, AWS NLB, Cloudflare Spectrum). Cliquenet does not parse the PROXY header, so the prepended bytes corrupt the version exchange.

Two failure modes to keep separate:

  • Detection deadlock: the middlebox holds back the accepting side's first bytes while waiting for the client. Symptom: the node looks dead on the P2P port even though it is listening and otherwise healthy.

  • Byte corruption: the middlebox rewrites or prepends to the stream (PROXY protocol, TLS termination). Symptom: connections establish and then fail the version exchange or handshake.

What is safe in the path

Forwarding that never holds bytes back:

  • Kubernetes NodePort, hostPort, and hostNetwork pods.

  • iptables / nftables DNAT, port forwards, and NAT gateways.

  • AWS NLB with a TCP listener (not TLS), GCP passthrough network load balancer, Azure Load Balancer (L4).

  • WireGuard, Tailscale, and IPsec tunnels.

  • Stateful firewall rules that match only addresses and ports.

Passthrough must also preserve the source IP if an IP address is registered as the P2P address, because peers check the source IP of every incoming connection. Register a DNS name instead when the egress IP differs from the ingress IP. See Register the validator.

If a service mesh must stay in the path

Istio documents this failure class under server-first protocols. Both automatic protocol detection and PERMISSIVE mTLS wait for a connection's opening bytes before forwarding it, so both break cliquenet. The P2P port is not on Istio's list of ports exempted from sniffing automatically.

Option 1: exclude the port from the sidecar (recommended)

This removes the inbound iptables redirect, so packets reach the node's socket with no filter chain at all.

Three conditions:

  • The value is the pod's container port. If a Service maps a NodePort such as 30021 to container port 9977, the annotation takes 9977. Excluding the NodePort has no effect.

  • It applies only when inbound capture is the default wildcard. If traffic.sidecar.istio.io/includeInboundPorts is set to an explicit list, remove the P2P port from that list instead.

  • It is a pod annotation. Set it on the pod template in the Deployment or StatefulSet, not on the Service.

Option 2: keep the sidecar and disable both inspectors

Both changes are required. Either one alone leaves a byte-inspecting filter attached.

portLevelMtls keys are container ports. STRICT also removes the inspector but is wrong here: peer validators are outside the mesh and do not speak mTLS.

Confirm that no inspector is attached:

Under option 1 the listener is absent entirely. Under option 2 the command returns nothing.

For other Envoy-based proxies, the equivalent change is to remove the tls_inspector and http_inspector listener filters and route the port through a plain TCP proxy filter chain, with no filter chain match on transport protocol or ALPN.

After the change, rerun the nc probes from check 1. All three must print 00 01 00 01.

Getting help

The lists above are not exhaustive, and Espresso does not test every proxy, mesh, and firewall product. If a P2P connectivity problem persists after the checks on this page, open an issue at espresso-network/issues or ask on the Espresso Discord. Include the output of the three nc probes and the node's consensus_cliquenet_ metrics.

Related references

PreviousRun a Validator Node
NextNode Configuration Reference

Last updated