> For the complete documentation index, see [llms.txt](https://docs.espressosys.com/network/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.espressosys.com/network/developer/operators/run-a-node/p2p-troubleshooting.md). # Debug P2P Connectivity Checks for validator P2P connectivity: nc probes, key configuration, cliquenet metrics, and proxy settings. The validator P2P protocol (cliquenet) opens a connection symmetrically. As soon as the TCP connection is established, both sides concurrently send their supported protocol version range and read the peer's, then run an encrypted Noise handshake. Neither side waits to be spoken to first, and it makes no difference which one accepted and which one dialed. The version range is four bytes: a minimum and a maximum, each a big-endian `u16`. Nodes today support only version 1 and send `00 01 00 01`, which is what the examples below show. Supported ranges are configurable, so expect these bytes to change in a future release. What matters for the checks on this page is that four bytes arrive at all. Proxy and service mesh documentation calls this a **server-first protocol**, on the grounds that the accepting side writes before it has received anything. That is the term to search for in a proxy manual. This has two requirements: 1. Nothing in the path may **withhold** the connection's opening bytes. Anything that waits for the **client** to speak before forwarding deadlocks the connection. Observing bytes in passing is harmless; buffering them is not. 2. The node must hold the private key for the x25519 public key registered onchain, or peers cannot complete the handshake. The four checks below are ordered cheapest first. {% hint style="info" %} Examples on this page use `9977`, the default `ESPRESSO_NODE_CLIQUENET_BIND_ADDRESS`. Substitute the node's actual ports. The Istio and Envoy examples take the **container port** the node listens on. The `nc` probes take the **public port** peers dial, which may be a mapped `NodePort` such as `30021`. {% endhint %} ## 1. Probe the port with `nc` A healthy node sends its version range without being sent anything first, so four bytes come back from a bare connection. Run these from outside the network, for example from a laptop, against the registered public address and P2P port. ```bash # 1. Control: no input at all. This is what a real peer sees. nc -d -w3 $PUBLIC_HOST $P2P_PORT | xxd # 2. Four bytes: a version range, as a peer sends. printf "\x00\x01\x00\x01" | nc -w3 $PUBLIC_HOST $P2P_PORT | xxd # 3. Five bytes: one byte past the typical sniffer minimum. printf "\x00\x01\x00\x01\x00" | nc -w3 $PUBLIC_HOST $P2P_PORT | xxd ``` Healthy output, from all three: ``` 00000000: 0001 0001 .... ``` `-d` stops `nc` from reading stdin and `-w3` bounds the read wait. Each command returns after about 3 seconds; the reply itself arrives in milliseconds. Where `nc` has no `-d`, use `nc -w3 $PUBLIC_HOST $P2P_PORT < /dev/null | xxd` instead. | Result | Meaning | | --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | All three print `00 01 00 01` | Nothing is withholding bytes. The port is fine, continue to check 2 | | 1 and 2 are empty, 3 prints `00 01 00 01` | A byte-inspecting filter with a minimum-bytes threshold is in the path. Signature of PERMISSIVE mTLS or `tls_inspector`. See [Proxies and protocol sniffing](#proxies-and-protocol-sniffing) | | All three are empty but the connection succeeds | Something holds the connection open without forwarding it, or the node is not listening on that port | | Connection refused or timeout | Firewall, security group, or port mapping problem, not a sniffer | | Something other than a 4-byte version range arrives | The stream is being rewritten. Check for TLS termination or a PROXY protocol header | ## 2. Check what is registered onchain `stake-table-entry` prints everything the stake table contract holds for one address, including the x25519 public key and P2P address peers will use. ```bash docker run ghcr.io/espressosystems/espresso-network/staking-cli:main \ staking-cli --network mainnet stake-table-entry --address $VALIDATOR_ADDRESS ``` `--network` supplies the stake table address and a public Ethereum RPC, so no other configuration is needed. Use `--network decaf` for Decaf, and `--rpc-url` (or `L1_PROVIDER`) to substitute a different endpoint. `--address` defaults to the signer address, so pass it explicitly when running without a wallet configured. The relevant lines: ``` Validator: Status: Active ... x25519 public key: X25519_PK~... p2p address: node.example.com:9977 ``` `not set` for either value means the validator predates the V3 stake table and has never registered them. Peers cannot dial it at all. See [Register the validator](/network/developer/operators/run-a-node.md#register-the-validator). Confirm that the P2P address is the host and port the `nc` probes in check 1 succeeded against, and that the port matches `ESPRESSO_NODE_CLIQUENET_BIND_ADDRESS` after any port mapping. ## 3. Check the node's own x25519 key Peers encrypt to the public key registered onchain, so a node holding a different private key fails every inbound handshake. Two configurations cause this, and both are visible in the node's own logs. **No x25519 key configured.** The node starts anyway, with a random ephemeral key that changes on every restart, and logs at startup: ``` No x25519 key provided, generating a random ephemeral key. A persistent key (via ESPRESSO_NODE_PRIVATE_X25519_KEY or mnemonic) will be required for the Cliquenet protocol upgrade. ``` ```bash grep "No x25519 key provided" ``` This fallback is temporary and will be removed once x25519 keys are mandatory. A key file takes over x25519 configuration. When one is set, the key must be an `ESPRESSO_NODE_PRIVATE_X25519_KEY=...` line inside the file itself; setting it in the environment instead does not work. A key file with no such line starts with no error and lands here. The legacy `ESPRESSO_SEQUENCER_KEY_FILE` counts as a key file. The node migrates it automatically and logs: ``` migrated deprecated env var old="ESPRESSO_SEQUENCER_KEY_FILE" new="ESPRESSO_NODE_KEY_FILE" ``` ```bash grep "migrated deprecated env var" ``` Check that line even when no key file appears to be configured. A leftover `ESPRESSO_SEQUENCER_KEY_FILE` is easy to miss, and it has caused exactly this failure on Mainnet. **A key that does not match the registration.** Every inbound handshake fails and the node logs, for many different peers: ``` handshake failed err="noise error: decrypt error" ``` ```bash grep "noise error: decrypt error" ``` Either the node's configured key changed without re-registering, or the registered key was never the node's. Confirm against check 2 and re-register if needed. Both are fixed by putting a persistent x25519 key where the node's configuration expects it, in [Generate keys](/network/developer/operators/run-a-node.md#generate-keys). That section also covers printing the node's public key, for comparing against check 2 or re-registering. ## 4. Read the cliquenet metrics Enable the [`status` module](https://github.com/EspressoSystems/gitbook/tree/main/guides/operators/running-an-espresso-node.md#status) and read `/status/metrics`. Cliquenet metrics are prefixed `consensus_cliquenet_`. Every series carries a `peer` label holding a base58 x25519 public key, without the `X25519_PK~` prefix. The label means two different things depending on the metric: node-level gauges carry the node's **own** key, while per-peer counters and gauges carry the **remote** peer's key. | Metric | Type | `peer` label | Meaning | | --------------------------------------- | ------- | ------------ | --------------------------------------------------------------------------------------------------------------- | | `consensus_cliquenet_peer_tasks` | gauge | own | Established peer connections | | `consensus_cliquenet_accept_tasks` | gauge | own | Inbound TCP connections currently in the handshake | | `consensus_cliquenet_hello_tasks` | gauge | own | Inbound connections currently exchanging hellos | | `consensus_cliquenet_connect_tasks` | gauge | own | Outbound dials currently in flight | | `consensus_cliquenet_channel_size` | gauge | own | Queued outbound commands | | `consensus_cliquenet_lower_bound` | gauge | own | Lowest message slot still retained | | `consensus_cliquenet_hellos` | counter | remote | **Inbound** connections from that peer that got past the Noise handshake, including ones this node then rejects | | `consensus_cliquenet_connect_attempts` | counter | remote | **Outbound** dial tasks started for that peer | | `consensus_cliquenet_errors` | counter | remote | Peer failures after a connection was established | | `consensus_cliquenet_outbound_messages` | gauge | remote | Messages queued for that peer | | `consensus_cliquenet_retrying_messages` | gauge | remote | Messages awaiting retry for that peer | | `consensus_cliquenet_remaining_budget` | gauge | remote | Remaining send budget for that peer | {% hint style="info" %} These metrics are created lazily on first update. A missing series therefore carries information: a node that has never established a peer connection exports no `consensus_cliquenet_peer_tasks` at all. {% endhint %} `connect_attempts` is not a retry counter. One dial task retries internally on its own delay ladder and never gives up, so the counter sits at `1` per peer for as long as that first dial is outstanding. It increments again only when an established connection to that peer drops and a fresh dial task starts. ### Reading the numbers ```bash curl -s localhost:$ESPRESSO_NODE_API_PORT/v1/status/metrics | grep consensus_cliquenet ``` * **`peer_tasks` absent or 0**: the node has no P2P mesh connections at all. * **`peer_tasks` well below the number of other validators in the stake table**: partial connectivity. Compare the `peer` labels on `outbound_messages` against the stake table to find which peers are missing. * **No `hellos` series at all**: nothing has ever reached the node inbound. Because each side dials the other and one connection serves both directions, the node's own outbound dials can keep `peer_tasks` looking healthy while the inbound path is completely broken. This is the signature of an inspecting proxy, a closed port, or a registered P2P address that does not route to the node. Confirm with probe 1 above. * **`hellos` present for a key but no connection to it**: peers reach the node and get past the Noise handshake, but the node rejects them. Check the logs for `party has invalid ip addr` and `unknown party`. * **`hellos` absent for peers that should be dialing, while `handshake failed` with `err="noise error: decrypt error"` appears in the logs**: those peers are encrypting to a different x25519 key than this node holds. Go back to check 3. * **`connect_tasks` elevated and never draining, `connect_attempts` stuck at 1 for those peers**: outbound dials are hanging with nothing on the other end answering. Those peers' inbound path is broken, or egress from this node is being proxied. Their side shows the same `hellos` gap described above. * **`connect_attempts` above 1 and climbing for a peer**: established connections to that peer keep dropping and reconnecting. Cross-check `errors` and `peer failure` in the logs. * **`errors` rising for one peer**: connections establish and then drop. Look for byte corruption in the path, such as TLS termination or a PROXY protocol header. ### Logs These messages are `WARN` except where noted, and most carry an `err` field that identifies the cause. The `err` values listed below are the ones that actually occur on Mainnet. Messages about connections this node accepted: | Log message | Meaning | | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `handshake failed` | Any accept-side failure before the peer was identified. Read `err`: `noise error: decrypt error` means the dialing peer encrypted to a different x25519 public key than this node holds, and is by far the most common value; the rest are `incompatible versions`, `timeout`, `i/o error: early eof`, and connection resets | | `incompatible versions` | The two version ranges do not overlap. Check node versions, or check for bytes prepended to the stream: a PROXY protocol header parses as a nonsense version range | | `party has invalid ip addr` | An incoming connection's source IP does not match that peer's registered P2P address. Common on Mainnet, usually split ingress and egress paths on the peer's side | | `hello failed` | The peer was identified but a hello was not `Ok`. Accompanies `party has invalid ip addr` one for one | | `hello task error` | The hello exchange errored after the peer was identified | | `peer failure` | An established peer connection dropped | | `unknown party` (`INFO`) | A connection arrived from an x25519 key that is not in the stake table | Messages about connections this node dialed: | Log message | Meaning | | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `connect/handshake error` | The dial failed. Read `err`: `i/o error: Connection refused` (nothing listening, or a port mapping problem), `timeout` (a middlebox holding the connection without forwarding it, or a dropped SYN), `i/o error: unexpected end of file` (something accepted the connection and closed it), plus `Connection timed out`, `Network is unreachable`, `No route to host`, and DNS lookup failures | | `hello response was not ok` | The remote identified this node and refused it. That peer's logs carry the reason, typically `party has invalid ip addr` | | `failed to exchange hello` | The hello exchange timed out or errored after connecting | ## Proxies and protocol sniffing Byte inspection goes by many names. Any of the following in front of the P2P port may require your attention. **Service meshes and sidecars**: Istio (Envoy sidecar), Linkerd, Consul Connect, Cilium Service Mesh, AWS App Mesh, Kuma, Open Service Mesh, Traefik Mesh. **Envoy listener filters**: `tls_inspector`, `http_inspector`, automatic protocol detection, protocol sniffing, PERMISSIVE mTLS. **Ingress controllers and TCP routers**: NGINX Ingress (TCP/UDP ConfigMap, `ssl_preread`), Traefik TCP routers with `HostSNI`, HAProxy Ingress, Contour, Kong, Emissary / Ambassador, Gateway API TLSRoute. **Reverse proxies and SNI routers**: nginx `stream` with `ssl_preread`, HAProxy with `tcp-request inspect-delay` or `req.ssl_sni`, Traefik, Caddy `layer4`, standalone Envoy, sslh, sniproxy, stunnel, ghostunnel. **Cloud load balancers**: AWS ALB, AWS NLB with a TLS listener, GCP HTTPS / SSL Proxy / TCP Proxy load balancers, Azure Application Gateway, Azure Front Door, Cloudflare Spectrum, Cloudflare Tunnel, Fastly, Akamai. **Firewalls, IDS/IPS, and DPI**: deep packet inspection (DPI), next-generation firewall (NGFW), application control, application-layer gateway (ALG), TLS inspection or SSL/TLS interception, WAF, DDoS scrubbing, Palo Alto App-ID, Fortinet FortiGate, Check Point, Cisco Firepower, Snort, Suricata, Zenarmor, Zscaler, pfSense or OPNsense with an IPS plugin. **Header-prepending proxies**: PROXY protocol v1/v2 (HAProxy, AWS NLB, Cloudflare Spectrum). Cliquenet does not parse the PROXY header, so the prepended bytes corrupt the version exchange. Two failure modes to keep separate: * **Detection deadlock**: the middlebox holds back the accepting side's first bytes while waiting for the client. Symptom: the node looks dead on the P2P port even though it is listening and otherwise healthy. * **Byte corruption**: the middlebox rewrites or prepends to the stream (PROXY protocol, TLS termination). Symptom: connections establish and then fail the version exchange or handshake. ### What is safe in the path Forwarding that never holds bytes back: * Kubernetes `NodePort`, `hostPort`, and `hostNetwork` pods. * `iptables` / `nftables` DNAT, port forwards, and NAT gateways. * AWS NLB with a **TCP** listener (not TLS), GCP passthrough network load balancer, Azure Load Balancer (L4). * WireGuard, Tailscale, and IPsec tunnels. * Stateful firewall rules that match only addresses and ports. {% hint style="warning" %} Passthrough must also preserve the source IP if an **IP address** is registered as the P2P address, because peers check the source IP of every incoming connection. Register a DNS name instead when the egress IP differs from the ingress IP. See [Register the validator](/network/developer/operators/run-a-node.md#register-the-validator). {% endhint %} ## If a service mesh must stay in the path Istio documents this failure class under [server-first protocols](https://istio.io/latest/docs/ops/deployment/application-requirements/#server-first-protocols). Both automatic protocol detection and PERMISSIVE mTLS wait for a connection's opening bytes before forwarding it, so both break cliquenet. The P2P port is not on Istio's list of ports exempted from sniffing automatically. ### Option 1: exclude the port from the sidecar (recommended) This removes the inbound iptables redirect, so packets reach the node's socket with no filter chain at all. ```yaml metadata: annotations: # The node's cliquenet container port, 9977 by default. traffic.sidecar.istio.io/excludeInboundPorts: "9977" ``` Three conditions: * The value is the pod's **container port**. If a Service maps a `NodePort` such as `30021` to container port `9977`, the annotation takes `9977`. Excluding the `NodePort` has no effect. * It applies only when inbound capture is the default wildcard. If `traffic.sidecar.istio.io/includeInboundPorts` is set to an explicit list, remove the P2P port from that list instead. * It is a **pod** annotation. Set it on the pod template in the Deployment or StatefulSet, not on the Service. ### Option 2: keep the sidecar and disable both inspectors Both changes are required. Either one alone leaves a byte-inspecting filter attached. ```yaml # 1. Disable protocol sniffing: declare the port as TCP. apiVersion: v1 kind: Service spec: ports: - name: tcp-cliquenet # or set appProtocol: tcp port: 9977 # the node's cliquenet container port --- # 2. Disable PERMISSIVE-mode mTLS detection on the port. apiVersion: security.istio.io/v1 kind: PeerAuthentication metadata: name: cliquenet-plaintext spec: selector: matchLabels: app: portLevelMtls: 9977: mode: DISABLE ``` `portLevelMtls` keys are container ports. `STRICT` also removes the inspector but is wrong here: peer validators are outside the mesh and do not speak mTLS. Confirm that no inspector is attached: ```bash istioctl proxy-config listener --port 9977 -o json | grep -i inspector ``` Under option 1 the listener is absent entirely. Under option 2 the command returns nothing. For other Envoy-based proxies, the equivalent change is to remove the `tls_inspector` and `http_inspector` listener filters and route the port through a plain TCP proxy filter chain, with no filter chain match on transport protocol or ALPN. After the change, rerun the `nc` probes from check 1. All three must print `00 01 00 01`. ## Getting help The lists above are not exhaustive, and Espresso does not test every proxy, mesh, and firewall product. If a P2P connectivity problem persists after the checks on this page, open an issue at [espresso-network/issues](https://github.com/EspressoSystems/espresso-network/issues) or ask on the [Espresso Discord](https://discord.gg/GJa4gznGfU). Include the output of the three `nc` probes and the node's `consensus_cliquenet_` metrics. ## Related references * [Run a Validator Node](/network/developer/operators/run-a-node.md) * [Istio: server-first protocols](https://istio.io/latest/docs/ops/deployment/application-requirements/#server-first-protocols) * [Istio: explicit protocol selection](https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/) * [Istio: resource annotations](https://istio.io/latest/docs/reference/config/annotations/)