Private network nodes can't find peers

Question

I have 4 nodes (each on a separate droplet on digital ocean) in a private network:

• bootnode
• miner1
• miner2
• jsonrpc

Each node is initialized with geth --datadir ./data init ./genesis.json using the exact same genesis.json

{
    "config": {
      "chainId": 55055,
      "homesteadBlock": 0,
      "eip150Block": 0,
      "eip155Block": 0,
      "eip158Block": 0,
      "byzantiumBlock": 0,
      "constantinopleBlock": 0,
      "petersburgBlock": 0,
      "ethash": {}
    },
    "difficulty": "1",
    "gasLimit": "12000000",
    "alloc": {}
}

I generate the bootnode.key with bootnode -genkey bootnode.key and start the bootnode with

geth --datadir ./data --nodekey bootnode.key --nodiscover --ipcdisable 
    \ --networkid 55055 --identity bootnode console

The jsonrpc node with

geth --datadir ./data --bootnodes $ENODE_ADDRESS --allow-insecure-unlock --http 
    \ --http.addr="0.0.0.0" --http.api="eth,web3,net,admin,personal" 
    \ --http.corsdomain="*" --identity jsonrpc --networkid 55055 console

where the $ENODE_ADDRESS is the enode address of the bootnode, e.g. enode://pubkey@ip:30303

The miner nodes with

geth --datadir ./data --bootnodes $ENODE_ADDRESS --mine --miner.threads 1 
    \ --miner.etherbase $ACCOUNT --networkid 55055 --identity miner console

where $ACCOUNT is an existing ETH wallet address. p.s. the identity is "miner2" on the other miner node.

With this setup everything starts correctly, the jsonrpc and the miners connect to the bootnode. I can make HTTP requests (e.g. via PostMan) to the public IP of the jsonrpc node and I get the expected HTTP response. I can even connect MetaMask to the chain and the wallet balance of those accounts specified to the miners increase (with each mined block).

The only issue I have is that the nodes do NOT connect to each other. net.listening is true on each. But net.peerCount is 1 on the jsonrpc and the miner nodes (it is 3 on the bootnode as expected).

I can add peer manually via admin.addPeer. I tried to add miner2 to miner1 and it worked, net.peerCount now shows 2 on each of the miner nodes (and admin.peers shows the correct peers = [bootnode, miner1] on miner2 and [bootnode, miner2] on miner1).

But ... despite the jsonrpc and miner nodes being connected to the bootnode, they just don't find each other. The console says "Looking for peers" on regular basis but finds/adds nothing.

The console on the jsonrpc server does show the "Snapshot extension registration failed ... peer connected on snap without compatible eth support" error from time to time (as well as the "Server parity_netPeers" warning)

Any ideas how I can get the nodes to find (and connected to) each other after they connect to the bootnode?

p.s. I've read the following questions/answers (which have not helped)

• Peer discovery not working on private network
• How to get Nodes on local Geth Private Network to Discover Peers OF THEIR PEERS
• Connection between peers never happen on custom blockchain [duplicate]
• Can't join nodes/peers to private network
• geth not connecting to bootnodes on private test network (github)
• Ethereum private blockchain: peers can not see each others via internet

Edit #1: I can see the following console output on regular basis

INFO [02-11|10:03:43.180] Looking for peers     peercount=2 tried=25 static=0
> net.peerCount
1
> ERROR[02-11|10:03:50.094] Snapshot extension registration failed
   peer=84bfdb1d err="peer connected on snap without compatible eth support"

i.e. the INFO output gives peercount=2 but the net.peerCount is actualy 1

Answer 1

Try using --nat exip:<external-ip>, where the external IP is specified so that the nodes find reach other.

Answer 2

Had similar setup and the same problem but have found some clues to this. Same experience with peering, i.e. bootnode had peers but the worker nodes did not. Initially ran a cluster in docker locally without issues but didn't look too closely as my transactions were being processed.

Deployed the setup to a kubernetes cluster and started having issues with peering. Got ethstats running in the cluster for observability and noticed my transactions were not propagating from my rpc node to the miner nodes due to peering was my best guess although as far as I could tell the txpool was receiving the transactions so don't know why the nodes couldn't pick them up.

Finally got peering to work by turning off --nodiscover on the bootnode and adding a netrestrict policy to all nodes + service origins cidr's to the worker nodes to protect them. Result was all my expected nodes peered but I have a bunch of noise for invalid neighbours which I'm guessing is related to geth having some logic to do some discovery of it's own when --nodiscover flag is turned off.

So my best guess on this from observation is that --nodiscover behaviour is propagated to connected peers, causing them to ignore the peer information container on the bootnode. Turning this off with and adding the --netrestrict cidr is giving me the desired behaviour but I don't know yet if there are any negative consequences for this. Perhaps an expert on this could comment?