Goal: Running Nats containerized in K3S on a cluster of Raspberry Pis, with Nats to communicate between several applications. I have been following these guides: https://docs.nats.io/running-a-nats-service/nats-kubernetes and https://dev.to/karanpratapsingh/nats-at-edge-with-k3s-5377. In both cases, I have been using helm to install and run Nats service.
Problem: Nats times out whenever I am trying to run tests inside deployment/nats-k3s-box, yielding the following error: "nats: error: dial tcp: lookup my-nats: i/o timeout". I have tried both methods to try to test if the Nats service, which yield the following results.
kubectl exec -n default -it deployment/my-nats-box -- /bin/sh -l
_ _
_ __ __ _| |_ ___ | |__ _____ __
| '_ \ / _` | __/ __|_____| '_ \ / _ \ \/ /
| | | | (_| | |_\__ \_____| |_) | (_) > <
|_| |_|\__,_|\__|___/ |_.__/ \___/_/\_\
nats-box v0.13.0
my-nats-box-b864cbd4c-46q2z:~# nats sub test&
my-nats-box-b864cbd4c-46q2z:~# nats sub test &
my-nats-box-b864cbd4c-46q2z:~# nats pub test hi
nats: error: dial tcp: lookup my-nats: i/o timeout
nats: error: dial tcp: lookup my-nats: i/o timeout
nats: error: dial tcp: lookup my-nats: i/o timeout
[2]+ Done(1) nats sub test
[1]+ Done(1) nats sub test
my-nats-box-b864cbd4c-46q2z:~#
my-nats-box-b864cbd4c-46q2z:~# nats: error: dial tcp: lookup my-nats: i/o timeout
/bin/sh: nats:: not found
my-nats-box-b864cbd4c-46q2z:~# nats bench test --msgs=10000000 --pub 5 --sub 2
14:50:13 Starting Core NATS pub/sub benchmark [subject=test, multisubject=false, multisubjectmax=0, msgs=10,000,000, msgsize=128 B, pubs=5, subs=2, pubsleep=0s, subsleep=0s]
nats: error: nats connection 0 failed: dial tcp: lookup my-nats: i/o timeout
my-nats-box-b864cbd4c-46q2z:~#
Kubectl get all looks like this:
kubectl get all
NAME READY STATUS RESTARTS AGE
pod/my-nats-0 3/3 Running 0 2m47s
pod/my-nats-box-b864cbd4c-46q2z 1/1 Running 0 2m47s
pod/server-pod 1/1 Running 0 6d15h
pod/server-pods-veibox-7489b7bb94-8mv7d 1/1 Running 0 3d1h
pod/server-pods-veibox-7489b7bb94-k949q 1/1 Running 0 3d1h
pod/server-pods-veibox-7489b7bb94-rk4ts 1/1 Running 0 3d1h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 51d
service/my-nats ClusterIP None <none> 4222/TCP,6222/TCP,8222/TCP,7777/TCP,7422/TCP,7522/TCP 2m47s
service/server-pods-veibox ClusterIP 10.43.214.43 <none> 80/TCP 3d1h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/my-nats-box 1/1 1 1 2m47s
deployment.apps/server-pods-veibox 3/3 3 3 3d1h
NAME DESIRED CURRENT READY AGE
replicaset.apps/my-nats-box-b864cbd4c 1 1 1 2m47s
replicaset.apps/server-pods-veibox-7489b7bb94 3 3 3 3d1h
NAME READY AGE
statefulset.apps/my-nats 1/1 2m47s
And the logs also don't seem to be giving me any problems
kubectl logs service/my-nats
Defaulted container "nats" out of: nats, reloader, metrics
[7] 2022/09/26 14:47:58.528353 [INF] Starting nats-server
[7] 2022/09/26 14:47:58.528624 [INF] Version: 2.9.0
[7] 2022/09/26 14:47:58.528633 [INF] Git: [517d9b3]
[7] 2022/09/26 14:47:58.528640 [INF] Name: my-nats-0
[7] 2022/09/26 14:47:58.528647 [INF] ID: NC2QJKIVXAW3HAJA6DI6QXXAN72P6ZYEZRUNM5PHDOTSSQZS753DJBYJ
[7] 2022/09/26 14:47:58.528683 [INF] Using configuration file: /etc/nats-config/nats.conf
[7] 2022/09/26 14:47:58.530977 [INF] Starting http monitor on 0.0.0.0:8222
[7] 2022/09/26 14:47:58.531242 [INF] Listening for client connections on 0.0.0.0:4222
[7] 2022/09/26 14:47:58.531880 [INF] Server is ready
What have I been doing wrong with my installation of Nats, and how would I go about fixing it?
telnet? I'm having very similar problem - my application is usually failing (but sometimes succeeding) in connecting to the NATS cluster, and I can't identify the reason. But when Itelnetinto the NATS cluster from a debug pod I always succeed. I'm following docs.nats.io/reference/reference-protocols/nats-protocol-demo. Interestingly enough,telnetworks withdemo.nats.ioand with the k8s service IP, but not with the k8s service DNS name, and I don't have other DNS issues with other k8s applications in the cluster.