cqlsh cannot connect to some nodes in the cluster

Question

cqlsh connection shows below error for couple of machine, but it is successful in other machines in cluster. Bad nodes were removed out of cluster, system_auth replication is 5 for both dc's with NetworkTopologyStrategy. Anything I'm missing to check here ?

Connection error: ('Unable to connect to any servers', {'0.0.0.0': AuthenticationFailed('Failed to authenticate to 0.0.0.0: Error from server: code=0100 [Bad credentials] message="Unable to perform authentication: Operation timed out - received only 0 responses."',)})  

./cqlsh --debug --ssl
Using CQL driver: <module 'cassandra' from '/usr/local/akamai/dart_apache_cassandra/bin/../lib/cassandra-driver-internal-only-3.11.0-bb96859b.zip/cassandra-driver-3.11.0-bb96859b/cassandra/__init__.py'>
Using connect timeout: 5 seconds
Using 'utf-8' encoding
Using ssl: True
Connection error: ('Unable to connect to any servers', {'0.0.0.0': AuthenticationFailed('Failed to authenticate to 0.0.0.0: Error from server: code=0100 [Bad credentials] message="Unable to perform authentication: Operation timed out - received only 0 responses."',)})

1. These nodes consistently produce this error when attempting to connect via cqlsh?
2. Output of nodetool status look normal
3. System log show below errors, all nodes are in UN status, describecluster show all nodes are in one schema version

WARN  [OptionalTasks:1] 2024-02-15 14:48:59,872 CassandraRoleManager.java:379 - CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  [OptionalTasks:1] 2024-02-15 14:48:59,872 CassandraRoleManager.java:418 - Setup task failed with error, rescheduling

Answer 1

I'll say in advance that there isn't enough details in your post so our ability to respond is going to be handicapped.

I noted that you tagged your post with both cassandra-3.0 and cassandra-2.1 so I need to point out that if you're building a cluster, it won't work. Nodes with different versions won't be able to form a cluster because you cannot perform any operation that requires streaming such as bootstrapping in mixed-versions.

It would explain why you are seeing this warning:

WARN  [OptionalTasks:1] ... CassandraRoleManager.java:379 - \
  CassandraRoleManager skipped default role setup: some nodes were not ready

However, you pointed out that nodetool describecluster shows that all nodes have schema agreement so it contradicts the mixed-versions hypothesis.

The warning however indicates that some of the nodes were not available so Cassandra is unable to provision the default cassandra superuser. This is supported by the error message returned by cqlsh:

Unable to perform authentication: Operation timed out - received only 0 responses.

Additionally, it seems like you're connecting to 0.0.0.0. I should point out that it is incorrect to set listen_address to that IP. At the very least, you should configure listen_address to the private IP of the node.

In any case, nodetool status may show nodes as being "up" and "normal" (UN) but the logs otherwise indicate that nodes are not able to communicate with each other so you need to review the logs for clues since that will determine your next steps. Cheers!