Missing isci_tcp module ubuntu 23.10 on arm64

Question

On four raspberry PI 4 (arm64), I have Ubuntu 23.10 installed. For Longhorn storage on Kubernetes, I need iscsi_tcp module. Using kernel 6.5.0-1007-raspi (#9). I have installed linux-modules-extra-raspi, and I have tried installing open-iscsi (says already installed).

I've been trying steps from this question/answer without success.

I've been trying different combinations of installing/purging linux-modules-extra-raspi, linux-modules-extra-6.5.0-1007-raspi, and open-isci on the nodes, trying to get the isci_tcp module installed still to no avail.

Occasionally, I can get to a point where lsmod iscsi_tcp shows that it exists on a nodes (but later, after trying things, it does not appear). In all cases, at all times modprobe iscsi_tcp shows: modprobe: FATAL: Module iscsi_tcp not found in directory /lib/modules/6.5.0-1007-raspi

Also, when I install a generic version of the extras, I see two entries:

ii  linux-modules-extra-6.5.0-1007-raspi 6.5.0-1007.9                            arm64        Linux kernel extra modules for version 6.5.0 on ARMv8 SMP
ii  linux-modules-extra-raspi            6.5.0.1007.8                            arm64        Raspberry Pi Linux kernel extra modules

If I install the specific linux-modules-extra-6.5.0-1007-raspi, I see just the top entry.

I did a package search for "iscsi-top" (https://packages.ubuntu.com/search?suite=mantic&arch=arm64&mode=exactfilename&searchon=contents&keywords=iscsi_tcp) and for 23.10, I see these (partial list):

/usr/src/linux-headers-6.5.0-1004-laptop/include/config/ISCSI_TCP   linux-headers-6.5.0-1004-laptop
/usr/src/linux-headers-6.5.0-1005-raspi/include/config/ISCSI_TCP    linux-headers-6.5.0-1005-raspi
/usr/src/linux-headers-6.5.0-1007-azure/include/config/ISCSI_TCP    linux-headers-6.5.0-1007-azure
/usr/src/linux-headers-6.5.0-1007-gcp/include/config/ISCSI_TCP  linux-headers-6.5.0-1007-gcp

I've got a bunch of questions...

1. The top entry of linux-modules-extra-6.5.0-1007-raspi (6.5.0-1007.9) says ARMv8 SMP. Is that the correct module for Raspberry PI arm64?
2. In the package list, it seems like raspi version is only on 6.5.0-1005. Does that mean it is not in 6.5.0-1007 and I need to use an older kernel?
3. Am I missing some step here in trying to get the needed module installed?
4. If I have to downgrade the kernel, can anyone point me to instructions on how to do that?

Thanks!

Answer 1

The issue I had with Longhorn storage was that the volume was not healthy (stuck in attaching). The pod for longhorn was showing an error that indicated iscsci_tcp module was not found. I looked at the troubleshooting page and tried some of the steps to install, which is when I posted this issue.

They mentioned a check script (https://longhorn.io/docs/1.5.3/deploy/install/#using-the-environment-check-script). Running that, it did not complain about iscsid (even though I cannot see iscsi_tcp module), but it did complain about multipathd running on nodes. There was some mention that this could interfere with iscsid.

I stumbled on this thread and one part of it was that someone did these commands:

sudo apt install nfs-common
sudo systemctl enable iscsid
sudo systemctl start iscsid
sudo systemctl stop multipathd
sudo systemctl disable multipathd

I did these on each node, ran the check script and it passed.

There was more suggestions in the longhorn documentation (sorry it is for kubernetes), to install open-iscsi:

kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.5.3/deploy/prerequisite/longhorn-iscsi-installation.yaml

However, looking at the logs it just said that it was already installed. I didn't see any change in availability of iscsi_tcp module either. I continued setting up longhorn and now the volume is healthy!

Possibly the iscsi_tcp error was a red-herring and the problem was multipathd? In any case, it is working now.