Kubernetes Pod Unable To Mount Secret Volume

Question

I am following the HashiCorp tutorial and it all looks fine until I try to launch the "webapp" pod - a simple pod whose only function is to demonstrate that it can start and mount a secret volume.

The error (permission denied) is shown at the bottom of this command output:

kubectl describe pod webapp
Name:             webapp
Namespace:        default
Priority:         0
Service Account:  webapp-sa
Node:             docker-desktop/192.168.65.4
Start Time:       Tue, 14 Feb 2023 09:32:07 -0500
Labels:           <none>
Annotations:      <none>
Status:           Pending
IP:
IPs:              <none>
Containers:
  webapp:
    Container ID:
    Image:          jweissig/app:0.0.1
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /mnt/secrets-store from secrets-store-inline (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5b76r (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  secrets-store-inline:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            secrets-store.csi.k8s.io
    FSType:
    ReadOnly:          true
    VolumeAttributes:      secretProviderClass=vault-database
  kube-api-access-5b76r:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age              From               Message
  ----     ------       ----             ----               -------
  Normal   Scheduled    9s               default-scheduler  Successfully assigned default/webapp to docker-desktop
  Warning  FailedMount  2s (x5 over 9s)  kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/webapp, err: rpc error: code = Unknown desc = error making mount request: couldn't read secret "db-password": Error making API request.

URL: GET http://vault.default:8200/v1/secret/data/db-pass
Code: 403. Errors:

* 1 error occurred:
  * permission denied

I am executing this on a Win11 machine with DockerDesktop and Kubernetes running under Docker. (I am fairly new to K8s.) My K8s installation looks OK, since I am able to run other examples.

What permission do I need to grant here? Perhaps Vault needs to allow the REST call below?

curl http://localhost:8200/v1/secret/data/db-pass
curl: (7) Failed to connect to localhost port 8200: Connection refused

Vault is running in k8s on this machine, and in its logs says:

2023-02-14 09:07:14 You may need to set the following environment variables:
2023-02-14 09:07:14     $ export VAULT_ADDR='http://[::]:8200'

So it does indeed expose port 8200 - but it probably wants some authentication for the REST call?

UPDATE: looks like the problems are at a lower level, involving Vault:

kubectl get pods
NAME                                    READY   STATUS    RESTARTS      AGE
vault-0                                 1/1     Running   1 (22m ago)   32m
vault-agent-injector-77fd4cb69f-mf66p   1/1     Running   1 (22m ago)   32m

but

vault status
Error checking seal status: Get "http://[::]:8200/v1/sys/seal-status": dial tcp [::]:8200: connect: connection refused

So if it does not respond to vault status, it probably will not accept any other command / connection.

How can I troubleshoot this?

Answer 1

1. As pertains to the tutorial, it works. Not sure what I was doing wrong, but I ran it all again and it worked. If I had to guess, I would suspect that some of the YAML involved in configuring the pods got malformed (since white space is significant).
2. The vault status command works, but only from a terminal running inside the Vault pod. The Kubernetes-in-Docker-on-DockerDesktop cluster does not expose any ports for these pods, so even though I have vault-cli installed on my PC, I cannot use vault status from outside the pods.