Nomad secrets folder empty in Docker container

Question

I'm facing an issue with Nomad's NOMAD_SECRETS_DIR folder allocation for a Docker container.

I've got the following template stanza in my job description:

template {
    destination = "/local/pki/certs/my_certificate.crt"
    perms = "440"
    data = <<EOF
[ CERTIFICATE CONTENT ]
EOF
}

template {
    destination = "/secrets/pki/private/my_key.key"
    perms = "400"
    data = <<EOF
[ KEY CONTENT ]
EOF
}

With that, from the UI, when I look at the files allocated to the task, I can see that the file for the secret key is created.

However, the secrets folder is completely empty in the running Docker container:

# nomad alloc exec -i -t -task lb0 52a13d48 /bin/sh
/ # ls -al /secrets
total 0
drwxr-xr-x. 2 root   root    6 Feb 11 11:31 .
drwxr-xr-x. 1 root   root   70 Feb 11 11:31 ..

And if I look at the allocation folder on the system itself, the folder is also empty:

# ls -al /srv/nomad/alloc/52a13d48-b073-39b4-4b79-10225aece50e/lb0/secrets/
total 0
drwxr-xr-x. 2 root   root    6 Feb 11 12:31 .
drwxrwxrwx. 5 nobody nobody 45 Feb 11 12:31 ..

Where is Nomad finding the files in the secrets folder? Why is it empty in the container?

Answer 1

Strangely, Nomad does not really like when I mount the filesystem read-only.

I had put the following into the systemd unit file:

ProtectSystem=full
ProtectHome=read-only

Which remount some system folders read-only before launching the process. This was interfering with Nomad in a way.