ESM 401 Unauthorized error on Ubuntu 16.04

Question

I want to upgrade my Ubuntu 16.04 server to Ubuntu 18.04 and I'm running the following commands to do so;

apt update -y
apt upgrade -y
do-release-upgrade

apt update command runs fine with the following output;

# apt update -y
Hit:1 https://esm.ubuntu.com/infra/ubuntu bionic-infra-security InRelease
Hit:2 https://esm.ubuntu.com/infra/ubuntu bionic-infra-updates InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
5 packages can be upgraded. Run 'apt list --upgradable' to see them.

However when running the apt upgrade command, it returns the following errors;

# apt upgrade -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done

*The following packages could receive security updates with UA Infra: ESM service enabled:
  libkrb5-3 libgssapi-krb5-2 libk5crypto3 libkrb5support0 libzstd1
Learn more about UA Infra: ESM service for Ubuntu 16.04 at https://ubuntu.com/16-04

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

The following packages have been kept back:
  libk5crypto3 libkrb5support0
The following packages will be upgraded:
  libzstd1
1 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
1 esm-infra security update
Need to get 189 kB of archives.
After this operation, 132 kB of additional disk space will be used.
Err:1 https://esm.ubuntu.com/infra/ubuntu bionic-infra-security/main amd64 libzstd1 amd64 1.3.3+dfsg-2ubuntu1+esm1
  401  Unauthorized
E: Failed to fetch https://esm.ubuntu.com/infra/ubuntu/pool/main/libz/libzstd/libzstd1_1.3.3+dfsg-2ubuntu1+esm1_amd64.deb  401  Unauthorized

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

I'm not sure what to do at this point. How can I solve this?

Answer 1

Two ways exist depending on currently running Ubuntu version.

(a) Ubuntu 16.04 LTS as currently running version

You have to backup your sources.list by

sudo mv /etc/apt/sources.list ~/
sudo mv /etc/apt/sources.list.d/*.list ~/

and then fill main sources.list with correct URLs using one of commands below:

• plain sources.list replace

  cat <<EOF | sudo tee /etc/apt/sources.list
  deb http://archive.ubuntu.com/ubuntu/ xenial-backports main universe multiverse restricted
  deb http://archive.ubuntu.com/ubuntu/ xenial main universe multiverse restricted
  deb http://archive.ubuntu.com/ubuntu/ xenial-updates main universe multiverse restricted
  deb http://security.ubuntu.com/ubuntu/ xenial-security main universe multiverse restricted
  EOF
  

• using add-apt-repository

  sudo add-apt-repository "deb http://archive.ubuntu.com/ubuntu/ xenial-backports main universe multiverse restricted"
  sudo add-apt-repository "deb http://archive.ubuntu.com/ubuntu/ xenial main universe multiverse restricted"
  sudo add-apt-repository "deb http://archive.ubuntu.com/ubuntu/ xenial-updates main universe multiverse restricted"
  sudo add-apt-repository "deb http://security.ubuntu.com/ubuntu/ xenial-security main universe multiverse restricted"
  

and then resume upgrade to 18.04 LTS:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get purge ubuntu-advantage-tools --autoremove
sudo rm /etc/apt/sources.list.d/ubuntu-esm-infra.list
sudo do-release-upgrade

Then check ~/*.list files from previous release and partially move the lines to /etc/apt/sources.list.

(b) Ubuntu 18.04 LTS as currently running version

You have to backup your sources.list by

sudo mv /etc/apt/sources.list ~/
sudo mv /etc/apt/sources.list.d/*.list ~/

and then fill main sources.list with correct URLs using one of commands below:

• plain sources.list replace

  cat <<EOF | sudo tee /etc/apt/sources.list
  deb http://archive.ubuntu.com/ubuntu/ bionic-backports main universe multiverse restricted
  deb http://archive.ubuntu.com/ubuntu/ bionic main universe multiverse restricted
  deb http://archive.ubuntu.com/ubuntu/ bionic-updates main universe multiverse restricted
  deb http://security.ubuntu.com/ubuntu/ bionic-security main universe multiverse restricted
  EOF
  

• using add-apt-repository

  sudo add-apt-repository "deb http://archive.ubuntu.com/ubuntu/ bionic-backports main universe multiverse restricted"
  sudo add-apt-repository "deb http://archive.ubuntu.com/ubuntu/ bionic main universe multiverse restricted"
  sudo add-apt-repository "deb http://archive.ubuntu.com/ubuntu/ bionic-updates main universe multiverse restricted"
  sudo add-apt-repository "deb http://security.ubuntu.com/ubuntu/ bionic-security main universe multiverse restricted"
  

and then install all necessary upgrades by:

sudo apt-get purge ubuntu-advantage-tools --autoremove
sudo rm /etc/apt/sources.list.d/ubuntu-esm-infra.list

sudo apt-get update
sudo apt-get upgrade

Then check ~/*.list files from previous release and partially move the lines to /etc/apt/sources.list while replacing xenial with bionic.

Answer 2

On an 18.04 system (at least), look for a /etc/apt/sources.list.d/ubuntu-esm-infra.list file. That file erroneously lists sources for ESM updates that you would need to pay for, and apt is dutifully trying to download them.

I disabled those sources by putting a # before each of the two deb lines. The file now looks like this:

# Written by ubuntu-advantage-tools
#deb https://esm.ubuntu.com/infra/ubuntu bionic-infra-security main
# deb-src https://esm.ubuntu.com/infra/ubuntu xenial-infra-security main

#deb https://esm.ubuntu.com/infra/ubuntu bionic-infra-updates main
# deb-src https://esm.ubuntu.com/infra/ubuntu xenial-infra-updates main

I think deleting the file would have worked, too.

Now, when I update the system again, using my usual update command:

$ sudo apt update && sudo apt autoremove && sudo apt upgrade

...the apt upgrade portion doesn't try to download those unauthorized sources, and the upgrade is able to finish!

Answer 3

When you register a system with Ubuntu Pro a /etc/apt/auth.conf.d/90ubuntu-advantage file is created. This file contains credentials tied to your Ubuntu account which gives that system permission to retrieve ESM updates.

I registered a system and initially it had no issues, but after some weeks attempts to retrieve packages from https://esm.ubuntu.com/ gave a HTTP 401 "Unauthorized" error.

I resolved the issue by detaching and reattaching the machine using the current Token value from https://ubuntu.com/pro/dashboard.

This regenerated the /etc/apt/auth.conf.d/90ubuntu-advantage file with current credentials.

1. sudo pro detach
2. sudo pro attach TOKEN_VALUE_HERE
3. sudo apt-get update

At this point I could apply updates (e.g., via apt, apt-get, aptitude) as before.