Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

split base/layered versionlocked package problem (krb5-workstation) #415

Closed
cgwalters opened this issue Aug 1, 2016 · 52 comments
Closed

Comments

@cgwalters
Copy link
Member

cgwalters commented Aug 1, 2016

In wstree right now krb5-libs is a dependency of various things in the base OS. krb5-workstation is an app that uses it - however, it's version locked.

This introduces a problem with package layering, since if the compose server picks up a newer version from package mirror A, but the client only has an older version from mirror B, upgrades break. (Or if it's not installed, it can't be).

There are obviously a few solutions to this...likely krb5-workstation should be more flexible, and only introduce a hard requirement on the -libs package version if it needs it. Second, we may need to consider a way to retrieve rpm-md data from the same place as the ostree commit, or vice versa. Basically having unified metadata/data management.

--

EDIT (2019-09-08): rojig is a much bigger hammer attempting to solve this problem.

@jlebon
Copy link
Member

jlebon commented Aug 8, 2016

Second, we may need to consider a way to retrieve rpm-md data from the same place as the ostree commit, or vice versa. Basically having unified metadata/data management.

Isn't this completely up to the content provider, though? Doesn't seem like something we can do much about in rpm-ostree.

@cgwalters
Copy link
Member Author

I think if we provided the infrastructure to link to the rpm-md repos from the ostree commit, infra providers could use that. We basically have all the data already from the treecompose side. The infra provider would need to retain multiple versions of the rpm-md repo data though.

That said I think the general solution to this is to relax the strict version requirements.

@cgwalters
Copy link
Member Author

🏈 Punted on this

@cgwalters
Copy link
Member Author

cgwalters commented Apr 19, 2017

Hit this again today trying to layer virt-manager; through a depchain virt-manager-commonpython-libxml2 (not in the tree) → libxml2:

error: Could not depsolve transaction; 1 problem detected:
0. package python-libxml2-2.9.3-4.fc25.x86_64 requires libxml2 = 2.9.3-4.fc25, but none of the providers can be installed

And at the time:

$ rpm -q libxml2
libxml2-2.9.4-2.fc25.x86_64

@dustymabe
Copy link
Member

doesn't this problem go away if we can replace content in the base layer? #485

@cgwalters
Copy link
Member Author

I think we should definitely allow it, but doing that invalidates the concept that the tree is tested and run as a unit. I'm not sure I'd want it to just happen by default.

Particularly in this case, it turned out the problem is I was being served a very out of date (rpm-md) mirror.

@dustymabe
Copy link
Member

I think we should definitely allow it, but doing that invalidates the concept that the tree is tested and run as a unit. I'm not sure I'd want it to just happen by default.

right, but don't we invalidate that when we allow package layering at all? I know technically we are just "adding to" rather than "changing" but adding things breaks things sometimes too.

Maybe for base package overrides we default to spitting out a message about the fact that this transaction will override a base package and thus the integrity of the tree may suffer and then allow the user to (y/n). Also provide a cli flag for forcing this.

@cgwalters
Copy link
Member Author

Another good example of this is emacs-filesystem (pulled into base) and emacs.

@kallisti5
Copy link

vim-atom

Seeing this here as well with vim... this is a painful bug.

@miabbott
Copy link
Member

Like @kallisti5 above, I ran into the problem with vim-common. (Going to paste my session here for searchability later)

$ sudo rpm-ostree status                                                                                                                                                                     
[sudo] password for miabbott:
State: idle                                         
Deployments:                                                                                             
● atomicws:fedora/x86_64/workstation                                                                     
                   Version: 26.201 (2017-10-06 17:57:57)                                                                                                                                                           
                BaseCommit: c458ed1025d135b88f611eb169c515fd478d8751bcd742cb05284a929ab07a56
           LayeredPackages: chrome-gnome-shell ffmpeg-libs krb5-workstation libvirt-client sshpass tmux vagrant-libvirt vim virt-install virt-manager
                                                                                                                                                                                                                   
  atomicws:fedora/x86_64/workstation                                                                     
                   Version: 26.187 (2017-10-02 09:58:23)
                BaseCommit: 5637588b7331a5b49f4c417e4f5eca706f067d9b47911d5849fe9981ec30c3c9
           LayeredPackages: chrome-gnome-shell ffmpeg-libs krb5-workstation libvirt-client tmux vagrant-libvirt vim virt-manager                                                                                   
[/var/home/miabbott *]$ sudo rpm-ostree upgrade     

Updating metadata for 'updates': [==========================] 100%
rpm-md repo 'updates'; generated: 2017-10-09 14:42:40                                                                                                                                                              
                                                                                                         

Updating metadata for 'fedora': [==========================] 100%
rpm-md repo 'fedora'; generated: 2017-07-05 20:31:38                                                                                                                                                               
                                                                                                         

Updating metadata for 'rpmfusion-nonfree': [==================================] 100%
rpm-md repo 'rpmfusion-nonfree'; generated: 2017-07-10 12:08:07                                                                                                                                                    
                                                                                                         

Updating metadata for 'region51-chrome-gnome-shell': [============================================] 100%
rpm-md repo 'region51-chrome-gnome-shell'; generated: 2017-10-10 07:03:04                                                                                                                                          
                                                                                                         

Updating metadata for 'walters-syslinux-stub-i686': [========================================] 100%
rpm-md repo 'walters-syslinux-stub-i686'; generated: 2017-10-10 09:05:48                                                                                                                                           
                                                                                                         

Updating metadata for 'walters-syslinux-stub': [==================================== 100%
rpm-md repo 'walters-syslinux-stub'; generated: 2017-10-10 09:05:46                                                                                                                                                
                                                                                                         

Updating metadata for 'rpmfusion-nonfree-updates': [=========================================] 100%
rpm-md repo 'rpmfusion-nonfree-updates'; generated: 2017-10-06 17:41:43                                                                                                                                            
                                                                                                         

Updating metadata for 'rpmfusion-free': [===============================] 100%
rpm-md repo 'rpmfusion-free'; generated: 2017-07-10 12:06:27                                                                                                                                                       
                                                    
                                                    
Updating metadata for 'rpmfusion-free-updates': [======================================] 100%

                                                                                                                                                                                                                   
Importing metadata [======================] 100%
Resolving dependencies... done                      
Will download: 2 packages (196.2 kB)

  Downloading from updates: [========================] 100%

Importing: [=======================] 100%
Applying 134 overlays... error: Checking out vim-common-2:8.0.1171-1.fc26.x86_64: Hardlinking c0/679f8a0eefe6ddb91b60afe7ff885aa8ee2b39fea8a0dbf342d7758733a7d8.file to ex.1.gz: File exists

@jlebon
Copy link
Member

jlebon commented Oct 10, 2017

Right, this is the same issue though the output is different because v2017.9 has #974. A hacky workaround is to manually download the matching vim packages (https://koji.fedoraproject.org/koji/buildinfo?buildID=969318) and locally install that. Hacky because on the next upgrade that bumps vim-minimal again, you'll have to uninstall them and possibly get the latest RPMs if they're still not in sync (though if the tree is freshly composed, that shouldn't be an issue).

@jlebon
Copy link
Member

jlebon commented Oct 10, 2017

@miabbott, I think you're hitting something different. I opened a separate issue for your error here: #1047.

@cgwalters
Copy link
Member Author

This problem will be comprehensively fixed by jigdo ♲📦 #1081 because the rpm-md and ostree commit will move at the same cadence by default.

@dustymabe
Copy link
Member

This problem will be comprehensively fixed by jigdo ♲📦 #1081 because the rpm-md and ostree commit will move at the same cadence by default.

Can you explain this a little more? if yesterday's bodhi run had a new fedora-atomic-host rpm in it (i.e. a new release), and today's run includes new packages (and new base packages that are deps) then won't an rpm-ostree install still fail in the same way if a base package is part of the update?

@cgwalters
Copy link
Member Author

The jigdoRPM should be generated from the same rpm-md reposet as contains it. That's how it works today in FAHC.

@cgwalters
Copy link
Member Author

cgwalters commented Feb 1, 2018

Yes, this means you need to do: createrepo_c, compose tree, copy in jigdoRPM, createrepo_c.

@jlebon
Copy link
Member

jlebon commented Feb 1, 2018

But that still means you need to update to the new jigdoRPM first, right? (Which also assumes that there's a new jigdoRPM to update to). Just like OSTree mode right now?

Jigdo does improve the situation in the sense that, at the moment that the jigdoRPM is released, it is in sync with the rest of the RPMs, which is not a guarantee right now. But otherwise, this split issue is still something people can hit, right?

@dustymabe
Copy link
Member

Yes, this means you need to do: createrepo_c, compose tree, copy in jigdoRPM, createrepo_c.

yeah I figured that much. Specifically for this issue, though, tomorrows bodhi run fires and no new fedora-atomic-host rpm is in that, now there are newer rpms in the rpm-md than what was used to build the fedora-atomic-host rpm and we have this same problem all over again?

@cgwalters
Copy link
Member Author

But that still means you need to update to the new jigdoRPM first, right?

Yes, but we do that by default on upgrade right?

Specifically for this issue, though, tomorrows bodhi run fires and no new fedora-atomic-host rpm is in that, now there are newer rpms in the rpm-md than what was used to build the fedora-atomic-host rpm and we have this same problem all over again?

I don't quite understand that. Again the design is that the jigdoRPM is contained within and versioned by the rpm-md repo. The jigdoRPM should be regenerated whenever the RPM content changes. However what you may be getting at here is that today the compose tree process doesn't yet support "pure jigdo" mode; the canonical history is still stored in an ostree repo.

@jlebon
Copy link
Member

jlebon commented Feb 1, 2018

The jigdoRPM should be regenerated whenever the RPM content changes.

Right, assuming that all Bodhi updates are batched and each batch has a jigdoRPM, then there's no issue. But e.g. right now for FAH, we release every two weeks. Whereas Bodhi batches are every week, right? So there's potentially a one week window during which we can get mismatches. Isn't that the same issue we're hitting right now with the stable vs updates ref?

@dustymabe
Copy link
Member

I don't quite understand that.

Let me explain. Today we have the stable ref fedora/27/x86_64/atomic-host and the updates ref that gets updated every day fedora/27/x86_64/updates/atomic-host. In jigdo future we have the stable jigdoRPM fedora-atomic-host and an updates jigdoRPM fedora-atomic-host-updates. If we release the stable fedora-atomic-host jigdoRPM today then the rpm-md it was generated from will be out of date tomorrow, just like the stable ostree ref that we use today fedora/27/x86_64/atomic-host.

@dustymabe
Copy link
Member

Isn't that the same issue we're hitting right now with the stable vs updates ref?

yes, i think so

@cgwalters
Copy link
Member Author

But e.g. right now for FAH, we release every two weeks. Whereas Bodhi batches are every week, right?

Yes; making full use of jigdo requires that we synchronize those in actuality.

@cgwalters
Copy link
Member Author

I see two paths:

  1. We don't differ from Bodhi at all - this may mean we have an ostree update 3 days in a row if one of the RPMs changes each of those days
  2. We create /etc/yum.repos.d/fedora-atomic.repo

@jlebon
Copy link
Member

jlebon commented Feb 1, 2018

Alternatively, we could also add a kind of --allow-replacements switch so we automatically add replace overrides.

@cgwalters
Copy link
Member Author

Alternatively, we could also add a kind of --allow-replacements switch so we automatically add replace overrides.

Yeah true, but I really like the protection we have today against the depsolver deciding to swap out something from the base by default. By default we shouldn't offer depsolve errors and a magic switch "make it work" right?

@dustymabe
Copy link
Member

Ok, just so i'm clear on everything.

  • this is a problem with jigdo AND ostree transports (i.e. jigdo doesn't solve this problem)
  • the most desirable solution is actually in the releng side to make sure rpm repos sync with when we do releases
  • an alternative solution is to allow forcing overrides in the base via a command line switch

Some more responses:

We don't differ from Bodhi at all - this may mean we have an ostree update 3 days in a row if one of the RPMs changes each of those days

This takes us back to where we were before we 'slowed down' the cadence. Not saying that is a bad thing, but it is different.

We create /etc/yum.repos.d/fedora-atomic.repo

The problem with this is we don't get all of the rpms from the vast ecosystem that is Fedora. Is there a solution to this that would allow us to keep pulling rpms from the Everything package set but still have a subset of rpms in a separate repo?

@dustymabe
Copy link
Member

Not to mention that when e.g. enabling testing repositories, users currently need to understand how to simutaneously enable the updates-testingrepostiory and rebase to a testing ref.

I could imagine teaching rpm-ostree and the server side enough intelligence to "tie together" these things - something like each ref having metadata about which rpm-md repos generated it.

yeah that would be nice

or we could give them a Installing this RPM would require updateing the base OSTree. Continue? (y/N)

In general, I am not a fan of these sorts of dynamic interactive questions - I think they're very often indicative of bad design.

See also https://ometer.com/preferences.html

That doesn't mean we shouldn't have any questions like this, but in this specific case, I think we should take the "hard path" (rojig) and save our users confusing questions.

(Another important thing here is that any questions like this would have to be reimplemented in gnome-software and Cockpit)

Then we just make it not an option and upgrade the user by default just like in rojig. Solves the same problem. Then all that's left is the updates vs updates-testing problem mentioned above.

@dustymabe
Copy link
Member

Then we just make it not an option and upgrade the user by default just like in rojig.

although I think this would render rpm-ostree install && rpm-ostree ex livefs useless because we would always be updating the base and requiring a reboot, right?

@edwintorok
Copy link

edwintorok commented Dec 27, 2018

Edit: this actually seems to be a problem with the repository itself, libxcrypt-devel-4.4.2-3.fc29 is claimed to be not available at all, although it is available on koji: #1724

It would be useful if rpm-ostree printed more details on which package is causing the problem, I just ran into this problem today with a very generic Some base packages would be replaced:

sudo rpm-ostree override reset fuse-overlayfs
sudo rpm-ostree upgrade
note: automatic updates (stage) are enabled
1 metadata, 0 content objects fetched; 569 B transferred in 7 seconds
Checking out tree a6d00a9... done
Enabled rpm-md repositories: fedora updates
Updating metadata for 'fedora'... done
rpm-md repo 'fedora'; generated: 2018-10-24T22:20:15Z
Updating metadata for 'updates'... done
rpm-md repo 'updates'; generated: 2018-12-26T02:39:50Z
Importing rpm-md... done
⠁ Resolving dependencies... 
Forbidden base package replacements:
Resolving dependencies... done
error: Some base packages would be replaced

I tried using --uninstall, but it still complained:

sudo rpm-ostree upgrade --uninstall ansible bcc-tools chromium dconf-editor evince evolution fedora-toolbox firefox-wayland gmp-devel gnome-photos gnome-tweak-tool kernel-tools latencytop libreoffice mozilla-ublock-origin ncurses-term neovim ocaml ocamldoc opam pass perf perl-Pod-Html python3-psutil ripgrep stow strace tig weston zsh
[sudo] password for edwin: 
note: automatic updates (stage) are enabled
1 metadata, 0 content objects fetched; 569 B transferred in 5 seconds
Checking out tree a6d00a9... done
Enabled rpm-md repositories: fedora updates
rpm-md repo 'fedora' (cached); generated: 2018-10-24T22:20:15Z
rpm-md repo 'updates' (cached); generated: 2018-12-26T02:39:50Z
Importing rpm-md... done
⠁ Resolving dependencies... 
Forbidden base package replacements:
Resolving dependencies... done
error: Some base packages would be replaced

Finally I pinned the current deployment, uninstalled all layers, upgraded, used rpm-ostree install -n to narrow down the package (in this case ocaml):

sudo rpm-ostree uninstall --all
[...]
sudo rpm-ostree upgrade
note: automatic updates (stage) are enabled
1 metadata, 0 content objects fetched; 569 B transferred in 3 seconds
Staging deployment... done
Upgraded:
  container-selinux 2:2.76-1.git87fae85.fc29 -> 2:2.77-1.git2c57a17.fc29
  libxcrypt 4.4.2-1.fc29 -> 4.4.2-3.fc29
  runc 2:1.0.0-59.dev.gitccb5efd.fc29 -> 2:1.0.0-66.dev.gitbbb17ef.fc29
  zchunk-libs 0.9.17-1.fc29 -> 1.0.0-1.fc29
[...]
for i in bcc-tools dconf-editor evince evolution fedora-toolbox firefox-wayland gnome-photos gnome-tweak-tool kernel-tools latencytop mozilla-ublock-origin ncurses-term neovim ocaml ocamldoc opam perf ripgrep stow strace tig weston zsh; do echo $i; sudo rpm-ostree install -n $i; done
[...]
sudo rpm-ostree install ocaml
Checking out tree a6d00a9... done
Enabled rpm-md repositories: fedora updates
rpm-md repo 'fedora' (cached); generated: 2018-10-24T22:20:15Z
rpm-md repo 'updates' (cached); generated: 2018-12-26T02:39:50Z
Importing rpm-md... done
⠁ Resolving dependencies... 
Forbidden base package replacements:
Resolving dependencies... done
error: Some base packages would be replaced

However it still doesn't show which package in the base packages would be the problem...

Here is my current status:

sudo rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
  ostree://fedora-workstation:fedora/29/x86_64/silverblue
                   Version: 29.20181227.0 (2018-12-27T01:09:18Z)
                BaseCommit: a6d00a98f2bfb16e3cdae4f89f306a5035c7e67ae15012ccf3d757c11c4b8624
              GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
           LayeredPackages: ansible bcc-tools chromium dconf-editor evince evolution fedora-toolbox firefox-wayland gmp-devel gnome-photos gnome-tweak-tool kernel-tools
                            latencytop libreoffice mozilla-ublock-origin ncurses-term neovim pass perf perl-Pod-Html python3-psutil ripgrep stow strace tig weston zsh

● ostree://fedora-workstation:fedora/29/x86_64/silverblue
                   Version: 29.20181225.0 (2018-12-25T01:13:26Z)
                BaseCommit: 9c2273bb044c6b0a36c0e1549011091afd79f95700f27a568f5964ac12c7eb32
              GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
      ReplacedBasePackages: fuse-overlayfs 0.1-6.dev.git3d48bf9.fc29 -> 0.1-8.dev.git91bb401.fc29
           LayeredPackages: ansible bcc-tools chromium dconf-editor evince evolution fedora-toolbox firefox-wayland gmp-devel gnome-photos gnome-tweak-tool kernel-tools
                            latencytop libreoffice mozilla-ublock-origin ncurses-term neovim ocaml ocamldoc opam pass perf perl-Pod-Html python3-psutil ripgrep stow strace
                            tig weston zsh
                    Pinned: yes

  ostree://fedora-workstation:fedora/29/x86_64/silverblue
                   Version: 29.20181224.0 (2018-12-24T01:17:14Z)
                BaseCommit: a071235dd160f942385a1c2472521ef1906e066b26083e69fce734e1ac71a803
              GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
      ReplacedBasePackages: fuse-overlayfs 0.1-6.dev.git3d48bf9.fc29 -> 0.1-8.dev.git91bb401.fc29
           LayeredPackages: ansible bcc-tools chromium dconf-editor evince evolution fedora-toolbox firefox-wayland gmp-devel gnome-photos gnome-tweak-tool kernel-tools
                            latencytop libreoffice mozilla-ublock-origin ncurses-term neovim ocaml ocamldoc opam pass perf perl-Pod-Html python3-psutil ripgrep stow strace
                            tig weston zsh

AvailableUpdate:
        Version: 29.20181227.0 (2018-12-27T01:09:18Z)
         Commit: a6d00a98f2bfb16e3cdae4f89f306a5035c7e67ae15012ccf3d757c11c4b8624
   GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
           Diff: 4 upgraded, 232 removed

Edit: in this case the problem was libxcrypt-devel (needed by glibc-devel, needed by gcc, needed by ocaml-runtime, needed by ocaml):

sudo rpm-ostree install -n libxcrypt
error: "libxcrypt" is already provided by: libxcrypt-4.4.2-3.fc29.x86_64. Use --allow-inactive to explicitly require it.
edwin@bolt:~ % sudo rpm-ostree install -n libxcrypt-devel
Checking out tree a6d00a9... done
Enabled rpm-md repositories: fedora updates
rpm-md repo 'fedora' (cached); generated: 2018-10-24T22:20:15Z
rpm-md repo 'updates' (cached); generated: 2018-12-26T02:39:50Z
Importing rpm-md... done
⠁ Resolving dependencies... 
Forbidden base package replacements:
Resolving dependencies... done
error: Some base packages would be replaced

Note that rpm-ostree refresh-md and rpm-ostree cleanup -m didn't fix the problem, libxcrypt-devel is still uninstallable.

Copy link

I'm seeing this when trying to install gcc on Atomic host 29.20190219.0:

Forbidden base package replacements:
  libgcc 8.2.1-6.fc29 -> 8.3.1-2.fc29 (updates)
  libxcrypt 4.4.3-2.fc29 -> 4.4.3-10.fc29 (updates)
  libgomp 8.2.1-6.fc29 -> 8.3.1-2.fc29 (updates)

As soon as the update repos get out of sync from the latest base layer, installing some RPMs becomes impossible. (rpm-ostree cleanup -m doesn't help.) IMO this issue shouldn't be tagged just "enhancement", it's a significant usability bug.

Until a real fix is implemented, would it help to build Atomic host base layers with a nightly cadence?

@cgwalters
Copy link
Member Author

Until a real fix is implemented, would it help to build Atomic host base layers with a nightly cadence?

That exists; just rpm-ostree rebase fedora-atomic:fedora/29/x86_64/updates/atomic-host

(But I'd recommend doing development in a container, not the host)

Copy link

Mar 5, 2019

@cgwalters Thanks! Rebasing to the updates base layer helped.

(Re development in container i agree, we're perhaps using Atomic in a weird way here. We're trying to make a multi-user workstation without forcing everyone to work in containers all the time. We picked Atomic mainly because multiple users have admin rights on the machine, and in this situation Atomic should accumulate less bit rot over time than traditional distros.)

@dustymabe
Copy link
Member

@cgwalters Thanks! Rebasing to the updates base layer helped.

@jistr - I do want to point out that using the updates base layer (fedora/29/x86_64/updates/atomic-host) works, but it goes through less testing than the stable ref (fedora/29/x86_64/atomic-host) that only gets releases every two weeks. Just something to consider when making this decision.

@jlebon
Copy link
Member

jlebon commented Jun 9, 2020

It would be useful if rpm-ostree printed more details on which package is causing the problem, I just ran into this problem today with a very generic Some base packages would be replaced:

This will be solved by #2125.

@jlebon
Copy link
Member

jlebon commented Oct 7, 2020

Let's close this one because ultimately it's a problem solved at the distro/releng level, not something that rpm-ostree can do a whole lot about.

And anyway, we're mostly talking about Fedora here which will have this fixed very soon. See:
coreos/fedora-coreos-tracker#400
https://src.fedoraproject.org/rpms/fedora-repos/pull-request/79
coreos/fedora-coreos-config#673

(That last PR is Fedora CoreOS specific, but similar ones should follow for the other OSTree-based editions.)

@jlebon jlebon closed this as completed Oct 7, 2020
@dustymabe
Copy link
Member

Let's close this one because ultimately it's a problem solved at the distro/releng level, not something that rpm-ostree can do a whole lot about.

Well, #2125 did help a lot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants