Home_greyopenFATE - openSUSE feature tracking > #305634
Dashboard | Search | Sign up | Login

Please login or register to be able to edit or vote this feature.

Debian-like dist-upgrade live system full version upgrade

Feature state

openSUSE-11.2
Done

Description

With the 11.2 cycle, we want to offer users the ability to perform a live system upgrade in the manner of Debian's dist-upgrade.

For the purpose of this cycle, we want to support dist-upgrade from the previous version (11.1) only, as this is a sufficiently complicated problem as is.

From the user's view, the difference is between being able to update the system incrementally
within the given version or service pack running, to being ble to migrate with a system command ("zypper dup" or similar) to a higher version altogether.

In the Debian experience, the set of base distributions is not necessarily limited, but it has been Ubuntu's practice to define what starting points other than "release n-1" are allowed (for instance, all LTS versions are purported to be able to "apt dist-upgrade" to the top of the line, although I have heard of problems trying to jump two years - 6.06->8.10 - in a fell swoop in this manner :-).

In the openSUSE scope, we should aim to be able to "dup" between incremental versions, starting from 11.1 to 11.2, and later 11.x to 12.0.

User benefit:

With the introduction of the Zypper stack to SLE, we finally reached the state of a featureful (which YOU was not) and fast, reliable (which ZLM was not) update stack in the platform.

For enterprise use, some tweaks are still desirable (changelogs, rollback, ...) which we are looking at, as well as improvements on the Enterprise management front, which we are working on with our SRM colleagues.

The only really significant competitive feature we are missing at this point is the Debian/Ubuntu dist-upgrade functionality, which has a powerful psychological impact at the Enterprise level and a much more tangible impact at the small user / single user level: many with no IT department do use Ubuntu these days on the basis that "chasing" Fedora and openSUSE along the six-month upgrade cycle is too much for them, and feel they can save time on Ubuntu with the combination of dist-upgrade and the longer LTS cycle.

The rationale for pursuing this is to revoke the special status of coolness this functionality gives Ubuntu, and to terminate the negative influence that may have on our SLE sales (from the expert's personal opinion, the preference then easily spills into purchasing).

Relations

Discussion


icons/user_comment.png F. L. wrote: (5 years ago)

This is the #1 feature in the systems management scope for 11.2 - I have no doubt it will be fun :-)

icons/user_comment.png K. K. wrote: (5 years ago)

Passing to mls for technical evaluation (solver + autobuild)

icons/user_comment.png M. S. wrote: (5 years ago)

Any hint on what features are currently missing?

icons/user_comment.png K. K. wrote: (5 years ago)

From the top of my head:

How to handle

  • Library ABI changes (e.g. major gcc/g++ upgrades) ?
  • Core package changes (e.g. devs.rpm to udev) ?
  • Kernel changes (if application/deamons need a specific kernel abi, dbus comes into mind) ?
  • Failure handling (network breakdown, package update errors, ...) ?
  • Booting of the new kernel ?

icons/user_comment.png M. S. wrote: (5 years ago)

And debian does this in some way?

icons/user_comment.png R. D. wrote: (5 years ago)

The distupgrade feature differs from changing source list and doing upgrade, so some utility process could be started.
They may require some documented procedure to be followed, for "tough" changes.
But ABI change, how long since ELF/glibc was introduced? C++ ABI ought not be relevant for upgrade process, they are "just" applications and libaries to be updated.
Failure handling - you are told to back up your system first, it is "best effort" not guaranteed. The less you have installed the more likely it is to succeed.
Debian offer choice of kernels, this may "punt" the problem to a user controlled install and select from GRUB menu.
Debian have apt-cache possibility, to create central pool of packages, to decrease liklihood of network troubles, as well as huge number CD and DVD iso's.

icons/user_comment.png T. K. wrote: (5 years ago)

The glibc internal ABI changes frequently. Means, running applications will continue to use the old glibc with the old ABI, but installed are already the new plugins => running applications can crash.

icons/user_comment.png R. D. wrote: (5 years ago)

But your upgrade tools are buggy if they are built to rely on "plugins".
If you try to dist upgrade a live system within GUI, with all software running, and no breakage, you aim impossibly high.
The point is, Debian have had very positive user comments for years about this feature, despite there being many caveats and no guarantee of success.

icons/user_comment.png R. D. wrote: (5 years ago)

That wasn't clear. The upgrade tools are built so they won't suffer from changes.
It is reasonable to ask user to not be running uncessary applications during a dist upgrade.

icons/user_comment.png D. M. wrote: (5 years ago)

Thorsten is talking about general applications. Not plugins.

icons/user_comment.png C. R. wrote: (5 years ago)

ABI
is important during the upgrade process, I have seen applications crashing if you upgrade running a desktop enviroment, however we need to worry more about other stuff first IMHO.

icons/user_comment.png D. M. wrote: (5 years ago)

I think the "Debian" part of the title is missleading if what we want is just to support dup officially.

I would like to see a list from some "Debian expert" on how to Debian handles the issues Klaus described above. Appart of the network failure (included in another feature: commit) I ignore (and did not experience during my old Debian times) if Debian does anything special on dist-upgrade which give them the honour to name this feature "Debian like".

Otherwise this feature should be closed as "done", (or just waiting for the commit refactoring).

Federico, as you named the feature "Debian-like", It would be helpful to know, appart of the download-first feature, what are you missing from Debian so we can reach that point.

icons/user_comment.png R. D. wrote: (5 years ago)

Technically you may be right, but that does not solve the perceived problem for end users who want a very well tested, and documented upgrade path. 11.1 shipped with release notes without solutions for problems folk hit (eg) PAM stuff).

icons/user_comment.png D. M. wrote: (5 years ago)

Exactly, but if we don't know what specific problems are we hitting during distupgrade the feature has no real work and specific problems can be tracked as bugs.

FATE is not a place for "make this better" without concrete requirements or "make an unfalible distupgrade".

If requester, prjmgr, or pm expects anything to be done in the area, we need at least:

  • A list of problems we hit during dist upgrade (I am aware of one specific, which is download-install sequence if network goes down)
  • The scope. As you mention, asking to close every application (or service) is not much to ask. Therefore the scope has to be set, because if the scope is "distupgrade and glbc upgrade while oracle process hospital monitoring equipment transactions" then this is a very expensive feature.

icons/user_comment.png D. M. wrote: (5 years ago)

btw, can you describe in technical detail the pam upgrade problem so we can document it?

icons/user_comment.png T. K. wrote: (5 years ago)

Yes, the PAM maintainers would like to know more details, too.

icons/user_comment.png R. D. wrote: (5 years ago)

There were a number of different forum poster's hitting it, just after 11.1 released (I think due to zypper dup not YaST upgrade and therefore unreported to BugZilla). It is not something I had, because I did fresh installs. Interestingly though I have used Debian less, I have done several dist-upgrade's, simply because it is expected to work well. So it is a mistake to focus on specific tech issues alone. This problem is also about expectations and user perceptions. Announcing "support" for dist upgrade and appealing for test, so work rounds can be found, for release notes at time matters to.

icons/user_comment.png C. R. wrote: (5 years ago)

as you failed to elaborate on the problem, I will..
Some people end with broken /etc/pam.d/login they have the following line

session required pam_resmgr.so
but that module is gone, and rpm upgrade does not remove that line (probably because that file has been modified)

icons/user_comment.png J. R. wrote: (5 years ago)

Isn't that simply a packaging problem?

icons/user_comment.png K. K. wrote: (5 years ago)

In this case, it looks like a packaging problem, indeed.

However, we had more complicated upgrade tasks in the past, like dropping devs.rpm and running udev instead.

See
Removing devs.rpm for a discussion of the problem.

icons/user_comment.png P. P. wrote: (5 years ago)

Interesting discussion about that particular issue, but simple to solve. What works is keeping devs.rpm, and/or simply rpm -e --justdb it. (It's enough to do so later.)

The more complicated issues in the past where a case where the kernel changed its major version, and some networking applications could not start with the (still running) old kernel. Didn't prevent an online upgrade over the network though, and applications could keep running.

When skipping several openSUSE versions, and differences have accumulated, in can happen that an upgrade in one step is not possible. It's not possible to upgrade from 9.0 to 10.1 without first updating to 9.1, due to the kernel. These issues are quickly found by experimentation though.

Other than that, the most hurting issue during remote upgrades is the difficulty to ensure persistent device naming for NICs, because that is reimplemented / changed with each openSUSE version, the upgrade procedure is not well documented, and funny surprises are the rule, and it often simply doesn't work.

Other small hurdles are the increasing modularization of kernel, which can lead to lack of support for chipsets and storage in the initrd, package split-offs (mingetty) that lead to lack of essential components after the upgrade. It pays to make sure that all packages are installed which are in the "minimal" patter of that openSUSE version. And the required but disappeared PAM module locked us out once also...

A word to all the non-believers who think that upgrades don't work: I have upgraded dozens of systems remotely during the last decade. All my productive systems are repeatedly upgraded that way since day one. Unfortunately, the tools we have didn't help with this. Apart from using rpm, I started doing this with yum a few years ago (if it works, metadata is available, falling back to rpm if needed).

Last year I documented some steps: http://poeml.de/~poeml/10.1-11.0-update/update-11.0.howto . With today's conveniency of virtualization at hand, I actually did an upgrade by experiment with a VM first, to minimize the risk, and documented all the steps. Since I had to upgrade several 10.1 machines, this seemed like a good idea to reduce the risk and save brain energy. The notes from it are a good outcome, too.

icons/user_comment.png K. K. wrote: (5 years ago)

Whats the bug number for this problem ?

icons/user_comment.png R. D. wrote: (5 years ago)

On Scope, here are the Release Notes for Lenny - http://www.uk.debian.org/releases/testing/amd64/release-notes/ 4.1.4 - Prepare a safe environment for the upgrade - text mode or via ssh login,
Article on how to go from Etch to Lenny - http://www.debianadmin.com/howto-upgrade-from-debian-etch-40-to-lenny-50.html

icons/user_comment.png F. L. wrote: (5 years ago)

this is very helpful, we should look at the bumps that Debian hits while doing this.

On a separate note, just tried to "yum upgrade" frlom RHEL 5.2 to RHEL 5.3, and it went without a hitch. There may be something to be gathered from Fedora as well here.

icons/user_comment.png M. T. wrote: (5 years ago)

I think this is a really helpful characteristic, because a lot of people like me, use openSUSE as server, and is a true hell to upgrade servers in any datacenter, at kilometers off distance.
I hope this feature will be a reallity in the next 11.2.

icons/user_comment.png D. M. wrote: (5 years ago)

Please, if you just want to express your favorites, use the voting feature. Comments is for discussion around solving the problem itself.

icons/user_comment.png J. L. wrote: (5 years ago)

Just a few comments...
A few weeks ago, I upgraded one server from 11.0 to 11.1 via ssh using zypper dup.
zypper and its related libraries have to be upgraded first in order to avoid problems, especially if the upgrade process is interrupted.
Another issue besides the order of the upgrade, would be restarting daemons and dealing with new .conf. Maybe we could check if it has been edited, if not, then automerge it. Some stuff could malfunction due to old config files. We could also have a manual merge option.
Another thing... wouldn't it be better to download all the packages at once and install them afterwards instead of downloading/installing package by package?

On a side note, I want to say that my upgrade went smoothly (rcpbind wasn't installed but I consider that a small issue).
zypper has been improved a lot. It would be great if zypper dup was officially supported as another upgrade method.

icons/user_comment.png P. P. wrote: (5 years ago)

Regarding config merge on update, I have played with two things.
First, the yum plugin called yum-merge-conf, which allows for interactive merging of config files during the upgrade. Works and gets the job done, and could be done in zypper as well, however from my own experience I have to say that it protracts the update more than I sometimes like, and I typically want to do this job in a separate step (after the packages are updated). not protract the update for merging all files.

So I rather use another approach, which is a script called rpmresolve (https://build.opensuse.org/package/view_file?file=rpmresolve&package=servertools&project=home%3Apoeml ), which works in a similar way, based on presence of rpmsave/rpmnew files. It opens vim in diff mode, lets you merge and asks you which version you want to keep. Works well for me since years, it just might require refinement of some things of the implementation in a more generic way (e.g. calling $EDITOR with different option than -d).

icons/user_comment.png a. f. wrote: (5 years ago)

i think that we miss something like that:

https://help.ubuntu.com/community/JauntyUpgrades

it's so easy to upgrade an ubuntu version that also my 5 years old student was able to do it..

icons/user_comment.png R. U. wrote: (4 years ago)

I just did a distribution upgrade from 11.1 to 11.2 factory using "zypper dup". I did uninstall a lot before to have less problems.

After upgrade there were some repository problems, like I was not able to see opera any more. I remove all of /var/cache/zypp/raw/* after which I could install opera. Seems to be some work left on zypper to be able to dist-upgrade ....

icons/user_comment.png G. F. wrote: (4 years ago)

I think repository management is not getting enough attention in this discussion.

Per http://en.opensuse.org/Upgrade/11.2#Command_line it is step one, but I don't see any tool / script etc. shown for assisting the user in doing this.

At a minimum a simple script should be created and rolled via updates that allows a user to review the current repositories and disable the 11.1 ones and create the 11.2 ones.

Complicating this is all the OBS / packman / other repositories that many users now have in a standard setup.

I think the tool needs to somehow address these.  Even if only to say. "Automated replacement of this repository is not supported.  Disabled for distribution upgrade process.  Please seperately add a 11.2 repository for these functions if needed after the upgrade."

FYI: does this specific feature deserve its own fate entry?

icons/user_comment.png J. K. wrote: (4 years ago)

Not a bad idea, but the thing is that we can only do this reliably for the main repo (containing the openSUSE-11.1 product definition) and its update repo. Plus we can search for common OBS path components like openSUSE_Factory and tell the user that s/he needs to change these into openSUSE_11.2 where available (or do that automatically if the user agrees). Yeah, might be worth doing.

Should do something like this (with user interaction added, and corrected of course :O)

zypper lr -e my11.1repos.repo    # backup your list of repos 
zypper mr -da #disable all repos
zypper rr $(get-the-official-11.1-repos) # remove the official main/non-oss/update repos
zypper ar http://download.opensuse.org/distribution/11.2/repo/oss/ main
zypper ar http://download.opensuse.org/distribution/11.2/repo/non-oss/ nonoss
zypper ar http://download.opensuse.org/update/11.2 updates
# and optionally
for url in $(zypper lr -u | grep -E 'openSUSE_(Factory|11.1|11.0)' | extract-the-URL); do
zypper rr $url
newurl=$(echo $url | sed -r -e 's/openSUSE_(Factory|11.1|11.0)/openSUSE_11.2/')
curl ...something... $newurl # check if the new url is available
if $?; then
zypper ar $newurl $well-we-should-use-the-old-alias-here
fi
done
remove-all-disabled-repos-with-user-approval
# and optionally
zypper ref
zypper mr -kt
zypper up zypper libzypp
zypper dup --download-only
zypper dup

I'm not sure if this needs a separate fate request. Looks like this can be done quickly in bash or perl, i hope i'm right. I'm not sure about rolling this as an 11.1 update, though. Would need to be part of some package. Isn't having it available from a link at the update instruction enough?

icons/user_comment.png G. F. wrote: (4 years ago)

The instructions at http://en.opensuse.org/Upgrade/11.2#Command_line call for having the latest zypper package.  (ie. zypper in zypper)

If you take the script you wrote and call it "zypper-dup" or similar and add it to the zypper package, then tell everyone that an upgrade is accomplished via zypped-dup most people won't need instructions.

Currently most of the email posts say something like "I upgraded via zypper -dup and ....".  It gives the casual reader the false impression that they just need to call zypper -dup.

So my vote is have a single script that manages the full upgrade, including any necessary notes and warnings, and we don't require users in general to have to go find a website to read the update instructions.

icons/user_comment.png R. U. wrote: (4 years ago)

With the last 3 commands I migrated successfully from 11.1 to 11.2 ten days ago (some manual work of above done before).

As command "dup"  is defined "distribution upgrade" , a command "zypper dup" should automatically:

zypper up zypper libzypp; respawn zypper dup --download-before

icons/user_comment.png G. F. wrote: (4 years ago)

I just happened to look at the current generic Upgrade instructions.

The handling of previous repos there seems pretty drastic to me:

mv /etc/zypp/repos.d /etc/zypp/repos.d-backup

See:
http://en.opensuse.org/Upgrade#Removing_old_repositories

Seems to me with the community repos, one click installs, etc. this does not even come close to addressing the issues at hand.
The good part is it may allow the zypper dup to run cleanly.

I have never used the above process to try a zypper dup upgrade, so maybe my concerns are overblown.

icons/user_comment.png R. U. wrote: (4 years ago)

At the moment in the forums you can notice various problems upgrading to Kde-4.3 due to changed naming convention (drop of kde4- prefix).

A proper packaging system should be able to handle that without problems. Debian style to solve this problem is to use meta/transitional-packages: empty packages which pull in the needed packages

icons/user_comment.png S. B. wrote: (4 years ago)

zypper dup handles those renames just fine.

icons/user_comment.png R. U. wrote: (4 years ago)

; )

Are these problems people discuss only about problematic one-klick upgrades ....

icons/user_comment.png a. f. wrote: (4 years ago)

the problem here is the packager, any package that change name must provides and obsoletes the onld name in spec file.

icons/user_comment.png K. E. wrote: (4 years ago)

Docs department speaking ;)

From the documentation POV this is a prio 1 feature. This means we must know ASAP, what's the status of this feature and what do you intend to do about it until your deadline. Please, summarize the status otherwise it is difficult to provide useful documentation.

For example, we must know what are the pre-requisites the user is required to perform. Some of these are mentioned above (close all applications, add a catalog XYZ, update to version 11.1 + all online updates first, etc.).

Developers, please take the time to provide the wanted info. Project/Product Mangers, please clarify open issues.

icons/user_comment.png J. D. wrote: (4 years ago)

Youi have to manage the case where an application is not anymore part of the distro. For example dovecot was and is no more (and have no replacement). This is much difficult to handle.

Could it be possible to have a "diagnostic tool", zypper -dup-diag (?) that could scan the system and report the work to do?

- list the non compatible applications or running daemons

- list the new conf file that will have to be setup...

jdd

icons/user_comment.png C. R. wrote: (4 years ago)

dovecot11 Obsoletes dovecot, that is the easier case to handle.

icons/user_comment.png a. f. wrote: (4 years ago)

Are there any news here?

this topic looks to be dead

icons/user_comment.png K. E. wrote: (4 years ago)

Please, take the time to summarize the implementation--see comment #36. If you do not, I take it for granted that it works as outlined in the description (modulo gesunder menschenverstand/human sense).

We are quickly approaching the documentation deadlines for openSUSE 11.2.

icons/user_comment.png S. K. wrote: (4 years ago)

there is no "implementation" per se. It just works and if it does not work, we need to fix bugs in the packages.

icons/user_comment.png a. f. wrote: (4 years ago)

in other words...

how can i move from 11.2 to 11.3?

there will be any GUI? (something like that? https://help.ubuntu.com/community/JauntyUpgrades?action=AttachFile&do=get&target=um2.jpg )

icons/user_comment.png J. B. wrote: (4 years ago)

Even
Fedora manage something like this (although it's not an online upgrade). ;)

Last change: 4 years ago
Voting
Score: 245
  • Negative: 6
  • Neutral: 3
  • Positive: 251
Feature Export
Application-xmlXML   Text-x-logPlaintext   PrinterPrint