Home_greyopenFATE - openSUSE feature tracking > #309561
Dashboard | Search | Sign up | Login

Please login or register to be able to edit or vote this feature.

Delta downloads of repository metadata

Feature state

Hackweek V
Done
openSUSE-11.4
Done

Description

We already use deltarpms do speed up downloading of rpms when doing an update. Unfortunately, the repository metadata is also quite a chunk of data which considerably slows down update speed.

The idea is to use a zsync like algorithm to fetch only the changed parts of the metadata. The implementation will use libcurl's multi interface to work with multiple connections in parallel.

Another benefit is that the code can replace the aria2c interface, which does not fit well into the current code (different proxy handling and the like).

References

packages: libzypp

Discussion


icons/user_comment.png J. E. wrote: (7 years ago)

Say, if we were to (finally) use solv-style metadata instead of rpm-md, wouldn't the problem reduce itself?

Also, if it were possible to have an aggregate metadata instead that can reference multiple subrepositories, big repositories (like packman and mine) could split theirs up into grouped repositories without having to add extra .repo entries to /etc/zypp/repos.d.

E.g.

/multimedia/i586/MPlayer.rpm

/multimedia/i586/ffmpeg.rpm

/multimedia/repodata/*

/games/i586/trackball.rpm

/games/i586/...

/games/repodata/*

/repodata/aggregate.xml

Each group (multimedia, games) would have its own metadata, that is, being a self-contained repository. And aggregate.xml containins then:

<link ref="multimedia/" />

<link ref="games/" />

So that when there is an update in games, only games's md needs to be downloaded, instead of the big one.

icons/user_comment.png D. M. wrote: (7 years ago)

Closing this one as done.

icons/user_comment.png D. M. wrote: (7 years ago)

Coolo, giving the fact that aria is causing problems in 11.3 and this code is very isolated, would you accept a backport to 11.3 as online update in the context of:

- enabled only explicitly (env variable)

May serve as a good test for 11.4 as not many people is using Factory.

icons/user_comment.png S. K. wrote: (7 years ago)

if it's not default, it might be worthwhile. Michael asked for it and I considered a danger, but he forgot the detail about explicitly turning it on.

icons/user_comment.png M. M. wrote: (7 years ago)

BTW, what changes need to be done on the repository side? Or is this already fully implemented in Factory?

icons/user_comment.png S. K. wrote: (7 years ago)

afaik we patched the way the meta data is compressed (gzip --rsyncable) and the download redirector now gives out more details about the file blocks.

The first part is easy to do for older products, the metalink for SLES updates might be a bit challenging.

icons/user_comment.png D. M. wrote: (6 years ago)

libzypp part is done when request id 69748 is accepted

icons/user_comment.png M. A. wrote: (6 years ago)

This libzypp is available in SUSE:SLE-11:SP2

Last change: 6 years ago
Voting
Score: 23
  • Negative: 0
  • Neutral: 1
  • Positive: 23
Tags

No tags yet.

Feature Export
Application-xmlXML   Text-x-logPlaintext   PrinterPrint