Home_greyopenFATE - openSUSE feature tracking > #305320
Dashboard | Search | Sign up | Login

Please login or register to be able to edit or vote this feature.

Show accessed mirror(s)

Feature state

openSUSE-11.1
Rejected Information
openSUSE-11.2
Done

Description

This concerns both YaST and zypper.

It would help a lot if they would show the mirror which is accessed when
downloading a package.

(Technically, the download client is following redirects and the actually
URL needs to be shown, not just the first URL that was contacted but
that redirected to another URL)

It is a long-standing problem that this information is lacking and it
makes debugging and optimizing much harder than necessary, because this
information is only available deeply buried in some logfile.

We need this information for two reasons:

  1. Debugging - to see which URL was accessed when something goes wrong,
    like a mirror timing out. For example, Whenever a mirror returns
    garbage, it takes a lot of additional time to find out which one,
    because all reports are lacking the critical detail, and it is usually
    hard to find it out later.
  2. Optimizing the load sharing between different mirrors - to distinguish
    slow mirrors from fast ones. For example, I am very much in need of
    information about the situation in the US, but even though people in the
    US run installations for me and are willing to send me logs, I can't get
    this information.

In reply to a question from Stano:

> Do you want the information in the log or to be shown in e.g. zypper -vvv ?

Shown, in normal output. It is already in some log I believe, I don't
know which one exactly anymore though.

Thing is, users won't be able to tell us anything useful if they weren't
shown the information. And later it is not reproducible.

At least in a failure situation the URLs must be shown. That alone
wouldn't give us the chance to find out about slow mirrors, but at least
it would allow to debug problems.

Real life example #1:

If a mirror is behind a firewall that lets every 10th connection stall
(so the transfer hangs or the connection is reset), it usually takes
weeks (!) to find out about it. But it hits virtually every user from
mirrors country during the time. We have had this 4 times during the
last 18 months.

Real life example #2:

I got feedback from someone who did a network install of
openSUSE 11.0 from download.opensuse.org, located in the US. He had to
let it running overnight because it took a long time. Next morning, the
install hadn't even completed. YaST was hanging and waiting for somebody
to hit retry. Problem is, it took us a lot of work to find out which
mirror was at fault. We actually had to repeat the install, I had to
follow the log and live watching his machine accessing
download.opensuse.org, to see where it was redirected, and where it
spent most time (while downloading some package). We found a mirror
which was really slow, and I could disable it. I have a suspicion
though, that we have more such mirrors, because I have got some similar
reports from recently. But how to find out? People only complain about
slowness, without the ability to give any useful detail.
See
http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments for examples of this.

References

https://bugzilla.novell.com/show_bug.cgi?id=522260 - patch adding logging of redirections for curl backend

Testcase

For zypp on code 11 (openSUSE 11.1, SLE11):

Add any repository below download.opensuse.org via zypper or YaST and install a package from it (the package should be served from a mirror). Zypper/YaST log should contain lines with "Location:" string (grep 'Location: ' /var/log/zypper.log).

For zypp on code 12, the redirections are not logged, but are part of the error message details in case download error. See the testcase in Fate #302923 for info on how to get broken mirrors from the redirector in order to test this.

Discussion


icons/user_comment.png S. O. wrote: (9 years ago)

What if we focussed on something like torrent for distributing packages instead?

AFAICT these p2p network technologies have mechanisms to recover from missing
fragments, i.e. failing servers.

In that world, todays mirrors would be strong and almost always present peers in
the network.

icons/user_comment.png P. P. wrote: (9 years ago)

Torrents are not efficient for files of this small size.

Another concern is that, on large scale, the reasonability of P2P traffic with
regard to network topology is questionable; confer to what's called the
Application-Layer Traffic Optimization (ALTO) problem. See
http://alto.tilab.com/ .

There are working examples where e.g. university have rolled out a P2P update
distribution campus-wide; in a controlled limited environment this is feasible,
but not across the Internet.

Further, note that libzypp in 11.1 has preliminary support for using metalinks
for downlaods. Metalink clients recover nicely from missing fragments.
See
http://en.opensuse.org/Libzypp/Failover .

icons/user_comment.png M. G. wrote: (9 years ago)

Other usage of this feature would be possibility of "blacklisting" servers with
persistent problems. (See thread in opensuse list
"Is it possible to
"blacklist" specific download server in Yast?"
).

As Carlos E. R. suggested other useful option could be for YaST/zypper to fail
gracefully requesting another server to the redirector on failure on the one it
gave use previously.

icons/user_comment.png P. P. wrote: (9 years ago)

Gracefully falling back to other mirrors is already being implemented (currently
in prototype stage) by way of
http://en.opensuse.org/Libzypp/Failover .

The feature of blacklisting a specific mirrors is included in that proposal, and
I have added it explicitely.

icons/user_comment.png G. P. wrote: (9 years ago)

Looking at the discussion on opensuse-factory and my own experience, I also see this as high priority for 11.2.

icons/user_comment.png P. P. wrote: (9 years ago)

Just for the record, the way to enable logging that makes mirrors show up in the log seems to be:

$ ZYPP_MEDIA_CURL_DEBUG=1 zypper ...

Cf. http://en.opensuse.org/Debugging_YaST

icons/user_comment.png J. K. wrote: (9 years ago)

Yes, it is, but that's quite excessive log that we don't want to be enabled by default. Instead we need to pick up the redirects and log them one line per each.

Michal Marek already gave me a hint how to do this: "Use CURLOPT_HEADERFUNCTION callback, which gets each http header and then test for 'Location:' and log that." I'll try to do it ASAP.

icons/user_comment.png J. S. wrote: (8 years ago)

Let's try to display this information at least with zypper. However, we will need some investigation, since we switched to aria2c.

icons/user_comment.png P. P. wrote: (8 years ago)

Another example of the implications. What users get right now is
this  (screenshot that shows a rather meaningless URL, but not the accessed mirror). See
this thread on the English opensuse mailing list.

icons/user_comment.png J. K. wrote: (8 years ago)

OK, this is all about the CURL backend, so we're developing an improvment to be released as an online update for code 10 products (openSUSE 11.1 and SLE11), right?

Peter, is this needed also for the aria2c backend? If so, can you give me some hints how to do it with aria (i have zero knowlege of aria)?

icons/user_comment.png J. K. wrote: (8 years ago)

above, i meant code 11, of course

icons/user_comment.png P. P. wrote: (8 years ago)

I should mention that when I initially requested this feature, we were still developing 11.1, and the fail-safe aria2c backend wasn't implemented yet. Now that the new fail-safe backend is becoming the default download backend in 11.2, the need for this feature request is a lot smaller.

It would still be beneficial for us if users get direct feedback (via the user interface) about mirror performance, because we will still need to maintain an overall healthy mirror system, and tune here and there. The fail-safe backend will compensate for many or most problems.

The fail-safe backend collects its own mirror performance stats, and we could actually think about submit those stats to a central place to gather a high-level view on mirror performance. We could actually correlate the performance to geographical or network topological client location. This would allow for better mirror selection in the first place, which will allow for a better client experience overall.

openSUSE 11.1 and 11.0 would benefit a lot from the improvement requested here, of course. Furthermore, there might be edge cases where the aria2c backend isn't used - I'm not sure if we do (or can) use it from the install initial ramdisk when installing with a "mini iso", for instance.

The curl backend of 11.2 should definitely have good logging of all server accesses - not only in a debug mode. The information must be easily available. Best would be if its shown directly to the user, so they know what could be a possible problem, and know what to report.

aria2c logs exceptions in a way that we can use to enhance our logging (and UI). Tatsuhiro added the patch for us mentioned below: 

I created a patch to embed URI(redirected one) to exception message.  
Since there are many exception messages, I wrapped exception with new
one whose message contains URI. For example, the exception message for
404 error is now:
Exception: URI=http://localhost/abc
-> Resource not found
icons/user_comment.png J. K. wrote: (8 years ago)

Thanx Peter for the comment. So we have the curl logging for code 11 (
Bug #522260 ) and code 12 (and not only in debug mode) ready.

As for 11.2, looking at the MediaAria2c.cc code, the exceptions are already parsed from the output of aria2c and passed to callbacks, thus shown to the user, so nothing to do here (Duncan, please correct me if i'm wrong).

Regarding curl in 11.2, i guess having the redirects logged is enough, since aria2c is now the default backend (even though it may not be suitable for some uses, as Peter suggests).

All in all, i'd consider this feature done. Please reopen if i missed something.

icons/user_comment.png J. K. wrote: (8 years ago)

Alright, logging is easy. If a Location: line will be found in reponse header, it will be logged like: "MediaCurl.cc(log_redirects_curl):120 redirecting to Location: URL" in zypper.log. I have the patch ready and it is pretty simple and can be released as update for openSUSE 11.x and SLE11.

With showing the redirects in the UIs it is quite complicated since it requires MediaException API changes and adding the list of redirects to each MediaException thrown. Nothing for the released products i'd say.

Can we agree on this? Let me empasize that this is for the cURL backend used in code11 products. For openSUSE 11.2 and later (using aria2c backend) i'm still waiting for info asked in c#10.

Last change: 8 years ago
Voting
Score: 22
  • Negative: 0
  • Neutral: 0
  • Positive: 22
Feature Export
Application-xmlXML   Text-x-logPlaintext   PrinterPrint