#270 Some of our jobs randomly fails with:- Failed to connect to mirror.centos.org port 80
Closed: Fixed by arrfab. Opened by ykarel.

It fails like below randomly, starting from yesterday:-
DEBUG util.py:446: Advanced Virtualization mirror 0.0 B/s | 0 B 00:00
DEBUG util.py:444: Errors during downloading metadata for repository 'advanced-virtualization':
DEBUG util.py:444: - Curl error (7): Couldn't connect to server for http://mirror.centos.org/centos/8/virt/x86_64/advanced-virtualization/repodata/repomd.xml [Failed to connect to mirror.centos.org port 80: Connection refused]
DEBUG util.py:444: Error: Failed to download metadata for repo 'advanced-virtualization': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

repo config

[advanced-virtualization]
name=Advanced Virtualization mirror
baseurl=http://mirror.centos.org/centos/8/virt/$basearch/advanced-virtualization/
gpgcheck=0
enabled=1
module_hotfixes=1

Looks like it's hitting some bad mirror


Metadata Update from @arrfab:
- Issue priority set to: Waiting on External (was: Needs Review)
- Issue tagged with: need-more-info

as all nodes in actual mirror.centos.org GeoIP PDNS pool are tested and under monitoring, it can be that either :
* you're using dns override to point to a machine we removed already (check your infra and dns resolver)
* there are routing issues between your infra and that specific IP/node

So can you give us the ip of the node you're trying to reach ?

PS : using mirrorlist.centos.org is still the recommended way to spread the load accross more mirrors (external ones)

@arrfab i tried curl -v mirror.centos.org, and it once failed for 173.214.244.24, for other it passed, checking if there are more failed ones.

I had a look at three nodes in our pdns setup and they are all current/up2date with backend data.
So if you try dig +short @pdns1.centos.org mirror.centos.org from a machine in US, you'll never receive that specific IP as it was removed yesterday from infra .

Wondering about DNS cache in resolvers in between, not really obeying the short TTL we set for these records.

@ykarel tempted to close it as nothing really found at centos infra level and I guess that we can't get in touch with the resolver setup you use (it seems cloudflare and google).
If that's ok for you, can we agree on closing it ?

Metadata Update from @arrfab:
- Issue assigned to arrfab

@arrfab Thanks for looking, yes it can be closed, since last night we are not seeing issue, resolver seems resolved.

Metadata Update from @arrfab:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

Log in to comment on this ticket.

Metadata