centos-infra

#186 vault.centos.org responds to all download requests with a 302, preventing squid from caching packages locally
Closed: Fixed by arrfab. Opened by abraunegg.

Closed: Fixed
Reopen Issue

Hi all,

It appears that when requesting a package (via wget or yum, dnf etc) from vault.centos.org, the response from vault.centos.org is a 302 which is followed, and the file is downloaded.

However, when a 302 is presented, squid will not (regardless of configuration) cache the response, thus, rpm files sourced via wget, yum, dnf, when using squid as a local proxy, will refuse to cache the download.

Download file example:

wget -e use_proxy=yes -e http_proxy=http://127.0.0.1:3128 -O /dev/null http://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm --debug
DEBUG output created by Wget 1.14 on linux-gnu.

URI encoding = ‘UTF-8’
URI encoding = ‘UTF-8’
Converted file name 'ElectricFence-2.2.2-39.el7.x86_64.rpm' (UTF-8) -> 'ElectricFence-2.2.2-39.el7.x86_64.rpm' (UTF-8)
--2021-01-12 15:49:28--  http://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm
Connecting to 127.0.0.1:3128... connected.
Created socket 4.
Releasing 0x0000000000fcfaf0 (new refcount 0).
Deleting unused 0x0000000000fcfaf0.

---request begin---
GET http://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm HTTP/1.1
User-Agent: Wget/1.14 (linux-gnu)
Accept: */*
Host: vault.centos.org
Connection: Close
Proxy-Connection: Keep-Alive

---request end---
Proxy request sent, awaiting response... 
---response begin---
HTTP/1.1 302 Found
Date: Tue, 12 Jan 2021 04:49:29 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
X-Frame-Options: SAMEORIGIN
Location: https://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm
Content-Length: 274
Content-Type: text/html; charset=iso-8859-1
X-Cache: MISS from squid-proxy
X-Cache-Lookup: MISS from squid-proxy:3128
Via: 1.1 squid-proxy (squid/3.5.20)
Connection: close

---response end---
302 Found
URI content encoding = ‘iso-8859-1’
Location: https://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm [following]
Closed fd 4
URI content encoding = None
Converted file name 'ElectricFence-2.2.2-39.el7.x86_64.rpm' (UTF-8) -> 'ElectricFence-2.2.2-39.el7.x86_64.rpm' (UTF-8)
--2021-01-12 15:49:29--  https://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm
Resolving vault.centos.org (vault.centos.org)... 3.22.185.178
Caching vault.centos.org => 3.22.185.178
Connecting to vault.centos.org (vault.centos.org)|3.22.185.178|:443... connected.
Created socket 4.
Releasing 0x00000000010b52f0 (new refcount 1).
Initiating SSL handshake.
Handshake successful; connected socket 4 to SSL handle 0x000000000110b1d0
certificate:
  subject: /CN=vault.centos.org
  issuer:  /C=US/O=Let's Encrypt/CN=R3
X509 certificate successfully verified and matches host vault.centos.org

---request begin---
GET /7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm HTTP/1.1
User-Agent: Wget/1.14 (linux-gnu)
Accept: */*
Host: vault.centos.org
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... 
---response begin---
HTTP/1.1 200 OK
Date: Tue, 12 Jan 2021 04:49:30 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
Strict-Transport-Security: max-age=31536000
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
X-Frame-Options: SAMEORIGIN
Last-Modified: Fri, 04 Jul 2014 01:11:36 GMT
ETag: "8d14-4fd53ce886200"
Accept-Ranges: bytes
Content-Length: 36116
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: application/x-rpm

---response end---
200 OK
Registered socket 4 for persistent reuse.
Length: 36116 (35K) [application/x-rpm]
Saving to: ‘/dev/null’

100%[====================================================================================================================================================>] 36,116       146KB/s   in 0.2s   

2021-01-12 15:49:30 (146 KB/s) - ‘/dev/null’ saved [36116/36116]

squid cache access.log entries:

1610423513.961   1064 127.0.0.1 TCP_MISS/302 809 GET http://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm - HIER_DIRECT/3.22.185.178 text/html
1610423540.876    506 127.0.0.1 TCP_MISS/302 809 GET http://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm - HIER_DIRECT/3.22.185.178 text/html
1610426880.188    742 127.0.0.1 TCP_MISS/302 809 GET http://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm - HIER_DIRECT/3.22.185.178 text/html
1610426890.911    481 127.0.0.1 TCP_MISS/302 809 GET http://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm - HIER_DIRECT/3.22.185.178 text/html
1610426961.449    845 127.0.0.1 TCP_MISS/302 809 GET http://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm - HIER_DIRECT/3.22.185.178 text/html
1610426969.318    482 127.0.0.1 TCP_MISS/302 809 GET http://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm - HIER_DIRECT/3.22.185.178 text/html

squid cache debugging:

2021/01/12 15:49:21.448 kid1| 20,2| store.cc(953) checkCachable: StoreEntry::checkCachable: NO: not cachable
2021/01/12 15:49:21.448 kid1| 20,2| store.cc(953) checkCachable: StoreEntry::checkCachable: NO: not cachable
2021/01/12 15:49:21.448 kid1| 20,2| store.cc(953) checkCachable: StoreEntry::checkCachable: NO: not cachable
2021/01/12 15:49:21.449 kid1| 20,2| store.cc(953) checkCachable: StoreEntry::checkCachable: NO: not cachable
2021/01/12 15:49:21.449 kid1| 20,2| store.cc(953) checkCachable: StoreEntry::checkCachable: NO: not cachable
2021/01/12 15:49:29.318 kid1| 20,2| store.cc(953) checkCachable: StoreEntry::checkCachable: NO: not cachable
2021/01/12 15:49:29.318 kid1| 20,2| store.cc(953) checkCachable: StoreEntry::checkCachable: NO: not cachable
2021/01/12 15:49:29.318 kid1| 20,2| store.cc(953) checkCachable: StoreEntry::checkCachable: NO: not cachable
2021/01/12 15:49:29.318 kid1| 20,2| store.cc(953) checkCachable: StoreEntry::checkCachable: NO: not cachable
2021/01/12 15:49:29.318 kid1| 20,2| store.cc(953) checkCachable: StoreEntry::checkCachable: NO: not cachable

Download from non-vault site:

wget -e use_proxy=yes -e http_proxy=http://127.0.0.1:3128 -O /dev/null http://mirror.aarnet.edu.au/pub/centos/7.9.2009/updates/x86_64/Packages/389-ds-base-snmp-1.3.10.2-8.el7_9.x86_64.rpm --debug
DEBUG output created by Wget 1.14 on linux-gnu.

URI encoding = ‘UTF-8’
URI encoding = ‘UTF-8’
Converted file name '389-ds-base-snmp-1.3.10.2-8.el7_9.x86_64.rpm' (UTF-8) -> '389-ds-base-snmp-1.3.10.2-8.el7_9.x86_64.rpm' (UTF-8)
--2021-01-12 17:34:31--  http://mirror.aarnet.edu.au/pub/centos/7.9.2009/updates/x86_64/Packages/389-ds-base-snmp-1.3.10.2-8.el7_9.x86_64.rpm
Connecting to 127.0.0.1:3128... connected.
Created socket 4.
Releasing 0x0000000000f33cc0 (new refcount 0).
Deleting unused 0x0000000000f33cc0.

---request begin---
GET http://mirror.aarnet.edu.au/pub/centos/7.9.2009/updates/x86_64/Packages/389-ds-base-snmp-1.3.10.2-8.el7_9.x86_64.rpm HTTP/1.1
User-Agent: Wget/1.14 (linux-gnu)
Accept: */*
Host: mirror.aarnet.edu.au
Connection: Close
Proxy-Connection: Keep-Alive

---request end---
Proxy request sent, awaiting response... 
---response begin---
HTTP/1.1 200 OK
Date: Tue, 12 Jan 2021 01:08:18 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux)
Last-Modified: Thu, 17 Dec 2020 20:33:07 GMT
ETag: "2ccbc-5b6aee6493c10"
Accept-Ranges: bytes
Content-Length: 183484
Content-Type: application/x-rpm
Age: 19573
X-Cache: HIT from squid-proxy
X-Cache-Lookup: HIT from squid-proxy:3128
Via: 1.1 squid-proxy (squid/3.5.20)
Connection: close

---response end---
200 OK
Length: 183484 (179K) [application/x-rpm]
Saving to: ‘/dev/null’

100%[====================================================================================================================================================>] 183,484     --.-K/s   in 0.001s  

Closed fd 4
2021-01-12 17:34:31 (341 MB/s) - ‘/dev/null’ saved [183484/183484]

squid cache response - sucessful in cache & response:

1610433271.143      4 127.0.0.1 TCP_HIT/200 183885 GET http://mirror.aarnet.edu.au/pub/centos/7.9.2009/updates/x86_64/Packages/389-ds-base-snmp-1.3.10.2-8.el7_9.x86_64.rpm - HIER_NONE/- application/x-rpm

squid store debug:

2021/01/12 12:08:18.697 kid1| 20,2| store_io.cc(42) storeCreate: storeCreate: Selected dir 0 for e:=w1p2DV/0x55cf322f9eb0*4

squid debug with 'debug_options ALL,2' set:

2021/01/12 19:24:30.613 kid1| 5,2| TcpAcceptor.cc(218) doAccept: New connection on FD 17
2021/01/12 19:24:30.613 kid1| 5,2| TcpAcceptor.cc(293) acceptNext: connection on local=[::]:3128 remote=[::] FD 17 flags=9
2021/01/12 19:24:30.614 kid1| 11,2| client_side.cc(2364) parseHttpRequest: HTTP Client local=127.0.0.1:3128 remote=127.0.0.1:59642 FD 12 flags=1
2021/01/12 19:24:30.614 kid1| 11,2| client_side.cc(2365) parseHttpRequest: HTTP Client REQUEST:
---------
GET http://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm HTTP/1.1
User-Agent: Wget/1.14 (linux-gnu)
Accept: */*
Host: vault.centos.org
Connection: Close
Proxy-Connection: Keep-Alive


----------
2021/01/12 19:24:30.614 kid1| 85,2| client_side_request.cc(744) clientAccessCheckDone: The request GET http://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm is ALLOWED; last ACL checked: localhost
2021/01/12 19:24:30.614 kid1| 85,2| client_side_request.cc(720) clientAccessCheck2: No adapted_http_access configuration. default: ALLOW
2021/01/12 19:24:30.614 kid1| 85,2| client_side_request.cc(744) clientAccessCheckDone: The request GET http://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm is ALLOWED; last ACL checked: localhost
2021/01/12 19:24:30.615 kid1| 17,2| FwdState.cc(133) FwdState: Forwarding client request local=127.0.0.1:3128 remote=127.0.0.1:59642 FD 12 flags=1, url=http://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm
2021/01/12 19:24:30.615 kid1| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find IP destination for: http://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm' via vault.centos.org
2021/01/12 19:24:31.216 kid1| 44,2| peer_select.cc(280) peerSelectDnsPaths: Found sources for 'http://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm'
2021/01/12 19:24:31.216 kid1| 44,2| peer_select.cc(281) peerSelectDnsPaths:   always_direct = DENIED
2021/01/12 19:24:31.216 kid1| 44,2| peer_select.cc(282) peerSelectDnsPaths:    never_direct = DENIED
2021/01/12 19:24:31.216 kid1| 44,2| peer_select.cc(286) peerSelectDnsPaths:          DIRECT = local=0.0.0.0 remote=3.22.185.178:80 flags=1
2021/01/12 19:24:31.216 kid1| 44,2| peer_select.cc(295) peerSelectDnsPaths:        timedout = 0
2021/01/12 19:24:31.219 kid1| 11,2| http.cc(2237) sendRequest: HTTP Server local=192.168.1.2:53350 remote=3.22.185.178:80 FD 13 flags=1
2021/01/12 19:24:31.219 kid1| 11,2| http.cc(2238) sendRequest: HTTP Server REQUEST:
---------
GET /7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm HTTP/1.1
User-Agent: Wget/1.14 (linux-gnu)
Accept: */*
Host: vault.centos.org
Via: 1.1 squid-proxy (squid/3.5.20)
X-Forwarded-For: 127.0.0.1
Cache-Control: max-age=2592000


----------
2021/01/12 19:24:31.746 kid1| ctx: enter level  0: 'http://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm'
2021/01/12 19:24:31.746 kid1| 55,2| HttpHeader.cc(1782) httpHeaderNoteParsedEntry: cannot parse hdr field: 'Content-Length: 274'
2021/01/12 19:24:31.746 kid1| 55,2| HttpHeader.cc(1782) httpHeaderNoteParsedEntry: cannot parse hdr field: 'Content-Length: 274'
2021/01/12 19:24:31.746 kid1| 11,2| http.cc(750) processReplyHeader: HTTP Server local=192.168.1.2:53350 remote=3.22.185.178:80 FD 13 flags=1
2021/01/12 19:24:31.746 kid1| 11,2| http.cc(751) processReplyHeader: HTTP Server REPLY:
---------
HTTP/1.1 302 Found
Date: Tue, 12 Jan 2021 08:24:31 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
X-Frame-Options: SAMEORIGIN
Location: https://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm
Content-Length: 274
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="https://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm">here</a>.</p>
</body></html>

----------
2021/01/12 19:24:31.746 kid1| ctx: exit level  0
2021/01/12 19:24:31.746 kid1| 20,2| store.cc(953) checkCachable: StoreEntry::checkCachable: NO: not cachable
2021/01/12 19:24:31.746 kid1| 20,2| store.cc(953) checkCachable: StoreEntry::checkCachable: NO: not cachable
2021/01/12 19:24:31.746 kid1| 55,2| HttpHeader.cc(1782) httpHeaderNoteParsedEntry: cannot parse hdr field: 'Content-Length: 274'
2021/01/12 19:24:31.746 kid1| 55,2| HttpHeader.cc(1782) httpHeaderNoteParsedEntry: cannot parse hdr field: 'Content-Length: 274'
2021/01/12 19:24:31.746 kid1| 33,2| client_side_reply.cc(1472) buildReplyHeader: clientBuildReplyHeader: Connection Keep-Alive not requested by admin or client
2021/01/12 19:24:31.746 kid1| 88,2| client_side_reply.cc(1994) processReplyAccessResult: The reply for GET http://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm is ALLOWED, because it matched localhost
2021/01/12 19:24:31.746 kid1| 11,2| client_side.cc(1393) sendStartOfMessage: HTTP Client local=127.0.0.1:3128 remote=127.0.0.1:59642 FD 12 flags=1
2021/01/12 19:24:31.746 kid1| 11,2| client_side.cc(1394) sendStartOfMessage: HTTP Client REPLY:
---------
HTTP/1.1 302 Found
Date: Tue, 12 Jan 2021 08:24:31 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
X-Frame-Options: SAMEORIGIN
Location: https://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm
Content-Length: 274
Content-Type: text/html; charset=iso-8859-1
X-Cache: MISS from squid-proxy
X-Cache-Lookup: MISS from squid-proxy:3128
Via: 1.1 squid-proxy (squid/3.5.20)
Connection: close


----------
2021/01/12 19:24:31.746 kid1| 20,2| store.cc(953) checkCachable: StoreEntry::checkCachable: NO: not cachable
2021/01/12 19:24:31.746 kid1| 20,2| store.cc(953) checkCachable: StoreEntry::checkCachable: NO: not cachable
2021/01/12 19:24:31.747 kid1| 33,2| client_side.cc(817) swanSong: local=127.0.0.1:3128 remote=127.0.0.1:59642 flags=1
2021/01/12 19:24:31.747 kid1| 20,2| store.cc(953) checkCachable: StoreEntry::checkCachable: NO: not cachable

Replication Steps:
1. Install squid, default configuration
2. Update squid configuration with the following:

# Debugging
debug_options ALL,1 20,2
#debug_options ALL,2

# Performance Related Stuff here
cache_mem 256 MB
minimum_object_size 0 KB
maximum_object_size 128 MB
maximum_object_size_in_memory 1024 KB
  1. Test / download rpm file from various sites (examples)
    • http://mirror.aarnet.edu.au/pub/centos/7.9.2009/updates/x86_64/Packages/389-ds-base-snmp-1.3.10.2-8.el7_9.x86_64.rpm
    • http://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm

Additional Information:
Unfortunately any 'refresh_pattern' config added to the squid configuration makes zero difference to files hosted on vault.centos.org - however it 100% makes sure rpm files from other sites are cached:

refresh_pattern -i .rpm$ 10080 90% 43200 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-store

By rights this 'should' tell squid to cache the file, but even in this configuration, squid still kicks out a store.cc(953) checkCachable: StoreEntry::checkCachable: NO: not cachable response for anything hosted on vault.centos.org.

If we look at redbot.org for any vault.centos.org RPM URL, REDBot details that the response allows all caches to store it - which, is not entirely accurate either - as squid fails to cache the file.

Is this an infra issue with vault.centos.org or is there some sort of other issue going on here?

the following small redirect change was pushed to git and automatically applied by ansible :

https://github.com/CentOS/ansible-role-mirror-vault/commit/988d18ad669067f8ff6f04abc58d39b980d50377

Can you confirm that it works now for you ?

Metadata Update from @arrfab:
- Issue assigned to arrfab

Metadata Update from @arrfab:
- Issue priority set to: Waiting on Reporter (was: Needs Review)
- Issue tagged with: centos-common-infra, low-gain, low-trouble

@arrfab
Confirm that I am now getting a 301 response & the RPM files are being correctly cached / saved:

1610440242.193    436 127.0.0.1 TCP_MISS/301 845 GET http://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm - HIER_DIRECT/54.186.51.210 text/html
1610440299.025      0 127.0.0.1 TCP_HIT/301 852 GET http://vault.centos.org/7.8.2003/os/x86_64/Packages/ElectricFence-2.2.2-39.el7.x86_64.rpm - HIER_NONE/- text/html
1610440332.565   1170 127.0.0.1 TCP_MISS/301 839 GET http://vault.centos.org/centos/6/updates/x86_64/Packages/tzdata-2020d-1.el6.noarch.rpm - HIER_DIRECT/3.22.185.178 text/html
1610440345.513      0 127.0.0.1 TCP_MEM_HIT/301 846 GET http://vault.centos.org/centos/6/updates/x86_64/Packages/tzdata-2020d-1.el6.noarch.rpm - HIER_NONE/- text/html

Thanks for the quick fix

Metadata Update from @arrfab:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

Log in to comment on this ticket.

Metadata

centos-common-infra low-gain low-trouble

None
None
Highest
Powered by Pagure 1.0.0
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.