Request to support proxied S3 connections

[ZachSARAO]

Hi Ludwig,

I am experimenting with accessing the RDB files via a proxy. This works:

import katdal
link = "https://archive-gw-1.kat.ac.za/1750206187/1750206187_sdp_l0.full.rdb?token=ey...etc"
data = katdal.open(link)
print(data)

But this does not:

import katdal
proxied_link = "https://archive.sarao.ac.za/s3/1750206187/1750206187_sdp_l0.full.rdb?token=ey...etc (same token)"
data = katdal.open(proxied_link)
print(data)

BUT. In datasources.py I found if I pass the proxied URL with the token - then the proxied URL works:

 (463)  rdb_store = S3ChunkStore(store_url, **kwargs)
 (464)  # rdb_data = rdb_store.request('GET', rdb_url, process=_read_object)
 (464)  rdb_data = rdb_store.request('GET', proxied_link, process=_read_object)
 (465)  telstate.load_from_file(io.BytesIO(rdb_data))

Somewhere the request wrapper is dropping the token. Looking into this a little more. In chunkstore_s3.py:

        # Use _standard_errors to filter errors emanating from within with-block
        with self._standard_errors(chunk_name), self._session_pool() as session:
            adapter = session.get_adapter(url)
            while True:
                # Initialise and reuse the same Retry object for the entire session
                adapter.max_retries = retries
                try:
                    with _request(session, method, url, timeout, **kwargs) as response:  <====== using a URL with the token works here
                        _raise_for_status(response, chunk_name, ignored_errors)
                        retries = response.raw.retries.new()
                        return process(response)

The /s3 proxy path is stripped by the proxy server, so the the claim is correct without the s3 path (I saw there was an explicit check in katdal for this).

@contextlib.contextmanager
def _request(session, method, url, timeout=(None, None), **kwargs):
    """A beefed-up request that facilitates retries on reading the response.

    This catches socket timeouts and reset connections while the response data
    is being read and reraises them as appropriate urllib3 exceptions that can
    be passed to a `Retry` object to trigger a read retry.
    """
    try:
        with session.request(method, url, timeout=timeout, **kwargs) as response:   <======== session.request is not getting the token so far as I can tell
            yield response

if I set url to the original URL (with the token param) then it works

[ZachSARAO]

I would say tha the session._request function is not passing auth - but obviously it is since the katarchive-gw-1 address works

[ZachSARAO]

*also - I am happy to make a PR. Is that okay?

[ludwigschwardt]

Well, there are two issues:

1. I pre-emptively check the token authorisation before I make the request, to give a better error message. The check would need to be relaxed for a proxy server. Since the real auth happens on the archive side, this is not a biggie.
2. I strip the token and add a Bearer: <token> header instead, which is how the auth works. In the proxy case we need to transmit the token as a parameter like before.

If we are going to add an archive proxy, it would have to be supported in katdal in an agnostic fashion. No hardcoded URLs or paths...

One possibility: when doing the token prefix check, if the token has prefix (e.g. "1750346172") and the URL has a path that ends in prefix (e.g. "s3/1750346172"), then assume it's a proxy server and retain the token as a parameter.

[ZachSARAO]

Yes - it would definitely be good to support Ceph behind a proxy in general (actually it already is - but in this case, i have a path rewrite).

Regarding (2) I saw that the token is recognized, and I do see a bearer <token> header, but it seems to be dropped before making the actual request. (I just manually deleted the (1) pre-emptive check in my local copy of katdal to test this).

But another thing that would be useful to me - I can't easily set Verify=SSL to False for the outgoing requests, which I discovered when testing on a local (dev) instance of the archive server (https://localhost:3000/s3/etc).

Would it be possible to allow for injecting kwargs into the request session/request? Then I would be able handle auth from my python script instead of it being handled via katdal. This would also allow for supporting other options - like routing requests through a forward proxy.

response = requests.get("https://archive-gw-1.kat.ac.za", proxies={"https": "some-squid-address.gov.za"})

[ludwigschwardt]

Regarding (2) I saw that the token is recognized, and I do see a bearer header, but it seems to be dropped before making the actual request.

Interesting... do you mean the requests.get call in katdal or the one going out on the wire? I would be surprised if it is totally absent, cause otherwise what's the point of the token? 😅

I can't easily set Verify=SSL to False for the outgoing requests

Do you mean that the https forces this and you can't use http?

Would it be possible to allow for injecting kwargs into the request session/request?

It sounds doable.

[ZachSARAO]

For Verfy=False. We use self-signed certs for development, which http clients don't allow by default (unless we tell the client not to verify certs via Verify=False). This is also useful for internal domain names served over https.

When I was debugging the auth within the katdal library I could see that the auth object was getting created, but then no auth was getting passed to the actual http call (I didn't get much further - I'm on leave until Tuesday next week and I will look more into it)

[bmerry]

For Verfy=False. We use self-signed certs for development, which http clients don't allow by default (unless we tell the client not to verify certs via Verify=False). This is also useful for internal domain names served over https.

Note that you can set REQUESTS_CA_BUNDLE in the environment to point to a local CA certificate, which avoids the need to modify any code (and is more secure since you are still verifying the certificate).

FYI we've got an internal CA set up in DPP - our dev servers get certificates from it with ACME, and then we only have to add that CA certificate to trust stores. Drop me a message if you'd like to know more about it and want to use it.

[ZachSARAO]

Thanks @bmerry - I didn't think of specifying a CA certificate to verify self signed certs. That will make our development more pleasant generally