Automatically Use Proxy from Environment Variables

[very-doge-wow]

It would be helpful if the download_file function could automatically detect and use standard proxy environment variables such as HTTP_PROXY, HTTPS_PROXY, NO_PROXY and their lowercase variants (http_proxy, https_proxy, no_proxy). This behavior is common in many HTTP client libraries (like curl, wget, and reqwest), and would improve developer ergonomics when running tools in environments with corporate or local proxies.

[schneems]

I'm leaning towards not doing this.

More: Looks like it's more of a convention: https://superuser.com/questions/944958/are-http-proxy-https-proxy-and-no-proxy-environment-variables-standard.

I happen to have a PR today to update this function (but not with your requested behavior) #947. I bring this up, because you can see that when adding behavior to that function, it means you also change the public API of the error. And it would implicitly change the behavior of all buildpacks using this function, possibly without them realizing.

I think in this case, the behavior of that function is already so relatively small, you can copy it and extend it however you like. I'm not even using it in the Ruby buildpack https://github.com/heroku/buildpacks-ruby/blob/e249a8bcbc75f7e5322f02bb4e32aaed50724355/buildpacks/ruby/src/layers/bundle_download_layer.rs. And other buildpacks use other download mechanisms https://github.com/heroku/buildpacks-nodejs/blob/09e118446c450c5c8e4e9c941458e59763f6dbb9/common/nodejs-utils/src/download_file.rs.

While libcnb (crate) is intended to be spec-focused, the libherokubuildpack is a bit more opinionated and seeks to serve the needs of heroku buildpack maintainers (which, others are also welcome to use) but it makes adding in functionality that's not directly supported (like https://github.com/heroku/buildpacks-ruby/blob/e249a8bcbc75f7e5322f02bb4e32aaed50724355/docs/application_contract.md) a larger hurdle to clear.

An alternative path forwards could be to make the case that ALL buildpacks or their providers provide some kind of proxy support in an upstream RFC https://github.com/buildpacks/rfcs, but I think the difficulty to clear that bar would be even higher than here. If you want to go that route, writing more details with specifics. For example:

• What buildpacks are you developing? (I see one fork of the classic buildpack very-doge-wow/heroku-buildpack-jvm-common@074975f, but no CNBs in your public profile)
• Examples of specific network policies you're trying to work around and how they affect one or more buildpacks
• Examples of proxy implementation or configuration
• A strong ownership model of who should be allowed to configure a proxy and at what levels? Should it be only configured by the platform or should other buildpacks be allowed to secretly change the behavior of where your binaries come from if they run before your buildpack? Should the user be allowed to configure this?
• What workarounds are available, and why is this the preferred solution?

Just some examples of information that might be helpful.

[very-doge-wow]

Looks like it's more of a convention

Yes indeed it is, which is why I wrote This behavior is common in many HTTP client libraries but not that it is a standard which has to be implemented everywhere. But still, it would be nice to have it.

the behavior of that function is already so relatively small, you can copy it and extend it however you like
What buildpacks are you developing?

We're not developing our own buildpacks. Instead, we're using the official Heroku JVM buildpack via the heroku/builder image. Our environment is corporate and restricted—internet access is only allowed through a proxy, which uses a whitelist configuration.

Some domains, like amazonaws.com, are whitelisted and should be accessed directly, bypassing the proxy. The JVM buildpack tries to download JDKs from an AWS S3 bucket, which falls into this category. So for these downloads, the requests must go directly to AWS. All other traffic should go through the proxy.

This setup can usually be handled by setting the HTTP_PROXY and NO_PROXY environment variables appropriately. However, the current implementation of the download_file function in the buildpack does not support this behavior. Since we're using a prebuilt buildpack and not developing our own, we unfortunately have no control over how this is handled.