Enable additional authentication methods to allow publishing to private Python repos #741
Comments
|
Rye just dispatches to twine. Is it not possible to pass |
There are no real username and password for the authentication, but everything is stored in the |
|
I understand that you store this in |
|
Aha ok, so rye publish is basically an "alias" for Twine? I thought it only supported the options that were mentioned in the documentation. I can give it a try and see what happens. |
|
It takes it's own options. Sorry the flags today are |
|
@mitsuhiko I also tested using only the --repository flag but then it automatically assumed pypi as well. 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.1/10.1 kB • 00:00 • ?
WARNING Error during upload. Retry with the --verbose option for more details.
ERROR HTTPError: 403 Forbidden from https://upload.pypi.org/legacy/
Username/Password authentication is no longer supported. Migrate to API Tokens or Trusted Publishers instead. See
https://pypi.org/help/#apitoken and https://pypi.org/help/#trusted-publishersIt seems to only accept publishing to pypi? Edit: When I'm using twine to publish I only supply the command "upload" together with the --repository-argument and the files. This is not possible as it is hard-coded to always supply command such as username, password and url. I got the debugger up and running, I will play around a little bit and maybe put together a PR. Edit2: |
|
I think you should provide the |
If I try this I get the error |
|
FWIW If you don't use |
|
I'd think adding .pypirc and OIDC support would be cool. If there's interest in this I could try to PR something this weekend, unless someone else gets to it before me. I can also try to scoop up some housecleaning for the publish code. |
|
Note Twine will default to use API tokens.
IIUC it should pick up the username for non-PyPI usage, but I'm also curious of how your repository is set up. |
|
@cnpryer that would be great. I started looking into it but with my experience in Rust this will take weeks. The repository is set up according to the guildelines for setting up a Python-repo in Google Cloud. To be honest, I have no idea exactly what is wrong. If I try to feed in all the information from command-line, I get an HTTP 403 (I.e. forbidden). So the connection works, but it rejects the authenatication for some reason. If I use twine with only the --repository-flag, it grabs everything from .pypirc and it works. Might be something internally how twine sends the data depending on from where the input comes? I'm pretty sure it is free to set up a Google Cloud account and a repo, might be something to consider as otherwise testing will be quite the hassle. I will of course be available for any testing or discussions. |
|
Can you try and isolate the auth resource you're using? I believe Twine will use keyring creds if a password isn't provided, but, for me at least, it's hard to narrow down your exact issue without understanding your exact setup better. If you're using a If your Of course if a MWE is shared I can check that out. |
|
@cnpryer I'm not sure if I follow you. I do authenticate with the .pypirc, it looks in my first post in this thread. I tried to change the password with one character in this file (Generated key of 256 bits or something) and I get a HTTP 401 error, i.e. Unauthorized. Changing the character back, everything works immediately. So I can in other words confirm that it does read the content in the .pypirc-file. As mentioned, I tried to take all the information from the .pypirc-file and input it as arguments, see below: Looking in the .pypirc-file I assumed the following:
Running the command without --repository-url, --username and --password works fine: These commands where executed within a few minutes in between in the same order as they are written down. I do note one difference in Apart from that I can't really spot any differences. This is obviously something with Twine, but it is a fact that it works if you follow the guildlines from Google, i.e. only supply the repository and get the rest of the information from the .pypirc instead of supplying it via the CLI. And there is the problem with the current implementation in Rye, as it is hardcode to always supply certain arguments, and bail if they are not correct or missing. So either we dig into this deeper and figure out why Twine behaves as it does, or we just accept how it is and refactor the code to be more flexible and only pass the arguments that is actually supplied. |
|
I read through https://cloud.google.com/artifact-registry/docs/python/authentication#search-order a little. It's hard to learn more about your problem without having experience with the platform or having time to learn about it enough to test your issue. Are you using the Artifact Registry keyring backend? That key you're using looks like it's an encoded key. |
|
If we want something quick for now, cnpryer:0a76694222adec09ea450c5fc5fd041d1518fdf7 will hopefully have:
I'm also unsure if there are plans to either continue leaning on Twine the way we are, or if there are plans to implement more of the publish process ourselves. |
|
@cnpryer The only thing required is to provide a more direct interface to the internal Twine. I only need to use the --repository-argument, and nothing else. With the current setup this is not possible as several other arguments are hard-coded. Having for example an interface that just passes the arguments to Twine without to much checking and modification of data and everything will work. I have a PR, I can make a simple example in a PR to make it more clear. For example using a similar solution to below for all arguments would solve my issue: if let Some(cert) = cmd.cert {
publish_cmd.arg("--cert").arg(cert);
}These solutions is what creates my problem: let repository_url = match cmd.repository_url {
Some(url) => url,
None => {
let default_repository_url = Url::parse("https://upload.pypi.org/legacy/")?;
credentials
.get(repository)
.and_then(|table| table.get("repository-url"))
.map(|url| match Url::parse(&escape_string(url.to_string())) {
Ok(url) => url,
Err(_) => default_repository_url.clone(),
})
.unwrap_or(default_repository_url)
}
};The reason being, that if I don't provide a URL, there is logic to assume pypi. This is however not always true, as my case proves. There are other sources of this information than the command line arguments. That might even be the whole reason why the code looks like it does. It assumes all necessary information must come from the CLI. I will have a look and give your branch a test. Thanks! |
I was actually thinking of setting one up. I have a personal Google Cloud account primarily for playing around. I can see if I can set something up. |
|
If I have time later I'll finish up a first-pass. I can use test.pypi.org to manually test the changes. I wouldn't do anything with that branch yet. It's definitely not done. I'm sharing the branch here in case this comment becomes relevant:
Or if we'd like a quick fix like this and there's some feedback on the solution I'm working on. |
rye publish `
--repository devpi `
--repository-url http://pypi.private.repository `
--username test`
--token a_very_complex_password `
--skip-existingProviding both |
Background
I'm working almost daily with Python at work and first of all I have to tell you that Rye is a blessing. It makes my work so much quicker and easier by automating and isolating the environments, it is almost crazy.
The company have an internal Python repository where we keep Python packages that we use internally. Currently I have it set up in a Github Action using Twine, but I would like to migrate to using Rye here as well as then it would be one complete solution.
From what I understand, Pypi is primarily trusting a token to enable authenatication but it seems Google is using a completely different method. I was never able to get the publishing working. The documentation does not cover anything about any other methods either.
The whole process in Github Action is generally:
google-github-actions/auth@v2.pypircorpip.confFrom what I understand, the recommended way is to use
.pypirc.Typical entry in
.pypircTypical entry in pip.conf
Twine then uses these settings in either pip.conf or
.pypircwhen it uploads the files and the commands looks like:The issue
I tried to basically switch out
twine uploadwithrye publishbut as far as I can remember I just got an error that it was missing a token. As discussed above, gcloud doesn't use a token and as far as I know there are no way of generating one.I tried to add the
extra-index-urlas--repository-url(from the pip.conf) in the rye publish as well (both with and without--repositoryoption), but this didn't seem to do any difference.I tried to look at the src, but I couldn't really figure out how things work. It does however seem that Rye primarily uses the token and doesn't consider the pip.conf or
.pypirc.It would be great if the Rye publish could match the capabilities of Twine. That would really make it a "The tool to rule them rule all" :)
I have the intention to learn Rust and would love to contribute to this project, but I'm too much of a beginner to solver something like this.
References
The text was updated successfully, but these errors were encountered: