how to push multiple packages

[Borda]

Hello, I am wondering how we can push two packages... We have a package lets say PL which is running a standard release cycle eventually with some release candidates (which is working fine now), then we have created another package PL-nightly which we want to push every day, for that reasons I have in this nightly action change of version to YYYYMMDD and rename a package in setup.py to PL-nightly before creating package so the outputs in dist are:

-rw-r--r-- 1 runner docker 1.8M Sep 28 00:58 pytorch-lightning-nightly-20200928.tar.gz
-rw-r--r-- 1 runner docker 426K Sep 28 00:58 pytorch_lightning_nightly-20200928-py3-none-any.whl

but then I am getting a server error

Checking dist/pytorch_lightning_nightly-20200928-py3-none-any.whl: PASSED
Checking dist/pytorch-lightning-nightly-20200928.tar.gz: PASSED
Uploading distributions to https://test.pypi.org/legacy/
Uploading pytorch_lightning_nightly-20200928-py3-none-any.whl

  0%|          | 0.00/458k [00:00<?, ?B/s]
  2%|▏         | 8.00k/458k [00:00<00:06, 74.6kB/s]
100%|██████████| 458k/458k [00:00<00:00, 653kB/s]  
NOTE: Try --verbose to see response content.
HTTPError: 403 Forbidden from https://test.pypi.org/legacy/
Invalid API Token: InvalidMacaroon('invalid macaroon signature')

does it mean that I have to somehow create and multiple tokens?

The build is here: https://github.com/PyTorchLightning/pytorch-lightning/runs/1174122197

[webknjaz]

Some people would use dev-releases of the same dist name. e.g. something like pytorch_lightning_nightly-1.0.dev20200928-py3-none-any.whl.

But if you want a separate package name, bear in mind that API tokens can be either user-wide or project-scoped: https://pypi.org/help/#apitoken. If you use a user-scoped token, you should be able to create a project with the new name with the first upload. Although, it is recommended to use project-scoped tokens for security purposes. This one will only work if the project already exists so you could do a manual upload first and then switch to using a project-scoped API token.

Also, remember that PyPI and TestPyPI have completely separate accounts (hence different credentials: https://pypi.org/help/#invalid-auth).

[Borda]

Some people would use dev-releases of the same dist name. e.g. something like pytorch_lightning_nightly-1.0.dev20200928-py3-none-any.whl.

this is not a problem with the 1.0.dev20200928

But if you want a separate package name, bear in mind that API tokens can be either user-wide or project-scoped: https://pypi.org/help/#apitoken. If you use a user-scoped token, you should be able to create a project with the new name with the first upload. Although, it is recommended to use project-scoped tokens for security purposes. This one will only work if the project already exists so you could do a manual upload first and then switch to using a project-scoped API token.

that we have done, I can upload it locally, but not with the GH action... so do I need to set two tokens?

Also, remember that PyPI and TestPyPI have completely separate accounts (hence different credentials: https://pypi.org/help/#invalid-auth).

We also were able to push to normal and testing pypi with the PL name (not PL-nightly)

[webknjaz]

If you have a project scoped token, it won't work for projects with other names (https://pypi.org/help/#invalid-auth). So yes, if you have a different project, you'll need a different token for that project. You can use a separate secret in your repo for that.

[Borda]

so it means that in the action I need to do some remapping of __token__ depending on the publishing project, right?

[webknjaz]

Yes, you may use a different var conditionally or have a separate step calling the action. But it's up to you how you set it up according to your needs.