Improve action's setup time

[Jackenmen]

Currently, most of the time is spent on the setup of the action - especially the installation of the Python package. It would be good to look into ways of improving that such as:

• caching dependencies and the installed package (perhaps whole virtualenv) based on the action commit
• building a wheel and storing it as a release artifact (would probably require additional fetching unless I put it as part of the repository)
• switching from composite action to Docker action
  • this might not be the optimal direction as Docker actions are likely to be slower to set up (but the docker image is surely cached)