Update requirements

[ddkohler]

Due for an update (3 years since we checked the pins). notes:

• several dependencies can be updated to current versions (numpy, happi, WrightTools, attune)
• bluesky-queueserver is pinned lower than latest because kwargs change in an updated version--haven't accounted for those yet
• updating debian: needed to implement venvs for installing requirements
• re-manager needs python-3.11 still because we use databroker v1; a bypass option is to disable mongo, but it also looks like updating to latest bluesky-queueserver (v0.0.22) will also resolve the issue.
• keeping packages that we have some control over as unpinned (attune, WrightTools, wright-plans, etc), since we likely want our package to update as we update those.
• note that our queueserver container always pulls the latest version, but we pin lower versions in other containers that use the package (bluesky-queueserver build it not used anymore and queueserver is implemented in re-manager)

at some point we will want to update bluesky, but not today.

TODO

• test on guinea_pig system
• test on fs-system
• repin dependencies

[ksunden]

There is a bit of a balancing act to be done with things like this...

Ideally, there would be regular (preferably automated) updates and automated tests to ensure things work.

However, it is often recommended to pin all dependencies for deployment to get consistency and minimize risk of things breaking when you are unprepared to deal with it (e.g. in the middle of an experimental session). This is different than the recommendations for library development, which tend to leave the upper bound open and pin to a reasonable supported lower bound. "We know this configuration works, so keep using it".

Perhaps the thing to do is schedule regular manual updates but pin to latest at those times. Somewhere around the 3-month mark would probably be my recommendation, though perhaps rolling into the "6 month" maintenance schedule would also be a practical suggestion (that is still significantly better than it has been.

I'd love to have automated tests for this, but am unsure how complete such tests could be. (very high integration factor in this case, which makes testing significantly harder)

You should be able to configure something like dependabot to do regular checks for updates that then require a manual validation, but the tool could help set the schedule/ease the actual pinning changes

[ddkohler]

Thanks @ksunden . Good to know the existing pins were just for guaranteed success--I was worried we were pinning due to some updated dependency that broke things in a terrible way.

I completely agree: pin for deployment, but schedule checks for updated pins. I will look into dependabot.

For this PR, I will pin to the latest I can get working.

[ksunden]

Yeah, I do think there are likely to be some changes needed for the major pieces (e.g. bluesky in particular) Not sure how much, could "just work", but that is one that needs particular attention to update.

[ddkohler]

Note to self re; latest bluesky-queueserver: https://github.com/bluesky/bluesky-queueserver/blob/main/docs/source/cli_tools.rst#instances-of-run-engine-and-databroker