Speculative visit directory service #283

callumforrester · 2023-07-31T09:48:31Z

PLEASE DO NOT MERGE!
This PR is intended as a speculative prototyping exercise only.

Design

In order to comply with the DLS visit directory structure, we need to group acquisitions into files. There are many solutions to this, the basics of this one:

A new service is responsible for determining where data will go for a particular acquisition. This functionality is separated from all other applications for two reasons:
- So that other data collection applications can also use it each one is guaranteed unique data locations.
- This involves making directories and therefore requires write access to places where data goes. For security reasons, only trusted code should have that level of privilege, and it should be kept small and isolated.
Every time a group of devices is staged, blueapi will call out to this service (or use a dummy version for offline development) and get a unique data directory for this stage. This is achieved via a plan preprocessor that detects groups of stages and makes the call.
Compatible Ophyd detectors will be aware of a mutable state which is updated when this call occurs, so they can write to the "current" visit.
The current group is bundled into the run start metadata. Allowing for a group:run mapping of 1:1 or 1:many. The use case for the latter is writing many runs to a single file for performance reasons.
Any aggregating service, such as a nexus file writer, can take the group information from the run starts and use it how it likes. Our initial thinking was to map each group to a nexus file and reach run to an entry, but that is TBD.

Design Issues

I remain very sceptical about point 2! It introduces a shared mutable state between all detectors just to get around the fact that there is no direct, universal way to pass the current visit down to them, by design! The other alternative I explored is to put all support for configuring individual detectors in one place, in blueapi, and if your detector isn't supported it will just write as it normally does. Simple example

def configure_data_writing(
    devices: Iterable[Device],
    collection: DataCollection,
) -> None:
    for device in devices:
        if isinstance(device, FileStoreBase):
            path_template = str(collection.raw_data_files_root)

            # Configure Ophyd Device to setup HDF5 writer
            device.reg_root = "/"
            device.read_path_template = path_template
            device.write_path_template = path_template

What's Prototyped Here?

The actual contents of this PR:

A very simple visit service that can create and inform clients about unique directories for "acquisition groups".
A plan preprocessor that calls out to that service (or use an internal dummy version) to create a new group when a set of devices is staged. It also puts metadata into the run start document to show which group it is a part of. Each start document now contains a key called data_collection_number, which can be passed back to the visit service for more information.
A dummy device that has access to the cached result of the service call.
A series of tests ensuring that the correct mapping of run -> file is achieved for various use cases, such as multi-run plans.
A small amendment to the bluesky context that makes blueapi use this preprocessor on every plan it runs (if the plan does not contain any stages, it will be unaffected)

How to test

Checkout this branch. To test with the dummy detector, run

tox -e pytest -- tests/plugins/test_data_writing.py

To start the visit service:

python -m blueapi.plugins.data_writing_server

It should run on port 8089, you can test it with

curl -X POST "http://localhost:8089/collection/mybeamline"

Which should create a new group and return a blob of JSON about it.

Running a scan with blueapi should now produce a start document that contains an attribute called data_collection_number which we can use to aggregate runs.

Still to do

This PR contains a dummy detector with the correct hooks. It would be good to get this working with an actual Ophyd v2 detector, and ideally a v1 one too. We could then run a full collection via blueapi that uses this functionality @coretl @callumforrester

More discussion on point 2. of the design would be helpful, I think. @coretl @DiamondJoseph @keithralphs @callumforrester

We should try to get the nexus writer to interpret the group number in the run start document and ensure there are no problems there @DiamondJoseph

Possibly leverage the data_session metadata in the run start schema. @callumforrester @DiamondJoseph

codecov · 2023-07-31T16:20:16Z

Codecov Report

Merging #283 (60354a8) into main (3f87223) will decrease coverage by 0.73%.
The diff coverage is 76.00%.

❗ Current head 60354a8 differs from pull request most recent head 49dd587. Consider uploading reports for the commit 49dd587 to get more accurate results

@@            Coverage Diff             @@
##             main     #283      +/-   ##
==========================================
- Coverage   89.07%   88.34%   -0.73%     
==========================================
  Files          39       44       +5     
  Lines        1446     1536      +90     
==========================================
+ Hits         1288     1357      +69     
- Misses        158      179      +21

Files	Coverage Δ
src/blueapi/core/__init__.py	`100.00% <100.00%> (ø)`
src/blueapi/core/bluesky_types.py	`89.74% <100.00%> (+1.17%)`	⬆️
src/blueapi/core/device_walk.py	`100.00% <100.00%> (ø)`
src/blueapi/plans/plans.py	`100.00% <100.00%> (ø)`
src/blueapi/plugins/data_writing.py	`83.60% <83.60%> (ø)`
src/blueapi/plugins/data_writing_server.py	`55.55% <55.55%> (ø)`

... and 9 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

src/blueapi/core/device_walk.py

coretl · 2023-08-08T14:27:07Z

tests/plugins/file_writing_detector.py

+    def stage(self) -> List[object]:
+        collection = self._provider.current_data_collection
+        if collection is None:
+            raise Exception("No active collection")


This seems pretty readable to me, requires explicit knowledge of a DataCollectionProvider when making the class, but that makes it obvious where the collection number is coming from

I agree it's readable but it is also a bodge:

Do we really want to bake such a workaround into Ophyd v2 when we have a chance to design a nice, shiny new system?

From discussion, happy with this because it is theoretically easy to strip out later, as long as:

We keep in mind that we want to strip it out later

We make an interface or ABC for the singleton in ophyd-async

We make a DLS-specific implementation in blueapi

We maintain a parallel implementation in ophyd-async for offline use, that just writes to a preconfigured directory or similar

We inject references to the singleton into each device

src/blueapi/plugins/data_writing.py

DiamondJoseph · 2023-08-14T09:06:32Z

src/blueapi/core/context.py

+        wrapped_plan = functools.reduce(
+            lambda wrapped, next_wrapper: next_wrapper(wrapped),
+            self.plan_wrappers,
+            plan,


Might be obvious to people coming from python, but making plan be a kwarg initializer=plan would stop me running off to read the docs for functools every time I see it.

Reduce explicitly disallows keyword arguments

src/blueapi/plugins/data_writing_server.py

callumforrester · 2023-10-02T10:58:14Z

src/blueapi/plugins/data_writing.py

+DATA_COLLECTION_NUMBER = "data_collection_number"
+
+
+class DataCollectionProvider(ABC):


This should be removed and replaced with

bluesky/ophyd-async@7b50350

I can't see a direct replacement to the DataCollectionProvider in the file you've linked there...

your DataCollectionProvider gives a DataCollection, which has 4 things: collection_number, group, raw_data_files_root and nexus_file_path. Tom's work that you've linked has a DirectoryProvider which gives a DirectoryInfo object, which just contains a directory path and filename prefix.

So it's not a direct mapping. Perhaps I'll chat to you about this tomorrow morning in more detail.

It doesn't need to be a direct mapping, you can just change the logic here to match up with Tom's DirectoryProvider.

Cool, got it

I've come across a bit of an annoyance; I want to subclass DirectoryProvider so it gets the directory from the visit service. However, I want to use aiohttp or some similar asynchronous library... DirectoryProvider has one synchronous method, __call__.

My solution for now, is to make a subclass GDADirectoryProvider (suggestions for a better name?) and have a async update method on this. Then the synchronous __call__ can do something like asyncio.wait_for(update).

Yep, I think that's what I told @coretl I had in mind at the time

As discussed, the pre-processor should do the async access to the directory service, then cache the result in DirectoryProvider, then the __call__ method just returns the cached value

callumforrester · 2023-10-02T11:00:34Z

src/blueapi/plugins/data_writing.py

+def data_writing_wrapper(
+    plan: MsgGenerator,
+    provider: DataCollectionProvider,
+) -> MsgGenerator:
+    staging = False
+    for message in plan:
+        if message.command == "stage":
+            if not staging:
+                yield from bps.wait_for([provider.update])
+                staging = True
+            if provider.current_data_collection is None:
+                raise Exception("There is no active data collection")
+        elif staging:
+            staging = False
+
+        if message.command == "open_run":
+            if provider.current_data_collection is None:
+                yield from bps.wait_for([provider.update])
+            if provider.current_data_collection is None:
+                raise Exception("There is no active data collection")
+            message.kwargs[
+                DATA_COLLECTION_NUMBER
+            ] = provider.current_data_collection.collection_number
+        yield message
+
+
+data_writing_decorator = make_decorator(data_writing_wrapper)


This is the only actual part of this file that should stick around. I'm not convinced it should be called plugins/data_writing.py either. Maybe preprocessors/xyz.py

If it's going to be a plan preprocessor should it be moved to dls-bluesky-core?

I thought about this. It is specifically a part of blueapi, that fudges plans to write data in our structure, maybe it shouldn't be allowed to get outside of blueapi?

I'm fine with it existing only within BlueAPI and being applied on plan import: prevents possibility of attempting to apply multiple times (pretty sure from glancing through the logic that would just doubly increment the data collection number each run, but that's disruptive enough) or being missed.

Should anyone want to be a. using vanilla Bluesky/not BlueAPI, b. wanting to write to DLS filesystem, they can extract and duplicate the logic. But I don't think that'll be very often.

But on the original comment: Yes, having a blueapi.preprocessors package seems best.

callumforrester · 2023-10-02T11:01:20Z

src/blueapi/core/context.py

    plans: Dict[str, Plan] = field(default_factory=dict)
    devices: Dict[str, Device] = field(default_factory=dict)
    plan_functions: Dict[str, PlanGenerator] = field(default_factory=dict)

    _reference_cache: Dict[Type, Type] = field(default_factory=dict)

+    def wrap(self, plan: MsgGenerator) -> MsgGenerator:


This should be tested

src/blueapi/core/bluesky_types.py

callumforrester · 2023-10-02T11:02:51Z

src/blueapi/worker/task.py

+        wrapped_plan_generator = ctx.wrap(plan_generator)
+        ctx.run_engine(wrapped_plan_generator)


This needs testing too

callumforrester · 2023-10-02T11:15:25Z

src/blueapi/plugins/data_writing.py

+        )
+
+
+def data_writing_wrapper(


There may be a nicer way of doing this, @rosesyrett to investigate and make my flowchart look less ugly

keithralphs

to be discussed further...

keithralphs · 2023-10-10T10:52:13Z

src/blueapi/config.py

-            return str(self.sources) == str(other.sources)
+            return (
+                (str(self.sources) == str(other.sources))
+                and (self.facility == other.facility)


left out visit_id here

callumforrester · 2023-10-24T13:59:43Z

src/blueapi/data_management/gda_directory_provider.py

+        return DataCollectionIdentifier(collectionNumber=self._count)
+
+
+class VisitDirectoryProvider(DirectoryProvider):


Needs tests

callumforrester · 2023-10-24T14:00:38Z

src/blueapi/core/context.py

@@ -86,6 +104,18 @@ def with_config(self, config: EnvironmentConfig) -> None:
            elif source.kind is SourceKind.DODAL:
                self.with_dodal_module(mod)

+        call_in_bluesky_event_loop(self.connect_devices(self.sim))
+
+    async def connect_devices(self, sim: bool = False) -> None:


Should make private and rename to _connect_ophyd_async_devices

…315) Production version of #283 This PR allows coordination with a central service for creating unique groups of data (called collections) and configures ophyd-async detectors to write their data to the same location for a given collection. Changes: - Add mechanism to preprocess all plans with set bluesky [preprocessors](https://blueskyproject.io/bluesky/plans.html#plan-preprocessors) - Create directory provider that knows how to talk to GDA's visit directory API and provide a unique collection number to group data files - Create dummy directory provider that works in a similar way without the need for an external server (useful for development) - Create preprocessor that uses the directory provider and groups detectors by staging, also bundles the data group information into run start documents on a best effort basis. - Add tests --------- Co-authored-by: Rose Yemelyanova <[email protected]>

stan-dot · 2024-07-22T17:02:37Z

willing to test with with the webcam at visr https://github.com/DiamondLightSource/ViSR

@coretl and @callumforrester is this a good idea?

callumforrester · 2024-07-24T08:12:31Z

This is very old and out-of-date, everything in it now exists in blueapi in some form.

callumforrester requested review from DiamondJoseph, keithralphs and coretl August 1, 2023 14:01

coretl reviewed Aug 8, 2023

View reviewed changes

DiamondJoseph reviewed Aug 14, 2023

View reviewed changes

callumforrester commented Oct 2, 2023

View reviewed changes

src/blueapi/plugins/data_writing_server.py Outdated Show resolved Hide resolved

callumforrester commented Oct 2, 2023

View reviewed changes

src/blueapi/core/bluesky_types.py Outdated Show resolved Hide resolved

callumforrester commented Oct 2, 2023

View reviewed changes

keithralphs reviewed Oct 10, 2023

View reviewed changes

rosesyrett force-pushed the visit-directory branch from 9387506 to c363f1b Compare October 19, 2023 11:35

callumforrester and others added 15 commits October 24, 2023 14:54

Speculative visit directory service

25772e9

Clean up

20f2ac0

More cleaning up

ebff3fd

Add tests

f0c287d

Simplify tests for initial version

e0d9905

Write plan preprocessors

a56a4b9

Ensure correct collection numbers are propagated to detectors

c45b0ba

Simplify preprocessor

92fdbb5

Fix mypy errors

7daaf01

Add kwarg to explain reduce

d4176ae

init

44d9c13

temp

7765855

link directory provider

dabed95

Add capability for context to connect ophyd-async devices

de1a2ad

Post-rebase changes

1108ee6

Rose Yemelyanova and others added 10 commits October 24, 2023 14:55

Import inject from dls-bluesky-core instead

54d882f

Use dodal singleton

ece0e84

Setup visit directory service connection

06daaf3

Begin fixing attach metadata tests

2cf7ff5

Fix test cases

16593a3

Allow for dummy visit service handler

a258e2c

Move visit directory service from dodal into blueapi

c8edb91

Fix tests, remove kwargs from context

44fe64a

Fix formatting

5a830c0

Make use of dodal directory provider singleton optional

c572c02

callumforrester force-pushed the visit-directory branch from 8c2e5c1 to c572c02 Compare October 24, 2023 13:56

Remove cloud vscode settings

49dd587

callumforrester commented Oct 24, 2023

View reviewed changes

callumforrester mentioned this pull request Oct 24, 2023

Use external service to configure ophyd-async detector data writing #315

Merged

callumforrester closed this Jul 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Speculative visit directory service #283

Speculative visit directory service #283

callumforrester commented Jul 31, 2023 •

edited

Loading

codecov bot commented Jul 31, 2023 •

edited

Loading

coretl Aug 8, 2023

callumforrester Aug 16, 2023

callumforrester Sep 4, 2023

DiamondJoseph Aug 14, 2023

callumforrester Oct 24, 2023

callumforrester Oct 2, 2023

callumforrester Oct 2, 2023

rosesyrett Oct 2, 2023

callumforrester Oct 2, 2023

rosesyrett Oct 3, 2023

rosesyrett Oct 4, 2023 •

edited

Loading

callumforrester Oct 4, 2023

coretl Oct 4, 2023

callumforrester Oct 2, 2023

DiamondJoseph Oct 2, 2023

callumforrester Oct 2, 2023

DiamondJoseph Oct 2, 2023

DiamondJoseph Oct 2, 2023

callumforrester Oct 2, 2023

callumforrester Oct 2, 2023

callumforrester Oct 2, 2023

keithralphs left a comment

keithralphs Oct 10, 2023

callumforrester Oct 24, 2023

callumforrester Oct 24, 2023

stan-dot commented Jul 22, 2024

callumforrester commented Jul 24, 2024

		DATA_COLLECTION_NUMBER = "data_collection_number"


		class DataCollectionProvider(ABC):

		wrapped_plan_generator = ctx.wrap(plan_generator)
		ctx.run_engine(wrapped_plan_generator)

		return DataCollectionIdentifier(collectionNumber=self._count)


		class VisitDirectoryProvider(DirectoryProvider):

Speculative visit directory service #283

Speculative visit directory service #283

Conversation

callumforrester commented Jul 31, 2023 • edited Loading

codecov bot commented Jul 31, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rosesyrett Oct 4, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

keithralphs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stan-dot commented Jul 22, 2024

callumforrester commented Jul 24, 2024

callumforrester commented Jul 31, 2023 •

edited

Loading

codecov bot commented Jul 31, 2023 •

edited

Loading

rosesyrett Oct 4, 2023 •

edited

Loading