Skip to content
This repository has been archived by the owner on Sep 2, 2024. It is now read-only.

Testing v9.5.x #1465

Closed
dperl-dls opened this issue Jun 25, 2024 · 10 comments
Closed

Testing v9.5.x #1465

dperl-dls opened this issue Jun 25, 2024 · 10 comments
Assignees

Comments

@dperl-dls
Copy link
Collaborator

dperl-dls commented Jun 25, 2024

  • on first startup, stuck at init'ing dodal devices
    • hanging on `[2024-06-25 14:17:43,299] Hyperion context DEBUG: Looking for device panda of type <class 'ophyd_async.panda._hdf_panda.HDFPanda'> in context
    • trying to connect to panda by hand gives
>>> await p.connect()
Traceback (most recent call last):
  File "/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "<console>", line 1, in <module>
  File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/panda/_hdf_panda.py", line 44, in connect
    await fill_pvi_entries(self, self._prefix + "PVI", timeout=timeout, mock=mock)
  File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/epics/pvi/pvi.py", line 298, in fill_pvi_entries
    await _get_pvi_entries(root_entry, timeout=timeout)
  File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/epics/pvi/pvi.py", line 254, in _get_pvi_entries
    await _get_pvi_entries(sub_entry)
  File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/epics/pvi/pvi.py", line 257, in _get_pvi_entries
    _verify_common_blocks(entry, entry.common_device_type)
  File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/epics/pvi/pvi.py", line 99, in _verify_common_blocks
    raise RuntimeError(
RuntimeError: sub device `datasets:<class 'typing._GenericAlias'>` was not provided by pvi
  • after commenting out panda, sticks on robot
r = i03.robot()
ERROR:root:device `error_code` raised unexpected exception TypeError
Traceback (most recent call last):
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/core/signal.py", line 88, in connect
   await self._backend.connect(timeout=timeout)
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/epics/_backend/_aioca.py", line 279, in connect
   self.converter = make_converter(self.datatype, self.initial_values)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/epics/_backend/_aioca.py", line 228, in make_converter
   raise TypeError(
TypeError: BL03I-MO-ROBOT-01:PRG_ERR_CODE has type float not int
Traceback (most recent call last):
 File "/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/concurrent/futures/_base.py", line 456, in result
   return self.__get_result()
          ^^^^^^^^^^^^^^^^^^^
 File "/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
   raise self._exception
 File "/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/asyncio/__main__.py", line 34, in callback
   coro = func()
          ^^^^^^
 File "<console>", line 1, in <module>
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/dodal/src/dodal/beamlines/i03.py", line 461, in robot
   return device_instantiation(
          ^^^^^^^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/dodal/src/dodal/utils.py", line 102, in wrapper
   return func(*args, **kwds)
          ^^^^^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/dodal/src/dodal/common/beamlines/beamline_utils.py", line 112, in device_instantiation
   wait_for_connection(device_instance, mock=fake)
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/dodal/src/dodal/common/beamlines/beamline_utils.py", line 55, in wait_for_connection
   call_in_bluesky_event_loop(
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/run_engine.py", line 2782, in call_in_bluesky_event_loop
   return fut.result(timeout=timeout)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/concurrent/futures/_base.py", line 456, in result
   return self.__get_result()
          ^^^^^^^^^^^^^^^^^^^
 File "/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
   raise self._exception
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/core/utils.py", line 129, in wait_for_connection
   raise NotConnected(exceptions)
ophyd_async.core.utils.NotConnected: 
coros: NotConnected:
   error_code: TypeError: BL03I-MO-ROBOT-01:PRG_ERR_CODE has type float not int
[2024-06-25 15:40:28,380] bluesky run_engine ERROR: Run aborted
Traceback (most recent call last):
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/run_engine.py", line 1547, in _run
   msg = self._plan_stack[-1].send(resp)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/device_setup_plans/utils.py", line 41, in start_preparing_data_collection_then_do_plan
   yield from bpp.contingency_wrapper(
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 606, in contingency_wrapper
   ret = yield from plan
         ^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/device_setup_plans/utils.py", line 39, in wrapped_plan
   yield from plan_to_run
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/experiment_plans/pin_centre_then_xray_centre_plan.py", line 73, in pin_centre_then_xray_centre_plan
   yield from detect_grid_and_do_gridscan(
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/experiment_plans/grid_detect_then_xray_centre_plan.py", line 113, in detect_grid_and_do_gridscan
   yield from ispyb_activation_wrapper(
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 351, in run_wrapper
   yield from contingency_wrapper(plan,
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 606, in contingency_wrapper
   ret = yield from plan
         ^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/experiment_plans/grid_detect_then_xray_centre_plan.py", line 192, in _detect_grid_and_do_gridscan
   yield from flyscan_xray_centre(
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/utils/__init__.py", line 1203, in dec_inner
   return (yield from plan)
           ^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 1380, in set_run_key_wrapper
   return (yield from msg_mutator(plan, _set_run_key))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 263, in msg_mutator
   msg = plan.send(_s)
         ^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/utils/__init__.py", line 1203, in dec_inner
   return (yield from plan)
           ^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 351, in run_wrapper
   yield from contingency_wrapper(plan,
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 606, in contingency_wrapper
   ret = yield from plan
         ^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 675, in dec_inner
   ret = yield from plan
         ^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/utils/__init__.py", line 1203, in dec_inner
   return (yield from plan)
           ^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/device_setup_plans/xbpm_feedback.py", line 81, in transmission_and_xbpm_feedback_for_collection_wrapper
   yield from finalize_wrapper(
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 535, in finalize_wrapper
   ret = yield from plan
         ^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/device_setup_plans/xbpm_feedback.py", line 78, in _inner_plan
   return (yield from plan)
           ^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/experiment_plans/flyscan_xray_centre_plan.py", line 382, in run_gridscan_and_move_and_tidy
   yield from run_gridscan_and_move(fgs_composite, params)
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/utils/__init__.py", line 1203, in dec_inner
   return (yield from plan)
           ^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 1380, in set_run_key_wrapper
   return (yield from msg_mutator(plan, _set_run_key))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 263, in msg_mutator
   msg = plan.send(_s)
         ^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/utils/__init__.py", line 1203, in dec_inner
   return (yield from plan)
           ^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 351, in run_wrapper
   yield from contingency_wrapper(plan,
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 606, in contingency_wrapper
   ret = yield from plan
         ^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/experiment_plans/flyscan_xray_centre_plan.py", line 299, in run_gridscan_and_move
   yield from run_gridscan(fgs_composite, parameters)
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/utils/__init__.py", line 1203, in dec_inner
   return (yield from plan)
           ^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 1380, in set_run_key_wrapper
   return (yield from msg_mutator(plan, _set_run_key))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 263, in msg_mutator
   msg = plan.send(_s)
         ^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/utils/__init__.py", line 1203, in dec_inner
   return (yield from plan)
           ^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 351, in run_wrapper
   yield from contingency_wrapper(plan,
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 606, in contingency_wrapper
   ret = yield from plan
         ^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/experiment_plans/flyscan_xray_centre_plan.py", line 236, in run_gridscan
   yield from read_hardware_for_ispyb_pre_collection(
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/src/hyperion/device_setup_plans/read_hardware_for_setup.py", line 35, in read_hardware_for_ispyb_pre_collection
   yield from bps.read(aperture_scatterguard)
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/plan_stubs.py", line 120, in read
   return (yield Msg('read', obj))
           ^^^^^^^^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 251, in msg_mutator
   _s = yield msg
        ^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 251, in msg_mutator
   _s = yield msg
        ^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 251, in msg_mutator
   _s = yield msg
        ^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/run_engine.py", line 1607, in _run
   new_response = await coro(msg)
                  ^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/run_engine.py", line 1889, in _read
   ret = await maybe_await(obj.read(*msg.args, **msg.kwargs))
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/bluesky/utils/__init__.py", line 1852, in maybe_await
   return await ret
          ^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/core/standard_readable.py", line 96, in read
   return await merge_gathered_dicts([sig.read() for sig in self._readables])
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/core/utils.py", line 174, in merge_gathered_dicts
   for result in await asyncio.gather(*coros):
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/hyperion/.venv/lib/python3.11/site-packages/ophyd_async/core/standard_readable.py", line 246, in read
   return await self.signal.read(cached=self.cached)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/dodal/src/dodal/devices/aperturescatterguard.py", line 179, in read
   await self._backend.put(await self.parent._get_current_aperture_position())
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/dls_sw/i03/software/bluesky/hyperion_v9.5.0/dodal/src/dodal/devices/aperturescatterguard.py", line 252, in _get_current_aperture_position
   raise InvalidApertureMove(
dodal.devices.aperturescatterguard.InvalidApertureMove: Current aperture/scatterguard state unrecognised at current_ap_y=34.4056
@olliesilvester
Copy link
Contributor

olliesilvester commented Jun 26, 2024

Looking at the panda issue, i03's PandablocksIOC is out of date with our ophyd_async since we were running the ioc from a local install from when main was broken. I can update it, and try and find out how to make sure it updates automatically

@DominicOram
Copy link
Collaborator

If we update it will it break old hyperion versions?

@olliesilvester
Copy link
Contributor

olliesilvester commented Jun 26, 2024

It definitely could, but I couldn't say for certain without having a deep dive into all the panda server + ioc changes

It would only break panda related things though

@DominicOram
Copy link
Collaborator

I think the float/int issue should eventually be fixed in ophyd-async, see bluesky/ophyd-async#280. I think we make the change in dodal now though to expedite release.

@DominicOram
Copy link
Collaborator

Proposal is to come down every day and try this

@DominicOram
Copy link
Collaborator

DominicOram commented Jul 11, 2024

Testing 11/07

@DominicOram
Copy link
Collaborator

DominicOram commented Jul 12, 2024

Testing 12/07

[31m[E 240712 10:12:17.954 client:84] Encountered an error reading from the config service.�[0m
Traceback (most recent call last):
  File "/dls_sw/i03/software/bluesky/hyperion_v9.5.1/hyperion/.venv/lib/python3.11/site-packages/daq_config_server/client.py", line 82, in best_effort_get_all_feature_flags
    return self._get(ENDPOINTS.FEATURE, options={"get_values": "true"})
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dls_sw/i03/software/bluesky/hyperion_v9.5.1/hyperion/.venv/lib/python3.11/site-packages/daq_config_server/client.py", line 41, in _get
    assert item in body, f"Malformed response: {body} does not contain {item}"
           ^^^^^^^^^^^^
AssertionError: Malformed response: {'set_stub_offsets': False, 'use_panda_for_gridscan': True, 'use_gpu_for_gridscan': True} does not contain None

@dperl-dls
Copy link
Collaborator Author

feature flag issue should be fixed by DiamondLightSource/daq-config-server#49

@DominicOram
Copy link
Collaborator

Testing 15/07

  • Timeouts due to file system issues - fixed by increase odin meta timeout to 30 dodal#671
  • Data seemed correct for two full run throughs of the test puck and once through a puck of very small crystals, calling it stable

@DominicOram
Copy link
Collaborator

Tagging for review @dperl-dls, @olliesilvester or @rtuck99.

To review:

  • Confirm that all points made above
  • Confirm there are no changes in /dls_sw/i03/software/bluesky/hyperion_v9.5.1 that are not covered above

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
Archived in project
Development

No branches or pull requests

3 participants