Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Send LIF to TIFF conversion to a cluster #287

Merged
merged 4 commits into from
May 24, 2024
Merged

Send LIF to TIFF conversion to a cluster #287

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c32a46b
Add workflows endpoint to inject workflow triggering
d-j-hatton May 23, 2024
a7c426c
Add Zocalo cluster submission workflow for LIF to TIFF conversion
d-j-hatton May 23, 2024
3f476ba
Recipe name change
d-j-hatton May 23, 2024
9d5ab49
Add clarifying comment
d-j-hatton May 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 75 additions & 0 deletions logfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
2024-05-09 09:10:03.740 BST [4144] LOG: starting PostgreSQL 15.1 on aarch64-apple-darwin20.0.0, compiled by clang version 14.0.6, 64-bit
2024-05-09 09:10:03.742 BST [4144] LOG: listening on IPv4 address "127.0.0.1", port 5432
2024-05-09 09:10:03.742 BST [4144] LOG: listening on IPv6 address "::1", port 5432
2024-05-09 09:10:03.752 BST [4144] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2024-05-09 09:10:03.771 BST [4147] LOG: database system was shut down at 2024-04-18 00:09:47 BST
2024-05-09 09:10:03.798 BST [4144] LOG: database system is ready to accept connections
2024-05-09 09:10:11.460 BST [4184] FATAL: password authentication failed for user "kif41228"
2024-05-09 09:10:11.460 BST [4184] DETAIL: Password does not match for user "kif41228".
Connection matched pg_hba.conf line 87: "host all all 127.0.0.1/32 md5"
2024-05-09 09:12:06.698 BST [4442] ERROR: null value in column "visit" of relation "session" violates not-null constraint
2024-05-09 09:12:06.698 BST [4442] DETAIL: Failing row contains (2, Client connection, null, null).
2024-05-09 09:12:06.698 BST [4442] STATEMENT: INSERT INTO session (name) VALUES ('Client connection') RETURNING session.id
2024-05-09 09:12:07.026 BST [4447] ERROR: column rsyncinstance.client_id does not exist at character 110
2024-05-09 09:12:07.026 BST [4447] STATEMENT: SELECT rsyncinstance.source AS rsyncinstance_source, rsyncinstance.destination AS rsyncinstance_destination, rsyncinstance.client_id AS rsyncinstance_client_id, rsyncinstance.files_transferred AS rsyncinstance_files_transferred, rsyncinstance.files_counted AS rsyncinstance_files_counted, rsyncinstance.transferring AS rsyncinstance_transferring
FROM rsyncinstance
WHERE 0 = rsyncinstance.client_id
2024-05-09 09:15:03.771 BST [4145] LOG: checkpoint starting: time
2024-05-09 09:16:21.945 BST [4145] LOG: checkpoint complete: wrote 199 buffers (1.2%); 0 WAL file(s) added, 0 removed, 0 recycled; write=78.122 s, sync=0.006 s, total=78.174 s; sync files=69, longest=0.001 s, average=0.001 s; distance=1479 kB, estimate=1479 kB
2024-05-09 09:20:03.953 BST [4145] LOG: checkpoint starting: time
2024-05-09 09:20:05.094 BST [4145] LOG: checkpoint complete: wrote 12 buffers (0.1%); 0 WAL file(s) added, 0 removed, 0 recycled; write=1.125 s, sync=0.004 s, total=1.142 s; sync files=9, longest=0.002 s, average=0.001 s; distance=37 kB, estimate=1335 kB
2024-05-09 09:25:03.096 BST [4145] LOG: checkpoint starting: time
2024-05-09 09:25:03.636 BST [4145] LOG: checkpoint complete: wrote 6 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.509 s, sync=0.020 s, total=0.540 s; sync files=6, longest=0.018 s, average=0.004 s; distance=4 kB, estimate=1202 kB
2024-05-09 09:30:03.638 BST [4145] LOG: checkpoint starting: time
2024-05-09 09:30:04.161 BST [4145] LOG: checkpoint complete: wrote 6 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.511 s, sync=0.003 s, total=0.524 s; sync files=6, longest=0.001 s, average=0.001 s; distance=2 kB, estimate=1082 kB
2024-05-09 09:35:03.164 BST [4145] LOG: checkpoint starting: time
2024-05-09 09:35:03.696 BST [4145] LOG: checkpoint complete: wrote 6 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.512 s, sync=0.003 s, total=0.532 s; sync files=6, longest=0.002 s, average=0.001 s; distance=4 kB, estimate=974 kB
2024-05-09 09:46:13.109 BST [4145] LOG: checkpoint starting: time
2024-05-09 09:46:13.984 BST [4145] LOG: checkpoint complete: wrote 9 buffers (0.1%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.849 s, sync=0.003 s, total=0.876 s; sync files=9, longest=0.002 s, average=0.001 s; distance=17 kB, estimate=878 kB
2024-05-09 09:51:13.982 BST [4145] LOG: checkpoint starting: time
2024-05-09 09:51:14.737 BST [4145] LOG: checkpoint complete: wrote 8 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.724 s, sync=0.003 s, total=0.755 s; sync files=9, longest=0.001 s, average=0.001 s; distance=20 kB, estimate=793 kB
2024-05-09 10:16:13.735 BST [4145] LOG: checkpoint starting: time
2024-05-09 10:16:14.286 BST [4145] LOG: checkpoint complete: wrote 6 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.520 s, sync=0.003 s, total=0.552 s; sync files=6, longest=0.001 s, average=0.001 s; distance=2 kB, estimate=713 kB
2024-05-09 10:26:13.313 BST [4145] LOG: checkpoint starting: time
2024-05-09 10:26:13.876 BST [4145] LOG: checkpoint complete: wrote 6 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.522 s, sync=0.003 s, total=0.564 s; sync files=6, longest=0.002 s, average=0.001 s; distance=2 kB, estimate=642 kB
2024-05-09 10:41:13.885 BST [4145] LOG: checkpoint starting: time
2024-05-09 10:41:14.429 BST [4145] LOG: checkpoint complete: wrote 6 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.520 s, sync=0.002 s, total=0.544 s; sync files=6, longest=0.001 s, average=0.001 s; distance=2 kB, estimate=578 kB
2024-05-09 10:56:13.437 BST [4145] LOG: checkpoint starting: time
2024-05-09 10:56:13.978 BST [4145] LOG: checkpoint complete: wrote 6 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.521 s, sync=0.002 s, total=0.542 s; sync files=6, longest=0.001 s, average=0.001 s; distance=4 kB, estimate=521 kB
2024-05-09 11:01:13.974 BST [4145] LOG: checkpoint starting: time
2024-05-09 11:01:14.501 BST [4145] LOG: checkpoint complete: wrote 6 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.508 s, sync=0.003 s, total=0.528 s; sync files=6, longest=0.001 s, average=0.001 s; distance=3 kB, estimate=469 kB
2024-05-09 18:10:32.856 BST [4144] LOG: received smart shutdown request
2024-05-09 18:10:32.861 BST [4144] LOG: background worker "logical replication launcher" (PID 4150) exited with exit code 1
2024-05-09 18:10:32.861 BST [4145] LOG: shutting down
2024-05-09 18:10:32.862 BST [4145] LOG: checkpoint starting: shutdown immediate
2024-05-09 18:10:32.870 BST [4145] LOG: checkpoint complete: wrote 0 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.002 s, sync=0.001 s, total=0.010 s; sync files=0, longest=0.000 s, average=0.000 s; distance=0 kB, estimate=422 kB
2024-05-09 18:10:32.881 BST [4144] LOG: database system is shut down
2024-05-18 20:06:25.662 BST [18530] LOG: starting PostgreSQL 15.1 on aarch64-apple-darwin20.0.0, compiled by clang version 14.0.6, 64-bit
2024-05-18 20:06:25.665 BST [18530] LOG: listening on IPv4 address "127.0.0.1", port 5432
2024-05-18 20:06:25.665 BST [18530] LOG: listening on IPv6 address "::1", port 5432
2024-05-18 20:06:25.667 BST [18530] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2024-05-18 20:06:25.682 BST [18533] LOG: database system was shut down at 2024-05-16 18:11:47 BST
2024-05-18 20:06:25.701 BST [18530] LOG: database system is ready to accept connections
2024-05-18 20:11:25.681 BST [18531] LOG: checkpoint starting: time
2024-05-18 20:11:25.730 BST [18531] LOG: checkpoint complete: wrote 2 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.021 s, sync=0.007 s, total=0.050 s; sync files=3, longest=0.007 s, average=0.003 s; distance=0 kB, estimate=0 kB
2024-05-18 22:01:25.746 BST [18531] LOG: checkpoint starting: time
2024-05-18 22:01:48.742 BST [18531] LOG: checkpoint complete: wrote 226 buffers (1.4%); 0 WAL file(s) added, 0 removed, 0 recycled; write=22.935 s, sync=0.007 s, total=22.996 s; sync files=70, longest=0.001 s, average=0.001 s; distance=1666 kB, estimate=1666 kB
2024-05-18 22:14:16.494 BST [23372] ERROR: duplicate key value violates unique constraint "user_pkey"
2024-05-18 22:14:16.494 BST [23372] DETAIL: Key (username)=(murfey) already exists.
2024-05-18 22:14:16.494 BST [23372] STATEMENT: INSERT INTO "user" (username, hashed_password) VALUES ('murfey', '$2b$12$xkM0RPF2WO6XRFMBp/dKD.0C/8VKIdbnuknly.YyEEwemcgzrx3M2')
2024-05-18 22:16:21.359 BST [23489] ERROR: relation "murfeyuser" does not exist at character 13
2024-05-18 22:16:21.359 BST [23489] STATEMENT: INSERT INTO murfeyuser (username, hashed_password) VALUES ('murfey', '$2b$12$MAw4sOnG4/x18Hg4gFnFXeL1G4Mzh1OPXoH79HlUyL130L2opp2De')
2024-05-18 22:16:25.748 BST [18531] LOG: checkpoint starting: time
2024-05-18 22:16:26.205 BST [18531] LOG: checkpoint complete: wrote 5 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.438 s, sync=0.002 s, total=0.458 s; sync files=4, longest=0.001 s, average=0.001 s; distance=8 kB, estimate=1500 kB
2024-05-18 22:21:25.207 BST [18531] LOG: checkpoint starting: time
2024-05-18 22:21:46.496 BST [18531] LOG: checkpoint complete: wrote 209 buffers (1.3%); 0 WAL file(s) added, 0 removed, 0 recycled; write=21.236 s, sync=0.007 s, total=21.289 s; sync files=71, longest=0.002 s, average=0.001 s; distance=1599 kB, estimate=1599 kB
2024-05-18 22:36:25.518 BST [18531] LOG: checkpoint starting: time
2024-05-18 22:36:25.865 BST [18531] LOG: checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.316 s, sync=0.002 s, total=0.348 s; sync files=2, longest=0.001 s, average=0.001 s; distance=12 kB, estimate=1440 kB
2024-05-19 00:44:29.430 BST [18531] LOG: checkpoint starting: time
2024-05-19 00:44:30.930 BST [18531] LOG: checkpoint complete: wrote 15 buffers (0.1%); 0 WAL file(s) added, 0 removed, 0 recycled; write=1.469 s, sync=0.003 s, total=1.501 s; sync files=9, longest=0.002 s, average=0.001 s; distance=41 kB, estimate=1300 kB
2024-05-19 00:49:30.684 BST [18530] LOG: received smart shutdown request
2024-05-19 00:49:30.685 BST [18530] LOG: background worker "logical replication launcher" (PID 18536) exited with exit code 1
2024-05-19 00:49:30.686 BST [18531] LOG: shutting down
2024-05-19 00:49:30.686 BST [18531] LOG: checkpoint starting: shutdown immediate
2024-05-19 00:49:30.691 BST [18531] LOG: checkpoint complete: wrote 0 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.001 s, total=0.006 s; sync files=0, longest=0.000 s, average=0.000 s; distance=0 kB, estimate=1170 kB
2024-05-19 00:49:30.703 BST [18530] LOG: database system is shut down
2 changes: 2 additions & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,8 @@ murfey = "murfey.client:run"
"murfey.spa_inject" = "murfey.cli.inject_spa_processing:run"
"murfey.spa_ispyb_entries" = "murfey.cli.spa_ispyb_messages:run"
"murfey.transfer" = "murfey.cli.transfer:run"
[project.entry-points."murfey.workflows"]
"lif_to_tiff" = "murfey.workflows.lif_to_tiff:zocalo_cluster_request"

[tool.setuptools]
package-dir = {"" = "src"}
Expand Down
17 changes: 14 additions & 3 deletions src/murfey/server/clem/api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
from __future__ import annotations

import importlib.metadata

from fastapi import APIRouter

from murfey.server import _transport_object
from murfey.util.lif import convert_lif_to_tiff
from murfey.util.models import LifFileInfo

Expand All @@ -15,7 +18,15 @@ def lif_to_tiff(
session_id: int, # Used by the decorator
lif_info: LifFileInfo,
):
convert_lif_to_tiff(
file=lif_info.name,
root_folder="images",
murfey_workflows = importlib.metadata.entry_points().select(
group="murfey.workflows", name="lif_to_tiff"
)
if murfey_workflows:
murfey_workflows[0].load()(
file=lif_info.name, root_folder="images", messenger=_transport_object
)
else:
convert_lif_to_tiff(
file=lif_info.name,
root_folder="images",
)
Empty file added src/murfey/workflows/__init__.py
View file
Open in desktop
Empty file.
24 changes: 24 additions & 0 deletions src/murfey/workflows/lif_to_tiff.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
from pathlib import Path

try:
from murfey.server.ispyb import TransportManager # Session
except AttributeError:
Fixed Show fixed Hide fixed
pass


def zocalo_cluster_request(
file: Path, root_folder: str, messenger: TransportManager | None = None
):
if messenger:
messenger.send(
"processing_recipe",
{
"recipes": ["lif-to-tiff"],
"parameters": {
"working_dir": str(file.parent.parent),
"lif_path": str(file),
"root_dir": root_folder,
},
},
new_connection=True,
)