Add SQL databases for CLEM workflow to Murfey #321

tieneupin · 2024-07-23T12:18:11Z

Part of continued development of issue #288. Also resolves issue #328.

Now that the CLEM workflow has grown in complexity, it is time to extend Murfey's database structure to record and keep track of the CLEM data collected and processed.

…propriate uint dtype

…reation, and added more parameters for more comprehensive metadata transfer during image stack creation

…database classes and which parts of the workflow they belong to

src/murfey/util/db.py

…QLAlchemy ValueError

…ign keys; updated relationships between tables

…propriate uint dtype

…reation, and added more parameters for more comprehensive metadata transfer during image stack creation

…database classes and which parts of the workflow they belong to

src/murfey/server/clem/api.py

…bspath() to resolve symlinks

src/murfey/server/clem/api.py

src/murfey/server/api/clem.py

d-j-hatton · 2024-08-22T15:16:24Z

src/murfey/cli/add_user.py imports from murfey.server.auth which is now murfey.server.api.auth

tieneupin · 2024-08-23T06:56:04Z

src/murfey/cli/add_user.py imports from murfey.server.auth which is now murfey.server.api.auth

Good spot, thank you! Fixed.

tieneupin · 2024-08-23T08:27:59Z

Checks:

murfey.create_db
murfey.add_user
murfey.server --demo
Registration of CLEM-related files

tieneupin added 8 commits July 3, 2024 15:47

Added custom NumPy DTypeError class; added function to return most ap…

c68c307

…propriate uint dtype

Merged recent changes from main branch

87f59bc

Added more comprehensive type hinting

971ae41

Added backwards compatibility of importlib module for Python 3.9

d23f8f5

Overhauled functions used for array manipulation during image stack c…

72fd16a

…reation, and added more parameters for more comprehensive metadata transfer during image stack creation

Updated type hinting for Python 3.9 compatibility

61e4d94

Added rudimentary docstrings to help with categorising the different …

c6c8815

…database classes and which parts of the workflow they belong to

Added initial framework for databases to be used in CLEM workflow

414de4f

tieneupin self-assigned this Jul 23, 2024

tieneupin requested a review from d-j-hatton July 23, 2024 12:18

tieneupin added enhancement New feature or request server Relates to the server component client Relates to the client component cryo-clem Part of the cryo-CLEM pipeline extension labels Jul 23, 2024

tieneupin changed the title ~~Add SQL databases for CLEM workflow~~ Add SQL databases for CLEM workflow to Murfey Jul 23, 2024

github-advanced-security bot found potential problems Jul 23, 2024

View reviewed changes

src/murfey/util/db.py Fixed Show fixed Hide fixed

src/murfey/util/db.py Fixed Show fixed Hide fixed

src/murfey/util/db.py Fixed Show fixed Hide fixed

src/murfey/util/db.py Fixed Show fixed Hide fixed

tieneupin added 6 commits July 23, 2024 17:16

Fleshed out the database classes for the CLEM workflow

23e3f52

Reverted to using List type hint from typing module to troubleshoot S…

e059af6

…QLAlchemy ValueError

Removed Optional typing from List type hint

9c0f446

Added more database structures and relationships for CLEM workflow

f872a06

Corrected incorrect types for some database fields

0f6dbea

Merging recent updates from main branch

f9dba79

tieneupin marked this pull request as ready for review July 24, 2024 16:09

tieneupin added 7 commits July 25, 2024 17:42

Merged CLEMSession and Session classes; used only unique IDs for fore…

89c4788

…ign keys; updated relationships between tables

Added custom NumPy DTypeError class; added function to return most ap…

748a63d

…propriate uint dtype

Overhauled functions used for array manipulation during image stack c…

0aa9aee

…reation, and added more parameters for more comprehensive metadata transfer during image stack creation

Updated type hinting for Python 3.9 compatibility

4355b29

Added rudimentary docstrings to help with categorising the different …

9eaad80

…database classes and which parts of the workflow they belong to

Added initial framework for databases to be used in CLEM workflow

ec71fd5

Fleshed out the database classes for the CLEM workflow

d1ded5d

Adjusted validation logic again

f8195bd

github-advanced-security bot found potential problems Aug 17, 2024

View reviewed changes

src/murfey/server/clem/api.py Fixed Show fixed Hide fixed

src/murfey/server/clem/api.py Fixed Show fixed Hide fixed

src/murfey/server/clem/api.py Fixed Show fixed Hide fixed

tieneupin added 2 commits August 17, 2024 18:42

Corrected messages raised messages

e2f9eb0

Tried validating and sanitising using os.path strings instead

aaa9121

github-advanced-security bot found potential problems Aug 17, 2024

View reviewed changes

src/murfey/server/clem/api.py Fixed Show fixed Hide fixed

Commented out function not being tested; used realpath() instead of a…

4d46291

…bspath() to resolve symlinks

github-advanced-security bot found potential problems Aug 17, 2024

View reviewed changes

src/murfey/server/clem/api.py Fixed Show fixed Hide fixed

src/murfey/server/clem/api.py Fixed Show fixed Hide fixed

src/murfey/server/clem/api.py Fixed Show fixed Hide fixed

src/murfey/server/clem/api.py Fixed Show fixed Hide fixed

Commented out file existence check

dbeb60e

github-advanced-security bot found potential problems Aug 17, 2024

View reviewed changes

src/murfey/server/clem/api.py Fixed Show fixed Hide fixed

tieneupin added 12 commits August 17, 2024 19:22

Use new sanitisation and validation function for code

9ccab2d

Updated code comments

4e78bb9

Fixed broken series-metadata registration

8225aee

Added docstrings and comments

e188dfa

Updated Field properties of 'parent_lif' column in CLEMImageMetadata

cc6432f

Refactored API endpoint modules to improve modularity

9cf6766

Adjusted comments and dependencies

e35f186

Replaced print messages and raised exceptions with logger instances

bb8d318

Updated dependency after API refactor

f201a90

Refactored more API endpoint modules2

911306a

Handle exception when default rsync base path is too short

f4e7792

Adjusted patch paths for server and API tests

2a55f7b

d-j-hatton reviewed Aug 22, 2024

View reviewed changes

src/murfey/server/api/clem.py Outdated Show resolved Hide resolved

Corrected broken import from API refactor

5468651

Removed unnecessary list length checks

2de40a1

d-j-hatton approved these changes Aug 23, 2024

View reviewed changes

tieneupin merged commit 557c1ea into main Aug 23, 2024
9 checks passed

tieneupin deleted the murfey-clem-db branch August 23, 2024 10:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add SQL databases for CLEM workflow to Murfey #321

Add SQL databases for CLEM workflow to Murfey #321

tieneupin commented Jul 23, 2024 •

edited

Loading

d-j-hatton commented Aug 22, 2024

tieneupin commented Aug 23, 2024

tieneupin commented Aug 23, 2024 •

edited

Loading

Add SQL databases for CLEM workflow to Murfey #321

Add SQL databases for CLEM workflow to Murfey #321

Conversation

tieneupin commented Jul 23, 2024 • edited Loading

d-j-hatton commented Aug 22, 2024

tieneupin commented Aug 23, 2024

tieneupin commented Aug 23, 2024 • edited Loading

tieneupin commented Jul 23, 2024 •

edited

Loading

tieneupin commented Aug 23, 2024 •

edited

Loading