Provide the full path on the disk for autoprocessing output

[gustalima]

Hi,

I would like to ask if it is possible to add the autoprocessing full path (including the uuid from the cluster pid) with in the API

queryURL: https://ispyb.diamond.ac.uk/api/processing/?prop=

Current response:

{
    "TYPE": "autoPROC+STARANISO",
    "IMAGESWEEPCOUNT": "1",
    "PROCESSINGSTATUS": "1",
    "PROCESSINGMESSAGE": "processing successful",
    "DCCOUNT": "1",
    "PROCESSINGJOBID": "########",
    "BEAM": {
        "XBEAM": "159.03",
        "YBEAM": "166.58",
        "REFINEDXBEAM": "166.64",
        "REFINEDYBEAM": "159.10"
    },
    "SHELLS": {
        "outerShell": {
            "CCHALF": "0.7",
            "CCANOMALOUS": "0.1",
            "ANOMALOUS": null,
            "TYPE": "autoPROC+STARANISO",
            "NTOBS": "8253",
            "NUOBS": "184",
            "RLOW": "2.79",
            "RHIGH": "2.52",
            "RMEAS": "8.357",
            "RMERGE": "8.263",
            "COMPLETENESS": "63.7",
            "ANOMCOMPLETENESS": "64.5",
            "ANOMMULTIPLICITY": "25.8",
            "MULTIPLICITY": "44.9",
            "ISIGI": "0.7",
            "RESISIGI": "0.00",
            "PROCESSINGSTATUS": "1",
            "PROCESSINGMESSAGE": "processing successful",
            "DCCOUNT": "1",
            "AUTOMATIC": "1",
            "INDEX": 0
        },
        "innerShell": {
            "CCHALF": "1.0",
            "CCANOMALOUS": "0.1",
            "ANOMALOUS": null,
            "TYPE": "autoPROC+STARANISO",
            "NTOBS": "7492",
            "NUOBS": "184",
            "RLOW": "28.59",
            "RHIGH": "4.97",
            "RMEAS": "0.063",
            "RMERGE": "0.062",
            "COMPLETENESS": "99.5",
            "ANOMCOMPLETENESS": "100.0",
            "ANOMMULTIPLICITY": "29.3",
            "MULTIPLICITY": "40.7",
            "ISIGI": "38.5",
            "RESISIGI": "0.00",
            "PROCESSINGSTATUS": "1",
            "PROCESSINGMESSAGE": "processing successful",
            "DCCOUNT": "1",
            "AUTOMATIC": "1",
            "INDEX": 1
        },
        "overall": {
            "CCHALF": "1.0",
            "CCANOMALOUS": "0.1",
            "ANOMALOUS": null,
            "TYPE": "autoPROC+STARANISO",
            "NTOBS": "50951",
            "NUOBS": "1106",
            "RLOW": "28.59",
            "RHIGH": "2.52",
            "RMEAS": "0.220",
            "RMERGE": "0.217",
            "COMPLETENESS": "91.1",
            "ANOMCOMPLETENESS": "90.1",
            "ANOMMULTIPLICITY": "28.4",
            "MULTIPLICITY": "46.1",
            "ISIGI": "14.4",
            "RESISIGI": "0.00",
            "PROCESSINGSTATUS": "1",
            "PROCESSINGMESSAGE": "processing successful",
            "DCCOUNT": "1",
            "AUTOMATIC": "1",
            "INDEX": 2
        }
    },
    "CELL": {
        "CELL_A": "####",
        "CELL_B": "####",
        "CELL_C": "####",
        "CELL_AL": "####",
        "CELL_BE": "####",
        "CELL_GA": "####"
    },
    "SG": "########",
    "AID": "#####",
    "MESSAGES": [],
    "AUTOMATIC": true
}

Addition

PROCESS_PATH: "/dls/##/data/2023/######/processed/#/#######/###/<pid#>/xia2-dials"

That would reduce the checks to programatically get the download folder for each samepl/autoprocess.

Best,
Gustavo

[ndg63276]

If it helps, you shouldn't need the pid# in the path, there should be a symlink eg:

xia2-dials -> 48b46f5c-b3c1-4eb8-95f4-a8636eeb072f/xia2-dials

[gustalima]

Hi Mark,

Thanks for the reply.

The issue I have with the suggested path is when we have autoPROC+STARANISO and aslo a autoPROC processing with different space groups (here I mean without any user input, when a second autoPROC process is trieggered automatically).

ISPyB will give me the response with

'TYPE': 'autoPROC+STARANISO',

if I query

https://ispyb.diamond.ac.uk/api/processing/<data_collection_id>?prop=<proposal>

But the path does not resolve (as it is the same as autoPROC, but pointing to the STARANISO files).

In more general terms, the response from the referred query does not point me to the correct folder in all cases (if you check sid=3367658, you will see one example of this).

In any case, I understand this might not be a priority as it is a very niche use. SynchWeb API is already very good.

[ndg63276]

Sorry, I might be being stupid but what do you mean by sid=3367658? SessionId, sampleId, dataCollectionId? Something else?

[gustalima]

Hi Mark,

sorry, I meant sampleId
https://ispyb.diamond.ac.uk/samples/sid/

[ndg63276]

Thanks, didnt want to look at the wrong data collections.

Technically, there is a filePath field for each AutoProcProgramAttachment, but I think in reality every attachment for a particular processing job will have the same path.

I've made internal ticket LIMS-922 to encourage some work on this.

[gustalima]

Thank you very much, Mark. I really appreciate it.

[ndg63276]

Just having another look at this, if you take the "AID" value from the original request, you can then query

https://ispyb.diamond.ac.uk/api/download/ap/attachments?prop=<proposal>&AUTOPROCPROGRAMID=<AID>

you will get a list returned (of each attachment to the autoprocessing). Each item in the list will have a "FILEPATH" value, which I think should be what you want.