Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: update README instructions #91

Merged
merged 5 commits into from
Apr 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 33 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,43 @@
# aind-data-transfer-service
FastAPI app to upload data transfer jobs.

[![License](https://img.shields.io/badge/license-MIT-brightgreen)](LICENSE)
![Code Style](https://img.shields.io/badge/code%20style-black-black)
[![semantic-release: angular](https://img.shields.io/badge/semantic--release-angular-e10079?logo=semantic-release)](https://github.com/semantic-release/semantic-release)

This service can be used to upload data stored in a VAST drive. It uses FastAPI to upload a job submission csv file that will be used to trigger a data transfer job in an on-prem HPC. Based on the information provided in the file, the data upload process fetches the appropriate metadata and starts the upload process.

## Metadata Sources

The associated metadata files get pulled from different sources.

- subject from LabTracks
- procedures from NSB Sharepoint, TARS
- instrument/rig from SLIMS


## Usage
It's possible to submit a job via the python api. Here is an example script that can be used.

There are two options for uploading data: a python API or a browser UI service.

### Browser UI
You can go to http://aind-data-transfer-service to submit a `.csv` or `.xlsx` file with the necessary parameters needed to launch a data upload job. Click on **Job Submit Template** to download a template which you may use as a reference.

What each column means in the job submission template:

- **platform**: For a list of platforms click [here](https://github.com/AllenNeuralDynamics/aind-data-schema/blob/main/src/aind_data_schema/models/platforms.py).
- **acq_datetime**: The time that the data was acquired
- **subject_id**: The unique id of the subject
- **modality0**: For a list of modalities, click [here](https://github.com/AllenNeuralDynamics/aind-data-schema/blob/main/src/aind_data_schema/models/modalities.py).
- **modality0.source**: The source (path to file) of **modality0** in VAST drive

Modify the job template as needed and click on **Browse** to upload the file. A rendered table with a message **Successfully validated jobs from file** appears to indicate a valid file. If there are errors in the job submit file, a message that says **Error validating jobs from file** appears.

To launch a data upload job, click on `Submit`. A message that says **Successfuly submitted jobs** should appear.

After submission, click on `Job Status` to see the status of the data upload job process.

### Python API
It's also possible to submit a job via a python api. Here is an example script that can be used.

Assuming that the data on a shared drive is organized as:
```
Expand Down
3 changes: 2 additions & 1 deletion src/aind_data_transfer_service/templates/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,8 @@
<nav>
<a href="/">Submit Jobs</a> |
<a href="/jobs">Job Status</a> |
<a title="Download job template as .xslx" href= "/api/job_upload_template" download>Job Submit Template</a>
<a title="Download job template as .xslx" href= "/api/job_upload_template" download>Job Submit Template</a> |
<a title="For more information click here" href="https://github.com/AllenNeuralDynamics/aind-data-transfer-service/tree/main?tab=readme-ov-file#aind-data-transfer-service" target="_blank" >Help</a>
</nav>
<h2>Submit Jobs</h2>
<div>
Expand Down
Loading