Parsing and categorization

[rmanaem]

Outcome

The model should be able to parse the uploaded participants.tsv and data dictionary (if there is one since it's optional in the annotation tool) to

• describe what columns there are in the tsv
• validate those against the ones in the data dictionary (if there is one)
• map columns to Neurobagel annotating categories
  • the challenge here is most likely diagnosis and assessment tools
  • getting automated coding on participant_id, age, and sex w/ an LLM agent would be the simplest case
  • temporal / longitudinal fields (session / visit) should probably be ignored for now

Resources:

• Hard coded annotation categories
• Example files used for testing in annotation tool
• An organization containing annotated datasets

Utilities

• llama3
• Gemma
• hand over each column separately to the model

Tasks

• column header extraction of .tsv #7
• create raw .json with column headers #10
• Define LLM output (Json output )parser #12
• categorization of participant_id and session_id #13
• categorization of ageand sex #20
• categorization of diagnosis and assessment_tool #22
• Refinement of annotations written to .json #17
• Create json annotations for Diagnosis #19
• Create json annotations for Assessment Tool #23
• Add tests (restructure repo) #37