Code

PubMed Download Pipeline#246

Code

Draft

star-nox wants to merge 33 commits intomainUIUC-Chatbot/ai-ta-backend:mainfrom pubmedUIUC-Chatbot/ai-ta-backend:pubmedCopy head branch name to clipboard

+859

Commits

Commits on Apr 2, 2024

• added functions for metadata extraction

  star-nox

  committed
  aefeac1

Commits on Apr 3, 2024

• completed all download functions

  star-nox

  committed
  b833524

• added supabase upsert

  star-nox

  committed
  8f14cf9

• updated comments

  star-nox

  committed
  6daa4df

Commits on Apr 4, 2024

• added processpool to extractMetadataFromXML()

  star-nox

  committed
  c8c9056

Commits on Apr 6, 2024

• minor changes

  star-nox

  committed
  f8a4d23

Commits on Apr 8, 2024

• Merge branch 'main' into pubmed

  star-nox

  committed
  84fd182

• yielded metadata after collecting 100 articles

  star-nox

  committed
  fa83ddf

Commits on Apr 9, 2024

• Merge branch 'main' into pubmed

  star-nox

  committed
  88a951b

Commits on Apr 11, 2024

• storing metadata into csv and upserting per XML file

  star-nox

  committed
  64a4142

Commits on Apr 12, 2024

• parallelized metadata update

  star-nox

  committed
  24425d4

• added minio to requirements.txt

  star-nox

  committed
  3ddec3b

• parallelized download

  star-nox

  committed
  e03fbf1

Commits on Apr 15, 2024

• parallelized upload

  star-nox

  committed
  8e5a1a0

Commits on Apr 17, 2024

• minor changes

  star-nox

  committed
  caac0bf

Commits on Apr 19, 2024

• restricted upload parallelization to 10

  star-nox

  committed
  fa60508

Commits on Apr 21, 2024

• changed starting XML file

  star-nox

  committed
  a61255c

Commits on Apr 22, 2024

• changed starting XML file

  star-nox

  committed
  4d86b85

Commits on Apr 29, 2024

• changed start file

  star-nox

  committed
  63a6cb6

• minor comment

  star-nox

  committed
  b17baea

Commits on May 1, 2024

• Merge branch 'pubmed' of https://github.com/UIUC-Chatbot/ai-ta-backend into pubmed

  Show description for 8f608c5
  star-nox

  committed
  8f608c5

Commits on May 6, 2024

• minor changes in main loop

  star-nox

  committed
  4cfc6b8

Commits on May 13, 2024

• minor changes

  star-nox

  committed
  8b533ba

Commits on May 14, 2024

• parallelized processing

  star-nox

  committed
  6dfe50b

• added try-except in getArticleIds()

  star-nox

  committed
  e98f3dc

• deleted csv file

  star-nox

  committed
  68057df

• merged changes

  star-nox

  committed
  a2a5f22

• test comment

  star-nox

  committed
  4338931

• print test comment

  star-nox

  committed
  048f41d

• Commented out prints for speed

  KastanDay

  committed
  472814e

• Commented out prints for speed

  KastanDay

  committed
  880ae97

Commits on May 15, 2024

• Merge branch 'pubmed' of https://github.com/UIUC-Chatbot/ai-ta-backend into pubmed

  star-nox

  committed
  d635fe8

• parallelized main for loop and added xml filename column

  star-nox

  committed
  304ec5d