Code

Web Scrape Updates#163

Code

Draft

jkmin3 wants to merge 26 commits intomainUIUC-Chatbot/ai-ta-backend:mainfrom web_scrape_enhancerUIUC-Chatbot/ai-ta-backend:web_scrape_enhancerCopy head branch name to clipboard

+306-202

Commits

Commits on Nov 6, 2023

• Updated Nomic in requirements.txt

  star-nox

  authored
  9ba6e9d

  Copy full SHA for 9ba6e9d

Commits on Nov 7, 2023

• fix openai version to pre 1.0

  KastanDay

  authored
  ca806eb

  Copy full SHA for ca806eb

• upgrade python from 3.8 to 3.10

  KastanDay

  committed
  17a7779

  Copy full SHA for 17a7779

• trying to fix tesseract // pdfminer requirements for image ingest

  KastanDay

  committed
  8562de7

  Copy full SHA for 8562de7

• adding strict versions to all requirements

  KastanDay

  committed
  be34f01

  Copy full SHA for be34f01

• Bump pymupdf from 1.22.5 to 1.23.6 (#136)

  Show description for d4b4e8f
  dependabot[bot]

  authored
  d4b4e8f

  Copy full SHA for d4b4e8f

• compatible wheel version

  KastanDay

  committed
  170ed79

  Copy full SHA for 170ed79

• upgrade pip during image startup

  KastanDay

  committed
  6b94aac

  Copy full SHA for 6b94aac

• properly upgrade pip

  KastanDay

  committed
  c084960

  Copy full SHA for c084960

• Fully lock ALL requirements. Hopefully speed up build times, too

  KastanDay

  committed
  f4b8bd9

  Copy full SHA for f4b8bd9

• Limit unstructured dependencies, image balloned from 700MB to 6GB. Hopefully resolved

  KastanDay

  committed
  4e80002

  Copy full SHA for 4e80002

• Lock version of pip

  KastanDay

  committed
  abf1fc2

  Copy full SHA for abf1fc2

• Lock (correct) version of pip

  KastanDay

  committed
  8a8eac2

  Copy full SHA for 8a8eac2

• add libgl1 for cv2 in Docker (for unstructured)

  KastanDay

  committed
  cf78800

  Copy full SHA for cf78800

• adding proper error logging to image ingest

  KastanDay

  committed
  62883e8

  Copy full SHA for 62883e8

• Installing unstructured requirements individually to hopefully redoce bundle size by 5GB

  KastanDay

  committed
  fcfa485

  Copy full SHA for fcfa485

• Reduce use of unstructured, hopefully the install is much smaller now

  KastanDay

  committed
  97bbbd9

  Copy full SHA for 97bbbd9

• Guarantee Unique S3 Upload paths (#137)

  Show description for 2103336
  KastanDay

  authored
  2103336

  Copy full SHA for 2103336

• Bump typing-extensions from 4.7.1 to 4.8.0 (#90)

  Show description for e547a94
  

  dependabot[bot]

  and

  KastanDay

  authored
  e547a94

  Copy full SHA for e547a94

• Bump flask from 2.3.3 to 3.0.0 (#101)

  Show description for 27a6680
  

  dependabot[bot]

  and

  KastanDay

  authored
  27a6680

  Copy full SHA for 27a6680

Commits on Nov 8, 2023

• Guard against kwargs failures during webscrape

  KastanDay

  committed
  a5b418c

  Copy full SHA for a5b418c

• HOTFIX: kwargs in html and pdf ingest for /webscrape

  KastanDay

  committed
  0d371ba

  Copy full SHA for 0d371ba

Commits on Nov 20, 2023

• Export conversation history on /analysis page (#141)

  Show description for ba1cbb1
  star-nox

  authored
  ba1cbb1

  Copy full SHA for ba1cbb1

Commits on Dec 5, 2023

• added option for extending one URL our when on baseurl or to opt out of it

  jkmin3

  committed
  cfca31c

  Copy full SHA for cfca31c

• removed depth search for now

  jkmin3

  committed
  36f7f90

  Copy full SHA for 36f7f90

• made filetypes a variable

  jkmin3

  committed
  95247b9

  Copy full SHA for 95247b9