Code

Add Beam for serverless /ingest#223

Code

Merged

rohan-uiuc merged 39 commits intomainUIUC-Chatbot/ai-ta-backend:mainfrom add_beam_serverless_ingestUIUC-Chatbot/ai-ta-backend:add_beam_serverless_ingestCopy head branch name to clipboardMar 12, 2024

+3,873-4,406

Commits

Commits on Feb 22, 2024

• Fully working initial work - in production for webscrape

  KastanDay

  committed
  9f41df9

• cleanup

  KastanDay

  committed
  6d5eba4

• Add retries to ingest

  KastanDay

  committed
  4a3cf55

• Change scaling strategy

  KastanDay

  committed
  ade5ab1

• Change scaling strategy

  KastanDay

  committed
  da26920

• trunk improvement -- add isort

  KastanDay

  committed
  aa70cc5

• disable timeouts, set max replicas to 10 (max we get for free)

  KastanDay

  committed
  316b1bd

Commits on Mar 5, 2024

• Reduce autoscaling to improve costs. Set workers from 1 to 2.

  KastanDay

  committed
  c6d5aed

• Remove all ingest code from vector_database.py

  KastanDay

  committed
  8cce02b

• Clean up and format all code

  KastanDay

  committed
  2f30285

• Revert "Clean up and format all code"

  Show description for cc27a28
  KastanDay

  committed
  cc27a28

• Merge branch 'main' into add_beam_serverless_ingest

  KastanDay

  authored
  bb6c41a

• Clean up requirements.txt, removed all related to ingest

  KastanDay

  committed
  80a1119

Commits on Mar 6, 2024

• Add detailed posthog logging for /ingest failures

  KastanDay

  committed
  9460933

• Add nomic logging and deleting back to /ingest

  KastanDay

  committed
  44e9c11

• Add nomic dependency

  KastanDay

  committed
  3281c70

Commits on Mar 7, 2024

• Major Refactor introducing Dependency Injection

  rohan-uiuc

  committed
  ea9c432

• Move utils_tokenization to utils

  rohan-uiuc

  committed
  f0542a8

• Merge branch 'add_beam_serverless_ingest' of github.com:UIUC-Chatbot/ai-ta-backend into dependency_injection

  rohan-uiuc

  committed
  f40d533

• Add Flask-Injector to dependencies

  rohan-uiuc

  committed
  f6a787e

• Added executors for async operations

  rohan-uiuc

  committed
  a14cc44

• Adding injection to ExportService __init__, and add SQLDatabase injection to document map functions in NomicService

  rohan-uiuc

  committed
  ad220a6

• fix posthog error logs

  KastanDay

  committed
  8451a6b

• Increase workers to 3, remove callback url for now

  KastanDay

  committed
  f6615b1

• Add nomic logging to Beam ingest

  KastanDay

  committed
  db076a7

• Fix sentry service instantiation issue in ExportService constructor

  rohan-uiuc

  committed
  484b2a2

Commits on Mar 8, 2024

• Clean up env vars and minor type errors

  KastanDay

  committed
  016e48b

• Clean up trunk check recommendation

  KastanDay

  committed
  3017603

• Reduce workers from 6 to 3, should be more than enough with reduced responsibilities (no /ingest)

  KastanDay

  committed
  81fc4ef

• Update OpenAI API type to be fetched from environment variable

  rohan-uiuc

  committed
  5951fe4

• Reduce threads from 1_000 to 100. more sensible

  KastanDay

  committed
  ae00694

• Update posthog /getTopContexts name in so we can track impovements from DI

  KastanDay

  committed
  75251d1

• Add new method to SQLDatabase, update environment variable usage, some debugging logs

  rohan-uiuc

  committed
  f5a36fc

• Increase workers from 3 to 4

  KastanDay

  committed
  3abc7eb

• Fix deletion bug in retrieval_service to check for materials before deleting

  rohan-uiuc

  committed
  8a17e3d

Commits on Mar 12, 2024

• Fix mimetype parsing for ValueError: not enough values to unpack (expected 2, got 1)

  KastanDay

  committed
  107eef5

• Remove ffmpeg and tesseract-ocr, no ingest here anymore

  KastanDay

  committed
  d6976fe

• Merge branch 'add_beam_serverless_ingest' of github.com:UIUC-Chatbot/ai-ta-backend into dependency_injection

  rohan-uiuc

  committed
  cf5e4df

• Merge pull request #228 from UIUC-Chatbot/dependency_injection

  Show description for bea873d
  rohan-uiuc

  authored
  bea873d