Code

132 smart memory management#142

Code

Closed

minump wants to merge 204 commits intogithub_agent_webhooksUIUC-Chatbot/ai-ta-backend:github_agent_webhooksfrom 132-smart-memory-managementUIUC-Chatbot/ai-ta-backend:132-smart-memory-managementCopy head branch name to clipboard

+3,318-1,569

Commits

Commits on Aug 14, 2023

• disable lintrule

  KastanDay

  authored
  0beb883

• disable lintrule

  KastanDay

  authored
  0cb1c6a

• Fix yapf formatting to properly save and commit changes

  KastanDay

  authored
  085a5ce

• show course metadata schema

  KastanDay

  authored
  f7fceab

• Supabase SQL Redesign -- to be faster and better organized (#27)

  Show description for 79770ac
  

  KastanDay

  and

  jkmin3

  authored
  79770ac

• Merge branch 'main' of https://github.com/UIUC-Chatbot/ai-ta-backend

  jkmin3

  committed
  5600260

• fixed the baseurl so that it is originally off

  jkmin3

  committed
  a3e2374

Commits on Aug 15, 2023

• adding prints for debugging

  KastanDay

  committed
  6bd9c8b

• Merge branch 'main' of https://github.com/UIUC-Chatbot/ai-ta-backend

  jkmin3

  committed
  ae14314

• adding prints for debugging

  KastanDay

  committed
  f706356

• Merge branch 'main' of https://github.com/UIUC-Chatbot/ai-ta-backend

  jkmin3

  committed
  46489e8

• changed all instances of pagenumber_or_timestamp to pagenumber and timestamp

  jkmin3

  committed
  31af46a

• added text to metadata for split and upload

  jkmin3

  committed
  d5ee953

• added the new filter system and changed page_content from text

  jkmin3

  committed
  da51458

• removed line that showed usecase of function

  jkmin3

  committed
  d51df0b

• Using the user query embedding in split

  jkmin3

  committed
  f7c36bf

• indexed the user_query

  jkmin3

  committed
  2f311ae

• fixed gettopcontexts

  jkmin3

  committed
  72be1e8

• webscrape no longer scrapes gifs and vector database prints have been cleaned up

  jkmin3

  committed
  1e1b56a

• fixed the getstuffed prompt function

  jkmin3

  committed
  7e6d7c9

• gettopcontexts aand getstuffed prompt functions are now fully working with Document objects

  jkmin3

  committed
  62b2f13

• fixed issues with the output of number of tokens

  jkmin3

  committed
  0f17d4e

• Fix Supabase SQL migration, esp getTopContexts() (#50)

  Show description for 255d040
  KastanDay

  authored
  255d040

• Merge branch 'main' of https://github.com/UIUC-Chatbot/ai-ta-backend

  jkmin3

  committed
  62c22c1

• added only safe filetypes in webscrape

  jkmin3

  committed
  101485c

• reverting changes in webscrape

  jkmin3

  committed
  1378333

• improve delete_entire_course to try each database separately

  KastanDay

  committed
  9d24970

Commits on Aug 16, 2023

• update getAll to include URLs

  KastanDay

  committed
  6c35405

• enhance parameter names in getAll

  KastanDay

  committed
  dbbca0a

Commits on Aug 17, 2023

• Merge branch 'main' of https://github.com/UIUC-Chatbot/ai-ta-backend

  jkmin3

  committed
  996caeb

Commits on Aug 19, 2023

• webscraping supports all filetypes supported by bulk_ingest

  jkmin3

  committed
  603e0f0

Commits on Aug 21, 2023

• Testing .txt and .py ingest functions (#55)

  Show description for 6ef06fd
  

  star-nox

  and

  KastanDay

  authored
  6ef06fd

Commits on Aug 22, 2023

• Merge branch 'main' of https://github.com/UIUC-Chatbot/ai-ta-backend

  jkmin3

  committed
  a4256da

• Removed dates for webscraping titles

  jkmin3

  committed
  4a82e3c

• fixed bug with checking url and filetype

  jkmin3

  committed
  9f3bc3d

• fixed bug with no variable of filetype when their is the soup

  jkmin3

  committed
  fb80a04

Commits on Aug 23, 2023

• changed response.text to content in web scrape

  jkmin3

  committed
  06b1577

• reverted changes to text

  jkmin3

  committed
  e9decaf

Commits on Aug 24, 2023

• Bump flask from 2.3.2 to 2.3.3 (#61)

  Show description for bdb9607
  dependabot[bot]

  authored
  bdb9607

• Update protobuf requirement from <=3.20 to <=4.24.1 (#62)

  Show description for e70be75
  dependabot[bot]

  authored
  e70be75

• Bump langchain from 0.0.256 to 0.0.270 (#60)

  Show description for 8a21dc0
  dependabot[bot]

  authored
  8a21dc0

• add libxcrypt1 to aptPkgs

  KastanDay

  committed
  0f846d5

• fix spelling of libcrypt1

  KastanDay

  committed
  35537ed

• set pymupdf version to fix build error

  KastanDay

  committed
  120c858

• adding git as dependency

  KastanDay

  committed
  f904893

• fixing git dependency

  KastanDay

  committed
  92583e6

Commits on Aug 25, 2023

• [SIMULTANIOUS MERGE with Frontend] User query visualizations with Nomic (#53)

  Show description for 777891e
  

  3 peopleauthored
  777891e

• make /delete single files INSTANT via background tasks

  KastanDay

  committed
  8214f82

• Merge branch 'main' of https://github.com/UIUC-Chatbot/ai-ta-backend

  jkmin3

  committed
  4dca008

• Stopped errors in ingest when you start with a pdf url

  jkmin3

  committed
  a898978

Commits on Aug 28, 2023

• Fixed delete course, so that it deletes some if not all

  jkmin3

  committed
  2cc2b5b

Commits on Aug 30, 2023

• Added much better print statements and also removed duplicate urls from supabase

  jkmin3

  committed
  70223fb

• Created the stay_on_baseurl to be a boolean paramater. Also I was able to fix some bugs with the supabase duplicate remover for pdfs and catching nonetype errors

  jkmin3

  committed
  eda0571

• forgot to uncomment the split and upload section of webscrabe

  jkmin3

  committed
  305ee13

• push to deploy on railway

  jkmin3

  committed
  6dcf7ed

Commits on Aug 31, 2023

• fixed webscrape nonetype bug

  jkmin3

  committed
  cdceb0e

• enhance split_and_upload to handle code files that don't have periods

  KastanDay

  committed
  b41478e

Commits on Sep 5, 2023

• reducing workers, increasing threads. will greatly reduce memory footprint at idle and under load

  KastanDay

  committed
  61a296b

• Reduce memory consumption to save cash on Railway (#82)

  Show description for 9161d8c
  KastanDay

  authored
  9161d8c

• adding defaults for web scraping

  KastanDay

  committed
  f8c3bc2

• increasing timeout from 30 sec to 30 min, web scrape takes a while sometimes

  KastanDay

  committed
  805936b

Commits on Sep 6, 2023

• Merge branch 'main' of https://github.com/UIUC-Chatbot/ai-ta-backend

  jkmin3

  committed
  e4aea5b

• fixed webscraper

  jkmin3

  committed
  0af41e6

• HOTFIX for cannot schedule new futures after interpreter shutdown, which prevented uploads to s3 on web-scrape

  KastanDay

  committed
  6ce20e7

• Merge branch 'main' of https://github.com/UIUC-Chatbot/ai-ta-backend

  jkmin3

  committed
  0347c7a

• quick fix for supabase using try except

  jkmin3

  committed
  32abf39

• Fix GitHub ingest: separate files created properly (#76)

  Show description for 6a1a38f
  

  star-nox

  and

  KastanDay

  authored
  6a1a38f

Commits on Sep 9, 2023

• Merge branch 'main' of https://github.com/UIUC-Chatbot/ai-ta-backend

  jkmin3

  committed
  16381d9

Commits on Sep 11, 2023

• Create .env.template

  KastanDay

  authored
  c6c21e6

• 📜 Add .env.template to install instructions

  KastanDay

  authored
  09010b5

Commits on Sep 13, 2023

• Merge branch 'main' of https://github.com/UIUC-Chatbot/ai-ta-backend

  jkmin3

  committed
  7f74f86

Commits on Sep 15, 2023

• Improve Nomic Visualizations with full conversations & date filtering (#72)

  Show description for 520444a
  

  3 peopleauthored
  520444a

• Refactor ingest file handling for maintainability; it's beautiful! (#73)

  Show description for 241f819
  

  jkmin3

  and

  KastanDay

  authored
  241f819

• HOTFIX: critical bug with duplicate doucments in Supabase for EVERY page in doc, fix Github ingest to standardize

  KastanDay

  committed
  4f6a863

• Enhancement to Github ingest, add full URLs, enable file deletion

  KastanDay

  committed
  26ee9f4

Commits on Sep 16, 2023

• HOTFIX: required parameter handling on /delete endpoint

  KastanDay

  committed
  0a9d074

• HOTFIX: url paramater name in /delete

  KastanDay

  committed
  8867a51

• remove --max-requests from flask run, hopefully eliminates missing requests without big memory hit

  KastanDay

  committed
  66b757e

• Merge branch 'main' of https://github.com/UIUC-Chatbot/ai-ta-backend

  jkmin3

  committed
  2bd63e3

Commits on Sep 21, 2023

• HOTFIX: Tiny typo in /webscrape input params

  KastanDay

  committed
  d7741d1

• remove libcrypt1 dependency

  KastanDay

  committed
  0f432ef

Commits on Sep 22, 2023

• Fix build issue with apt-get by holding back NIXPACKS version to 1.15.0

  KastanDay

  committed
  9eeef83

Commits on Sep 25, 2023

• Merge branch 'main' of https://github.com/UIUC-Chatbot/ai-ta-backend

  jkmin3

  committed
  a7fb1a8

Commits on Sep 27, 2023

• ignore all notebooks

  KastanDay

  committed
  b969ee4

• add prints to text ingest

  KastanDay

  committed
  6585981

• move nomic.login() out of the top level so it's only called when the functions are needed... bad idea, but nomic login is broken right now

  KastanDay

  committed
  b21aad7

• adding more prints to bulk_ingest... I think I see why text files in particular are failing

  KastanDay

  committed
  1124b7d

• add prints to /ingest route

  KastanDay

  committed
  7a7d92b

• should fix text ingest, we were using the wrong file extençsion AND mime type

  KastanDay

  committed
  1793b94

• add kwargs to ALL ingest methods to avoid errors

  KastanDay

  committed
  5792647

Commits on Sep 28, 2023

• fix killer bug that prevented mimetypes from working

  KastanDay

  committed
  c879b71

• further bugfix to mimetype ingest

  KastanDay

  committed
  08f0ecf

• I think this solves all mimetype troubles

  KastanDay

  committed
  4200c5e

• change 'is' to '==' for Callable == 'None'

  KastanDay

  committed
  7eb4ecd

• add 'wheel' requirement for faster builds

  KastanDay

  committed
  165dc51

• further refinement of bulk_ingest

  KastanDay

  committed
  6c3218d

• fixed video and audio ingest

  KastanDay

  committed
  7422429

Commits on Oct 2, 2023

• Webscrape Refactor for better ingestion (#92)

  jkmin3

  authored
  86c82c9

Commits on Oct 3, 2023

• Merge branch 'main' of https://github.com/UIUC-Chatbot/ai-ta-backend

  jkmin3

  committed
  9865897

• Bugfix Nomic convo logging "Project Locked" (#98)

  star-nox

  authored
  e0892b3

Commits on Oct 4, 2023

• Adding support for excel files

  KastanDay

  committed
  603c3e3

• add CSV loader support

  KastanDay

  committed
  fc817dd

• add dependency for excel ingest

  KastanDay

  committed
  76dc5b9

• add image OCR ingest, add ingest fallback to utf-8 encoding when all else fails

  KastanDay

  committed
  d11898d

Commits on Oct 9, 2023

• Merge branch 'main' of https://github.com/UIUC-Chatbot/ai-ta-backend

  jkmin3

  committed
  679033e

• only external sites have a depth of one

  jkmin3

  committed
  4d0566a

Commits on Oct 10, 2023

• update split_and_upload() to work on paragraphs first, then sentences, then chars

  KastanDay

  committed
  6fbd1bb

• Update PDF parsing to use utf-8 chars instead of ascii

  KastanDay

  authored
  97a300d

Commits on Oct 11, 2023

• removed content from supabase call, used regex to get rid of # at the end of URLs

  jkmin3

  committed
  563e0de

Commits on Oct 14, 2023

• disregarded the nav and footer completely, also added new base url requiremnets function to detect related urls better

  jkmin3

  committed
  b2e1911

• Merge pull request #107 from UIUC-Chatbot/external-site-handling

  Show description for 817eb2e
  jkmin3

  authored
  817eb2e

Commits on Oct 23, 2023

• Adding Canvas ingest (#33)

  Show description for 89f61aa
  

  4 peopleauthored
  89f61aa

Commits on Oct 25, 2023

• add chunk index

  KastanDay

  committed
  b0c6d3c

• add chunk index - minor hotfix

  KastanDay

  committed
  daa0fa9

Commits on Oct 26, 2023

• added chunk index to supabase metadata

  star-nox

  committed
  936ee47

• shifted chunk_index after text splitter

  star-nox

  committed
  710cce0

Commits on Oct 30, 2023

• Make all necessary adjustments to make our 'metadata' dict, a top-level opbject for search Index compatability (#125)

  KastanDay

  authored
  9f7e03f

• testing missing page content

  KastanDay

  committed
  1c92913

• HOTFIX: silly error in metadata parsing, should be fully fixed

  KastanDay

  committed
  3eb0607

• HOTFIX: update the delete endpoints to new Qdrant schema

  KastanDay

  committed
  285581f

• Improved Nomic error handling when nomic map does not yet exist (#118)

  star-nox

  authored
  3961141

• QDRANT tuning: disable rescore to increase speed

  KastanDay

  committed
  b495e9d

• Fix Canvas ingest; modified download_files and download_assignments (#124)

  star-nox

  authored
  72c6094

Commits on Nov 6, 2023

• playwright in requirements

  minump

  committed
  a1cc5da

• updated gitignore

  minump

  committed
  5dc61d7

• Updated Nomic in requirements.txt

  star-nox

  authored
  9ba6e9d

• Merge branch 'github_agent_webhooks' into 132-smart-memory-management

  Show description for aafcccd
  minump

  committed
  aafcccd

Commits on Nov 7, 2023

• fix openai version to pre 1.0

  KastanDay

  authored
  ca806eb

• upgrade python from 3.8 to 3.10

  KastanDay

  committed
  17a7779

• trying to fix tesseract // pdfminer requirements for image ingest

  KastanDay

  committed
  8562de7

• adding strict versions to all requirements

  KastanDay

  committed
  be34f01

• Bump pymupdf from 1.22.5 to 1.23.6 (#136)

  Show description for d4b4e8f
  dependabot[bot]

  authored
  d4b4e8f

• compatible wheel version

  KastanDay

  committed
  170ed79

• upgrade pip during image startup

  KastanDay

  committed
  6b94aac

• properly upgrade pip

  KastanDay

  committed
  c084960

• Fully lock ALL requirements. Hopefully speed up build times, too

  KastanDay

  committed
  f4b8bd9

• Limit unstructured dependencies, image balloned from 700MB to 6GB. Hopefully resolved

  KastanDay

  committed
  4e80002

• Lock version of pip

  KastanDay

  committed
  abf1fc2

• Lock (correct) version of pip

  KastanDay

  committed
  8a8eac2

• add libgl1 for cv2 in Docker (for unstructured)

  KastanDay

  committed
  cf78800

• adding proper error logging to image ingest

  KastanDay

  committed
  62883e8

• Installing unstructured requirements individually to hopefully redoce bundle size by 5GB

  KastanDay

  committed
  fcfa485

• Reduce use of unstructured, hopefully the install is much smaller now

  KastanDay

  committed
  97bbbd9

• Guarantee Unique S3 Upload paths (#137)

  Show description for 2103336
  KastanDay

  authored
  2103336

• Bump typing-extensions from 4.7.1 to 4.8.0 (#90)

  Show description for e547a94
  

  dependabot[bot]

  and

  KastanDay

  authored
  e547a94

• Bump flask from 2.3.3 to 3.0.0 (#101)

  Show description for 27a6680
  

  dependabot[bot]

  and

  KastanDay

  authored
  27a6680

Commits on Nov 8, 2023

• Guard against kwargs failures during webscrape

  KastanDay

  committed
  a5b418c

• HOTFIX: kwargs in html and pdf ingest for /webscrape

  KastanDay

  committed
  0d371ba

Commits on Nov 13, 2023

• memory management

  minump

  committed
  fd16261

• memory agent

  minump

  committed
  dc0050f

• removing unwanted import

  minump

  committed
  ac45fab

Commits on Nov 14, 2023

• Merge branch 'github_agent_webhooks' into 132-smart-memory-management

  minump

  committed
  ed67886

• resolve merge conflict

  minump

  committed
  9401d64

Commits on Nov 17, 2023

• Update ml4bio_agent.py

  minump

  committed
  3ba257c

• Merge remote-tracking branch 'origin/github_agent_webhooks' into 132-smart-memory-management

  Show description for 2ef52f2
  minump

  committed
  2ef52f2

• Merge remote-tracking branch 'origin' into 132-smart-memory-management

  minump

  committed
  2d4dc73

Commits on Nov 20, 2023

• updated gitignore with venv

  minump

  committed
  094256b

• memoryCallbacks

  minump

  committed
  4ac7d52

• ml4bio agent with memory callbacks

  minump

  committed
  7c2ee11

• method to get supabase client and langsmitID

  minump

  committed
  4ec4d73

• docker with nocache

  minump

  committed
  85b6068

• updated gitignre

  minump

  committed
  21091c7

• Implemented RNA-Seq workflow for PBMC Samples

  minump

  committed
  51cd814

Commits on Nov 21, 2023

• supabase client and langsmith id in utils

  minump

  committed
  ce8f4ab

• use above filtered result

  minump

  committed
  80516b4

• langsmith id from utils

  minump

  committed
  3d8a26d

• langsmith and supabase frmo utils

  minump

  committed
  c9c008e

• langsmith and supabase from utils

  minump

  committed
  8f312cb

• spacing changes

  minump

  committed
  52f885a

• added custom callback

  minump

  committed
  6f722c3

• custom callback handler

  minump

  committed
  6350c71

• image name for workflow agent

  minump

  committed
  5faf839

• rename to custom callbacks

  minump

  committed
  7b2d36f

• rename to customcallback

  minump

  committed
  4970f54

• Committing changes for RNA-Seq Workflow

  minump

  committed
  ca7922d

Commits on Nov 22, 2023

• added customcallback for tools

  minump

  committed
  6bf6bbe

• callback for tools

  minump

  committed
  e725235

• callback for vectorStore tool

  minump

  committed
  a6d2ebd

• comment out onChainStart callback

  minump

  committed
  fd53fc7

Commits on Nov 27, 2023

• all db calls in utils

  minump

  committed
  88e327f

• custom callbacks with db updates

  minump

  committed
  84f8071

• new memory context fetched from db

  minump

  committed
  086a218

Commits on Nov 28, 2023

• full path imports

  minump

  committed
  58af637

• insert in array format for db

  minump

  committed
  b2c13a1

• Merge branch 'github_agent_webhooks' into 132-smart-memory-management

  minump

  committed
  5211281

• remove unwanted method

  minump

  committed
  4dc2c5c

• revert to original main call to webhooks handle event

  minump

  committed
  3bbaae1

• changes in order of imports

  minump

  committed
  7c3a4f1

• fix typo

  minump

  committed
  dde2582

• minor change to import and extra space

  minump

  committed
  1315b5e

• Not used comment

  minump

  committed
  e9b8ce1

• delete unwanted scripts

  minump

  committed
  002096c

• delete payload

  minump

  committed
  43a3dcd

• imports with full paths and fixed typo

  minump

  committed
  f98f141

• Import fixes to get both flask and docker running

  rohan-uiuc

  committed
  d726a8a

Commits on Dec 1, 2023

• update imagename with ourtag

  minump

  committed
  ccf3535

• memory agent created

  minump

  committed
  b015858

Commits on Dec 7, 2023

• add marvin package

  minump

  committed
  07c600f

• agent log parsing using marvin

  minump

  committed
  e24d717

Commits on Jan 10, 2024

• Merge branch 'github_agent_webhooks' into 132-smart-memory-management

  rohan-uiuc

  committed
  b7e7654

Commits on Jan 11, 2024

• Changes to merge memory management with latest changes in github_agent_webhooks branch

  rohan-uiuc

  committed
  8e2f505

Commits on Jan 22, 2024

• removed marvin and fixed agent finish callback

  minump

  committed
  3acf82b

• added agent finish to memory context

  minump

  committed
  ad35ba7

• check for None response

  minump

  committed
  d3e7ebe