Nomic Code Clean-up, improve ingest stability (#238)

* updated create_doc_map()

* modified create_doc_map() to track last uploaded ids

* updated logging function to include missed records

* minor corrections in log function

* removed convo log code from beam folder

* started updates for convo logging

* modified function to create convo map from scratch

* minor changes to convo functions

* corrected function calling in ingest

* removed executor bec of errors

* fixed serialization error in main

ai_ta_backend/beam/ingest.py

@@ -1029,8 +1029,8 @@ def split_and_upload(self, texts: List[str], metadatas: List[Dict[str, Any]]):
 
       # add to Nomic document map
       if len(response.data) > 0:
-        inserted_data = response.data[0]
-        log_to_document_map(inserted_data)
+        course_name = contexts[0].metadata.get('course_name')
+        log_to_document_map(course_name)
 
       self.posthog.capture('distinct_id_of_the_user',
                            event='split_and_upload_succeeded',