Nomic Code Clean-up

[star-nox]

Ingest faces issue because of failing nomic log attempts.

Reasons for failed logs - map is rebuilding and gets locked for access. All subsequent calls to nomic_log after the 1st attempt fail.

Solution -

1. skip nomic log altogether if map is locked
2. keep track of last uploaded id in both maps
3. modify the log function to retrieve this last uploaded id and log all records since
4. what if there are a large number of unlogged records? it will increase overall ingestion time. can we move ingest to background? or is it okay as it is in beam?

[lintrule-review]

You need to setup a payment method to use Lintrule

You can fix that by putting in a card here.

[KastanDay]

Looks great. A complicated refactor to store all the documents that have yet to be added into nomic, but a necessary approach since we're using their service in a weird (cool) way. Thanks!

[sentry-io]

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ InvalidRequestError: Unsupported OpenAI-Version header provided: 2023-05-15. (HINT: you can provide any of the followi... logToNomic View Issue
• ‼️ IndexError: list index out of range logToNomic View Issue

_{Did you find this useful? React with a 👍 or 👎}