Issue

What is the issue?

If we need any data changes it can only be done by the developers, is complicated and it takes a very long amount of time.

Expected Behaviour

There should be either a UX page or a command/script that allows for full or partial data import and calculation.
there should be sensible progress reporting and the ability to easily target a database.
there should also be a list of which standards and other resources have been imported, had embeddings calculated and which paths had gap analysis calculated.

Things to do:

• Make importing progressive. First import the CRE structure, then import each standard.
• Report a list of standards that are imported from spreadsheets
• Parallelise importing (if we first import cre, then we can import every standard in parallel and we can import external projects that do not have dependencies on existing standard in parallel)
• Calculate Gap Analysis Progressively for every new standard.
• Prioritise Gap Analysis for largest standards first
• Update neo4j DB on every stage
• Embeddings calculation in parallel for each standard or in the background
• Allow reimporting of resources, without structural changes (use case: resource name or hyperlink changed, implementation: Update db table for specific resource, regenerate embeddings for specific resource sections/subsection where updates were needed)
• Allow reimporting of resource, with structural changes (use case: data quality improvement or resource version change, implementaton: Remove resource links, remove resource gap analysis, re-link, recalculate gap analysis)
• Add tests for Prompt Client with mocks

If we change importing to be per resource, then we can also report progress since we know how many resources we support so we can figure out how many processors are at which state (importing, linking, embedding calculation, gap analysis calculation)

Implementation Considerations:

• Since this is a big feature, create a design document
• Use the workers architecture as much as possible
• Make as many of the new features into API calls for a possible future frontent
• Write tests