Advance mapping with separation

[enoch3712]

From Doncat99 at #113

I see the bright future of this project. The top-level design may evolve from SOTA model extraction to an intelligent task flow pipeline. By indicating a (data model -> to do) relationship, such as an image extracted from a contract model, a sub-task will call up the image model to have the image summarized(predefined action 1) or re-generate(predefined action 2), etc.

This required a new description framework, followed by a capability declaration schema.

The balance between user-customized and model self-understanding is a big-thinking job to the above new framework and schema design.

****** OPENAI's UNDERSTANDING *******

From SOTA Model Extraction to an Intelligent Task Flow Pipeline
You aim to evolve from a state-of-the-art (SOTA) model primarily focused on extraction to an intelligent task flow pipeline. This pipeline orchestrates tasks intelligently based on predefined relationships and actions.
Key Idea:

You introduce a relationship between the data model (e.g., an extracted entity) and a "to-do" task.
For example, if an image is extracted under a contract model, the system can decide what to do with it:
Predefined Action 1: Summarize the image content (e.g., OCR + text summary).
Predefined Action 2: Re-generate the image in a new format (e.g., graph re-creation or enhancement).
The "intelligent task flow" determines the next step dynamically based on the extracted data type and predefined actions.

New Description Framework
This framework will describe the relationship between:
Data Model: Represents the data entities or assets extracted (e.g., images, graphs, sections of text).
Actions/Tasks: Predefined operations that can be performed on the data model.
For example, the description could look like this:

json
Copy code
{
"data_model": "ImageExtracted",
"to_do": [
{"action": "summarize", "description": "Summarize the image content using an image-to-text model"},
{"action": "regenerate", "description": "Recreate the image with updated formatting"}
]
}
This description framework acts as the instruction layer for the intelligent pipeline, enabling the system to decide which model or process to call based on the data_model and its associated actions.

Capability Declaration Schema
You need a schema where:
Each model or tool can declare its capabilities (what actions it supports and the inputs/outputs it expects).
The system dynamically matches the data model (and its required actions) with the available model/tool capabilities.
For example:

json
Copy code
{
"model": "ImageSummarizationModel",
"capabilities": [
{
"action": "summarize",
"input": "image",
"output": "text_summary"
},
{
"action": "regenerate",
"input": "image",
"output": "new_image"
}
]
}
With this schema, the intelligent pipeline can:

Match the required action (e.g., "summarize") to the model that declares support for it.
Pass the necessary inputs and retrieve the outputs.
4. Balance Between User-Customized and Model Self-Understanding
This is the core design challenge:

User-Customized: Allow the user to define tasks, workflows, and actions explicitly, providing maximum flexibility.
Model Self-Understanding: Enable the pipeline or the models themselves to infer the appropriate tasks/actions based on the input context, reducing the need for user-defined workflows.
The balance requires:

A clear framework to define tasks and relationships (for customization).
An adaptive task scheduler or reasoning engine (e.g., powered by an LLM) that dynamically understands and chooses actions when the workflow isn't explicitly defined.
For instance:

If the system sees an image extracted, it could infer:
Default action = "summarize" if no task is defined.
Custom action = "regenerate" if explicitly requested by the user.
Summary
Task Flow Pipeline: Intelligent pipeline connecting data models to actions.
Description Framework: Defines the relationship between extracted entities (data model) and associated tasks.
Capability Declaration Schema: Models/tools declare what tasks they can perform and their input/output.
Balance Challenge: Combine user-defined workflows with model-based self-understanding for flexible and dynamic task execution.
This approach lays the foundation for an intelligent, orchestrated AI task pipeline that can evolve and adapt to new tasks while maintaining a structured framework.

[enoch3712]

transfered to #133