Refactor DSPy adapters to make it more extensible

[chenmoneygithub]

We are reworking DSPy adapters for extensibility. For most users this change shouldn't cause backward compatibility issues, but if your workflow explicitly calls child methods of DSPy adapters, you need to make adjustments.

The goal here is with DSPy 3.0, we want Adapter to be a customizable interface rather than some tribal knowledge. We acknowledge that it will be common for users willing to write their own adapter to adjust to their LLMs and workflows. However, the current adapter doesn't have a decent abstraction, and to write a custom adapter users need to understand the source code, and go through a tedious debugging process without guidelines.

In this PR, we are trying to standardize the dspy Adapters, and open a few hooks for people to override during customization. We are aware that there is no single standard that fits all use cases, but trying to hit a stage where we don't over-simplify or over-engineer the base DSPy Adapter/

In a nutshell, we are making the following breakdown of Adapters:

• Adapter
  • format(): formats the type-based inputs into LM multiturn messages
    • System messages: The high level description of the task, and LM I/O format.
      • Fields description: format_field_description()
      • LM input/output structure description: format_field_structure()
      • task description: format_task_description
    • Few-shot examples (demo): multiturn few-shot examples
      • user message (inputs of demo): format_user_message_content()
      • assistant message (outputs of demo): format_assistant_message_content()
    • Conversation history: multiturn conversation history
      • user message (inputs of history message): format_user_message_content()
      • assistant message (outputs of history message): format_assistant_message_content()
    • Current input: the actual question/input
      • user message: format_user_message_content()
  • parse(): parse the LM response to type-based outputs. No sub-hook for parse() because it varies for different adapters.

Note that format_user_message_content() and format_assistant_message_content() are used in multiple places. Users can override any level of hooks for customization.

We will publish a guide on how to customize Adapter with concrete use cases after landing this PR.

[okhat]

Why this long docstring that will go stale as soon as someone changes the adapter slightly?

[chenmoneygithub]

simplified for easy maintenance!

[okhat]

This is very specific to chat adapter. Why does it belong in base.py?

Overall, base.py should be very short and lightweight to allow anyone to build any kind of adapter. Right now, this PR is introducing a lot of specific and complicated structure to base.py

[okhat]

How do we handle cases where the demos have multi-turn history?

[chenmoneygithub]

it will be a single turn entity with history as one field, for example:

User message:

This is an example of the task, though some input or output fields are not supplied.

[[ ## question ## ]]
What is the capital of France?

[[ ## history ## ]]
[{"question": "What is the capital of Germany?", "answer": "Berlin"}]

Respond with the corresponding output fields, starting with the field `[[ ## reasoning ## ]]`, then `[[ ## answer ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`.


Assistant message:

[[ ## reasoning ## ]]
Not supplied for this particular example. 

[[ ## answer ## ]]
Paris