Discover/issue 5846 opik vs elastic poc - #29
Conversation
180c2b3 to
5b14181
Compare
avillalba-elastic
left a comment
avillalba-elastic
left a comment
There was a problem hiding this comment.
Other than the questions I have left, this LGTM so I approve!
| expected=example.output, | ||
| metadata=example.metadata, | ||
| ) | ||
| correctness_result = await correctness_analysis.evaluate(params) |
There was a problem hiding this comment.
I think in run_experiment you already run the evaluator, right? And call the score ingestion API afterwards. I wonder if it is needed to call evaluate here again. Wdyt?
| run_data = executed["data"] | ||
| retrieved = _extract_retrieved_doc_ids(run_data.output, tool_id=SEARCH_TOOL_ID) | ||
| relevant = _to_string_list((run_data.metadata or {}).get("relevant_doc_ids")) | ||
| evidence = _orca_evidence(retrieved) |
There was a problem hiding this comment.
Ok I see what you've done here.
So what I see is that I can create a custom evaluator using the SimpleEvaluator class and the evaluate(EvaluationParams) -> EvaluationResult interface. Then you extract the retrieved doc IDs from the output steps, and build our EvaluationEvidence object in _orca_evidence. That is understood 👍🏻
My point is the following: this custom evaluator is not trace-based. If it was trace-based, then you will need to use your ElasticsearchTraceClient, you build the query and you fetch the info from the traces stored in Elasticsearch. But you will need to implement all the logic for traversing the span tree, pick each tool call, see if it gives us retrieved doc IDs... that's just an example but my point is that that has some complexity and more importantly it depends on the agent. In the Kibana Evaluators API for example, you have instrumentation profiles for different versions of the OTEL GenAI conventions, for Claude Code... but for custom evaluators, you will need to reimplement that completely (at least in the current state), either in this Python SDK or even from other tools leveraging this one, like orca.
Just to be clear - not a blocker or hard requirement on our side, just something that I'm flagging here because I think it is a good conversation to have. But thanks for the example - it made everything more clear!
There was a problem hiding this comment.
@avillalba-elastic acknowledged!
The tracing part will definitely need to be reviewed, and I pointed that gap in the documentation: https://docs.google.com/document/d/1HyIX07Yhiy2TkctZax2hzvTGQTJcAcaqI3eXNDAOOYQ/edit?tab=t.l762gocbxj3
So we expect in the future to have an issue or do some discovery to improve this part.
|
|
||
| async def fetch_output_tokens() -> float: | ||
| nonlocal has_result, last_result | ||
| response = await trace_client.query(_build_query(trace_id)) |
There was a problem hiding this comment.
So for example, to make my point more clear: here you have a trace-based evaluator, and you query ES to get the output tokens. As long as the harness is instrumented with OTEL GenAI this works, but for others that do not (like Claude Code) - what would you do? Implement here, in the Python SDK, another "instrumentation profile" mirroring what Kibana does?
I know we have this evaluator in particular in Kibana too, so it is accessible through the Evaluators API. But this case will be the same for custom evaluators.
I hope I made that more clear!
b179db9 to
eeb189e
Compare
milistu
left a comment
milistu
left a comment
There was a problem hiding this comment.
issue: There is a reference and dependency point to Orca, which is a private repository. The script run2.py imports some of Orca's resources, and we should avoid this in public-facing repositories.
Therefore, as discussed offline, the action points are:
- Considering that Ana is already working on Orca’s side to migrate to the Python SDK, @mafaldasv-dev is going to get in touch with Ana to understand if she still needs
run2.py. If this script is no longer needed, it will be deleted, as well as the editable dependency. - After the previous step is done, the rest can be merged after optional comments are addressed/resolved.
| ### Elasticsearch | ||
|
|
||
| ```bash | ||
| cd /Users/mafaldasavelho/Documents/work-repos/kibana-fork/kibana |
There was a problem hiding this comment.
issue: We should remove all hardcoded personal machine paths and use generic ones instead.
7dbf70e to
198fe2f
Compare
3 participants
Rebased on PR #28 (Add Agent Builder and Evaluators API client)
Closes #5846.
Summary
This PR implements the Agent Builder evaluation PoC used to compare the existing Opik workflow with the Python SDK backed by
kbn/evals.It demonstrates two ways of conducting the same WixQA evaluation use case:
Managed workflow
run.pyusesElasticEvalsClient.run_experiment()to manage task execution, SDK-side evaluators, custom evaluation, tracing, and score ingestion.Granular workflow
run2.pyperforms the workflow withoutrun_experiment(). It directly coordinates:The PoC can load the public
Wix/WixQAdataset and knowledge base from Hugging Face or use the internal GCS source. Both scripts allow users to select a small sample or run the complete source dataset.The PR also includes setup documentation, environment templates, Elasticsearch indexing helpers, retained Opik comparison utilities, and mocked tests for the data and evaluation workflows.
Testing
Risks
The PoC depends on local Elasticsearch, Kibana, EDOT, configured inference connectors, and Agent Builder. Dataset synchronization replaces previous dataset contents, and the Elasticsearch index is recreated on every execution.
This PR must be merged after #28 because it uses the Agent Builder and Evaluators API clients introduced there.