docs: add Java SDK evaluate() documentation#4745
Conversation
Document evaluate(), evaluateAsync(), evaluateExisting(), and evaluateComparative() for langsmith-java 0.1.0-beta.11 with plain-language explanations and minimal Kotlin examples. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Thanks for opening a docs PR, sineha-mani! When it's ready for review, please add the relevant reviewers:
|
Add Java/Kotlin tabs and an options section to evaluate-llm-application.mdx instead of maintaining a separate Java SDK page. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace Kotlin tabs and examples with Java using Evaluation, Tracing.traceFunction, and DynamicRunEvaluatorKt APIs. Co-authored-by: Cursor <cursoragent@cursor.com>
Add dependency install instructions and notes for re-scoring overloads and EvaluationResult construction from Java. Co-authored-by: Cursor <cursoragent@cursor.com>
Point readers to per-language API reference and drop the install block that had no Python/TS counterpart on this page. Co-authored-by: Cursor <cursoragent@cursor.com>
The intro Javadoc link already anchors the required SDK version for Java evaluate examples. Co-authored-by: Cursor <cursoragent@cursor.com>
sineha-mani
changed the title
| After running an experiment, you may want to **add new evaluation metrics without re-running your application**. This is useful when you've added new evaluators or want to apply different scoring criteria to existing results. Instead of re-executing your target function on all examples, you can evaluate the existing experiment traces directly. | ||
|
|
||
| To add evaluators to an existing experiment, pass the experiment name or ID to `evaluate()` / `aevaluate()` instead of a target function. The evaluators will run on the cached traces from the original experiment, accessing the inputs, outputs, and any intermediate steps that were logged. | ||
| To add evaluators to an existing experiment, pass the experiment name or ID to `evaluate()` / `aevaluate()` (Python), or `Evaluation.evaluateExisting()` (Java) instead of a target function. The evaluators run on the cached traces from the original experiment, accessing the inputs, outputs, and any intermediate steps that were logged. |
There was a problem hiding this comment.
Guess we should add to JS too
| import java.util.function.Function; | ||
|
|
||
| // Optionally wrap with Tracing.traceFunction to record inputs and outputs in LangSmith. | ||
| Function<Map<String, Object>, Map<String, Object>> toxicityClassifier = |
There was a problem hiding this comment.
Hmm why can't we just automatically do this for functions passed into evaluate like we do in Python/JS?
There was a problem hiding this comment.
Oh good point - evaluate() already wraps the target with traceable() internally. The traceFunction in the docs is optional/custom-naming parity with Python's traceable section; we can simplify the Java example to a plain lambda. Will update.
| import java.util.Objects; | ||
|
|
||
| RunEvaluator correct = | ||
| DynamicRunEvaluatorKt.runEvaluatorFromOutputsReference( |
There was a problem hiding this comment.
This naming is completely meaningless to me - does it make sense if you are a Java dev?
There was a problem hiding this comment.
The Kt suffix is automatic Kotlin→Java interop: top-level functions in DynamicRunEvaluator.kt compile to DynamicRunEvaluatorKt.* on the JVM.
There was a problem hiding this comment.
DynamicRunEvaluator is the SDK adapter that turns the scoring lambda into a RunEvaluator the evaluate runner can call. It pulls outputs/reference from each run and normalizes the return value (e.g. 1.0) into LangSmith feedback.
There was a problem hiding this comment.
Did that answer what you meant or am I missing a spot of clarification?
Evaluation.evaluate() already traces the target when uploadResults is true. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Mintlify preview branch generated: Site preview: https://langchain-5e9cc07a-preview-docsja-1783484944-f86c4f0.mintlify.app Important Preview links may take a few minutes to start working while the deployment finishes. Changed documentation pages (preview deep links): |
2 participants
Summary
/langsmith/evaluate-llm-application-javathat explainsevaluate()in plain language for Java/Kotlin users onlangsmith-java0.1.0-beta.11"Here's how a Java developer runs an evaluation"
│ ✅ Install library
│ ✅ Step-by-step guide (same page as Python/TS)
│ ✅ Common options cheat sheet
│ ✅ Re-score existing experiment (short)
│ ✅ Version note + a few gotchas
Test plan
Made with Cursor