Agent primitive: AI agents as first-class Nextflow tasks

[pditommaso]

Summary

Adds agent as a task-shaped language construct whose body is an LLM call with an optional tool-calling loop. An agent takes one input record, produces one result, and executes as an ordinary TaskRun — so it inherits the machinery processes already have: parallelism, resume, lineage, tracing, container provisioning and error reporting.

record Sample {
    sample_id: String
    notes:     String
}

agent triage {
    model 'openai/gpt-5-mini'
    instruction 'You triage sequencing samples. Be concise.'

    input:
    sample: Sample
    output:
    verdict: String

    prompt:
    """
    Assess sample '${sample.sample_id}'. Notes: ${sample.notes}
    """
}

workflow {
    triage(samples).view { v -> "VERDICT=${v}" }
}

The rationale is recorded in adr/20260505-llm-agent-primitive.md. In short: agentic analysis is first and foremost a reproducibility problem, and the requirements it raises — versioned skill distribution, deterministic replay, audit trails, portability — are the ones Nextflow already solved for deterministic compute. Replay is caching, audit is lineage, skill distribution is module distribution. Rather than build a parallel stack outside the engine, this makes the agent a node in the workflow graph.

What's in it

Language (nf-lang) — AGENT lexer token and agentDef grammar rules, with agent and prompt still usable as identifiers; AgentNode AST, visitAgent on the script visitor, agent declarations on ScriptNode, and inclusion in symbol resolution, include visitors and the formatter; AgentToGroovyVisitor lowers a definition to a runtime agent(...) call; ImplicitStagers unifies agent and process Path I/O lowering; AgentDsl directive scope and type checking.

Runtime (modules/nextflow) — nextflow.agent holds the runtime model: typed I/O and record schemas, output plans, tool descriptors and dispatch, module-as-tool bridging, skills, filesystem tools and a sandbox guard. nextflow.agent.rpc handles driver-side registration and host resolution for out-of-process runners. AgentExecutor, AgentTaskHandler and AgentPollingMonitor run agents through the standard task path. AgentRunner is the SPI; AgentConfig backs the agent config scope.

Runners (plugins) — nf-agent is a langchain4j-backed runner with structured output via responseFormat, tool and skill adapters, and a portable schema mapped to langchain4j JsonSchema. nf-agent-pi is an out-of-process runner distributed as a container image, with a Go RPC proxy, a Node harness, TLS credentials and secret masking.

Lineage — an AgentRun record, with agent runs rendered in the lineage DAG.

Docs — docs/agent.mdx is the user guide; adr/specs/ holds the architectural reference for the design, the runner SPI and the RPC protocol, next to the decision record it elaborates.

Examples — examples/agents/ has 18 runnable examples covering structured output, tools, skills, modules-as-tools, filesystem tools, goal-directed loops, convergence loops, map/reduce and fan-in, plus validate.sh which runs them end to end (-r also checks each replays from cache on -resume).

Preview status

The feature is a preview: a warning is emitted whenever an agent is used, and the syntax and configuration surface may change before it is declared stable.

Testing

Unit and integration coverage lives beside the code — 79 new Spock spec files across nextflow.agent, nf-lang parsing/resolution/lowering, and both runner plugins.

End to end, the first five examples were validated against a real model on the local executor with the containerized pi runner:

example	result	agent RPC invocations
01_structured-output	ok, 15s	1
02_two-agents	ok, 43s	4
03_skills	ok, 16s	1
04_tool	ok, 12s	1
05_tool-parallel	ok, 32s	4

Every task ran containerized (remote=true) with no rejected RPC connections, and the outputs were checked for substance rather than exit status — 05_tool-parallel's reverse complements are correct, and its non-ACGT input was correctly flagged.

Notes for reviewers

• Release plumbing. release.sh gains a step 1 that publishes the nf-agent-pi runner image to public.cr.seqera.io/nextflow — the registry and namespace the release already pushes nextflow/nextflow to — reusing the existing SEQERA_PUBLIC_CR_* credential. It is deliberately ordered first, so a failure happens before anything irreversible is published. build.yml adds a QEMU setup step for the multi-arch build and raises the release job timeout from 10 to 45 minutes. This is the part most worth a close look, and it can be split into a follow-up PR if you'd prefer to land the language and runtime first.
• Version guards. build.gradle gains two checks that run on upload, deploy and release: one fails the release if the nf-agent-pi image build context drifted from its VERSION, since the image tag is immutable once published; the other fails if any plugin declares a nextflowVersion newer than the tree's own VERSION.
• Data fixtures. The four examples that assemble a real genome share one fetched FASTQ through examples/data/ and a data symlink, so the read set downloads once rather than per example. The fixture is gitignored; the directory and symlinks are committed so a fresh clone has the wiring in place.

---

🤖 Generated with Claude Code

[netlify]

✅ Deploy Preview for nextflow-docs ready!

Name	Link
🔨 Latest commit	74e5201
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs/deploys/6a858f94c5a0f300081fc3a9
😎 Deploy Preview	https://deploy-preview-7477--nextflow-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[bentsherman]

Please see the comments I left on the internal PR. Several of them are still unresolved

I will make a PR for the docs and ADR today or tomorrow

[pditommaso]

Oops, I had missed some of your comments since they were buried in the long GH conversation.

I've replied already to some of them and exploring the missing improvements requested.

However I'd prefer to not consider them blocking and move this forward.

[bentsherman]

The main follow-up we need to make before 26.10 is to have more comprehensive support for composite outputs

The current state:

• You can have multiple outputs, but this is a bad pattern because they become unrelated channels. The only way to re-join them would be to define two record types, which is bad data modeling
• You can have a record type, but you can't specify Path fields, presumably because there is no way to infer the expected file name from the record type

I think the solution is to support destructured records:

output:
record(id: id, result: file('result.txt'))

This seems to be the only way to support a structured output that contains files. Destructured record input is already trivial to implement because it is semantically equivalent to a record type input.

It is a different paradigm of output collection. Instead of giving the agent a record schema with which to structure its response, you simply give it the list of expected output file names. The agent need not know anything about the output structure aside from this.

It does match the way users currently think about tasks though. A process output typically contains (1) metadata forwarded from the inputs and (2) files written by the task. Any output "metadata" or "scalars" must be written to a file (or env() or stdout(), which are just shorthands for files)