feat(language): integrate hand-written C++ kernels via external_source

[lyfne123]

Summary

Adds a way to call an existing hand-written AICore kernel from a PyPTO orchestration by declaring only its signature — no PyPTO tile codegen for the kernel body.

A DSL function marked @pl.function(type=AIC/AIV, external_source="...cpp") with an empty ... body is a header-only declaration: the compiler assigns it a func_id, emits the orchestration submit exactly as for a DSL kernel, and compiles the referenced .cpp as the InCore kernel. A mixed AIC+AIV kernel is a pl.FunctionType.Group of one AIC + one AIV external member, dispatched as a single MixedKernels submit.

Also exposed under @pl.jit via @pl.jit.extern (single core, or core_type="mixed" which auto-expands to the AIC+AIV+Group form). The referenced .cpp content participates in the JIT cache key, so editing the kernel triggers recompilation.

@pl.function(type=pl.FunctionType.AIV, external_source="kernels/aiv/pa.cpp")
def PA_AIV(self, q: pl.Tensor[...], out: pl.Out[pl.Tensor[...]]) -> pl.Tensor[...]:
    ...   # body lives in the C++ source

# @pl.jit route
@pl.jit.extern(core_type="mixed", aic_source="k.cpp", aiv_source="k.cpp")
def pa(q: pl.Tensor[...], out: pl.Out[pl.Tensor[...]]) -> pl.Tensor[...]: ...

The hand-written kernel needs no adaptation: it already exports the same fixed extern "C" void kernel_entry(__gm__ int64_t*) entry PyPTO-generated kernels use (runtime dispatches by func_id), so its .cpp is copied verbatim into kernels/<ct>/<name>.cpp and listed in kernel_config.py identically to a generated kernel.

Changes

• decorator / parser: external_source kwarg + path resolution; header-only (empty ...) parse with AIC/AIV validation.
• verifier: exempt external kernels from ReturnParamsExplicit (a header has no ReturnStmt) — the only pass invariant an empty body violates; all other passes no-op on an empty body.
• backend: skip ptoas for external kernels, copy the .cpp into kernels/<ct>/<name>.cpp.
• jit: @pl.jit.extern decorator, specializer rendering (single + mixed auto-group), and .cpp content folded into the source hash.
• docs: en + zh external-kernel integration guide.

Testing

• Unit tests: tests/ut/codegen/test_external_kernel.py (5) + tests/ut/jit/test_extern_kernel.py (6) — manifest, MixedKernels submit, error cases, JIT specialization, cache invalidation.
• Regression: codegen + parser + jit suites (~2000 tests) green; ruff clean.
• On-device ST: tests/st/runtime/external_kernel/ drives a hand-written AIV kernel across 4 SPMD blocks — passes on both a2a3sim and real a2a3 via the standard harness.
• Documentation updated (en + zh).

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f97ae197-62f1-4e3d-b914-08cc0a4ad935

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR adds support for integrating hand-written external C++ kernels into PyPTO via a new external_source mechanism, usable through @pl.function/@pl.program or @pl.jit.extern. It spans parser validation, verifier exemptions, backend codegen bypass with source copying, JIT cache hashing, documentation, and tests.

Changes

External Kernel Feature

Layer / File(s)	Summary
Decorator support for external_source python/pypto/language/parser/decorator.py	Adds external_source keyword to @pl.function, resolving relative paths and stashing metadata for class methods within @pl.program.
Parser validation python/pypto/language/parser/ast_parser.py	Adds signature-only body detection and enforces AIC/AIV-only, empty-body requirements for external kernel declarations.
Verifier exemption src/ir/verifier/verify_return_params_explicit.cpp	Skips return-statement verification for functions with the external_source attribute.
Backend codegen bypass python/pypto/backend/pto_backend.py	Detects external kernels/groups, enforces uniform group composition, copies referenced .cpp sources, and skips DSL codegen/ptoas.
JIT extern decorator and specialization python/pypto/jit/decorator.py, python/pypto/jit/specializer.py	Adds @pl.jit.extern, extern metadata fields, source-hash-based cache invalidation, and specializer rendering of single-core/mixed extern declarations.
Documentation docs/en/dev/language/01-external-kernels.md, docs/zh-cn/dev/language/01-external-kernels.md	New English and Chinese guides covering the ABI contract, declaration routes, compiler behavior, and restrictions.
Unit tests tests/ut/codegen/test_external_kernel.py, tests/ut/jit/test_extern_kernel.py	Validates parsing, codegen copy/manifest/dispatch behavior, and JIT extern decorator/cache-hash validation.
System test tests/st/runtime/external_kernel/*	Adds a hand-written AIV kernel fixture and an end-to-end SPMD dispatch test.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Program as pl.program
  participant Parser as ASTParser
  participant Backend as pto_backend
  participant FS as Filesystem

  Program->>Parser: parse_function(external_source=...)
  Parser->>Parser: validate AIC/AIV + empty body
  Parser-->>Backend: func_attrs with EXTERNAL_SOURCE_ATTR
  Backend->>FS: read referenced .cpp
  Backend->>FS: copy to kernels/<core_type>/<name>.cpp
  Backend-->>Backend: skip PTOCodegen and ptoas

Loading

sequenceDiagram
  participant User as Caller
  participant JIT as jit.extern decorator
  participant Spec as Specializer
  participant Ctx as SpecializeContext

  User->>JIT: `@pl.jit.extern`(core_type, source/aic_source/aiv_source)
  JIT->>JIT: resolve and validate source paths
  JIT->>Ctx: build_specialize_context(external_core_type, ...)
  Ctx->>Spec: _specialize_function(func_type="extern")
  Spec->>Spec: _render_external() emits `@pl.function`(external_source=...)

Loading

Suggested labels: enhancement

Poem

A rabbit found a kernel, hand-carved and true,
"No need to reinvent," it hopped, "just wire it through!"
With .cpp in paw and external_source in sight,
It skipped the tiles and passes, straight into the night.
🐇✨ Mixed cores dancing, AIC and AIV in tune —
Compiled by moonlight, tested by noon.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main feature: adding hand-written C++ kernels via external_source.
Description check	✅ Passed	The description is detailed and directly matches the changeset, covering external kernels, JIT, backend, and tests.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[gemini-code-assist]

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 11bace40cd

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[Hzfengsy]

Approved. Review threads resolved; both P2 bot comments addressed by the author (stub + source-hash recomputation).