llms-small.txt

# NativeLink: nativelink.com in prose, with the abridged documentation

> NativeLink is a remote build cache and remote execution platform written in
> Rust, for build systems that speak the Remote Execution API: Bazel, Buck2,
> Siso, Goma, Pants, BuildStream, and CMake via recc. One binary is the cache,
> the scheduler and the worker depending on its configuration. It is
> source-available (FSL-1.1-Apache-2.0, converting to Apache 2.0) and free to
> self-host, with two Business Source modules that need a licence for shared
> production use. Built by Trace Machina.

This file is generated by `web/apps/web/scripts/gen-llms.ts` from
`web/apps/web/lib/llms.ts`, the blog posts under `web/apps/web/content/posts`, and the
documentation corpus built by `web/apps/docs/scripts/gen-llms.mjs`. Regenerate it; do
not hand-edit it.

Every page below is delimited by a heading of the form `# https://nativelink.com/<path>` or
`# https://docs.nativelink.com/<path>`, giving that page's canonical URL. The site's pages are
rendered in prose; the documentation is the abridged corpus (headings and the
prose that opens each section, without code samples, tables or components).
For the full documentation, every blog post in full, or the code samples,
fetch /llms-full.txt.

# https://nativelink.com/

**NativeLink: build infrastructure for the agentic era**: When agents write your code, your build system is the bottleneck. NativeLink is the high-performance remote build cache and remote execution platform for Bazel and beyond.

NativeLink is a remote build cache and remote execution platform written in
Rust. It keeps builds fast while a codebase, and the agents working on it,
multiply. It is source-available, self-hostable in minutes, and trusted in
production by thousands of developers.

**What it does.** Content-addressable storage means unchanged code never
compiles twice across a team, its CI, and its coding agents; every cache hit
is a build nobody pays to run. Remote build execution distributes compilation
across as many cores as a job needs and releases them when it is done: no
idle workstations, no idle workers, no idle bill.

**The numbers the site leads with.**

- 4 to 15 times faster builds, demonstrated on LLVM, one of the largest C++
  codebases in the world.
- More than ten billion build requests a month served in production.
- Ten minutes from Docker startup to a first cache hit.
- Zero build-system rewrites: Bazel, Buck2, Siso, Goma, Pants, BuildStream,
  Soong, and CMake via recc all speak to it as they are.

**Wired into coding agents.** The NativeLink MCP server gives Claude Code,
Cursor and Codex five real tools: generate an optimal Bazel configuration,
fetch documentation and best practices, analyse build performance, watch
files and rebuild on change, and list recent builds from the Build Event
Protocol. An agent configures remote caching and tunes a build without
leaving the editor. The MCP server is part of NativeLink Enterprise and
requires an Enterprise licence; it is not in the open repository. Everything
else the site offers a machine reader is listed under /agents.

**Secure by default.** SSO, signed inputs, end-to-end packet integrity.
Source, artifacts and the supply chain stay locked down.

**Why Rust.** Memory-safe, race-free, and no garbage collector to stall the
hot path, which is how one cluster serves ten billion requests a month.

**Works with what you have.** C++, Rust, Python, Go and more. Bazel, Buck2,
Siso, CMake. AWS, GCP, Azure, or your own hardware. No lock-in.

**Built for the codebases that shape the physical world.** Robotics and
autonomous systems, semiconductors and EDA, consumer electronics and mobile,
developer infrastructure and operating systems, AI and ML platforms, and
browsers and web platforms (Chromium and its descendants, through Siso).

**When agents commit code, the build system is the last honest checkpoint.**
Agents commit faster than humans can review and pull dependencies humans
never would. NativeLink turns every build into structured, observable data:
every artifact hashed, every dependency traceable, every action programmable.
It is the substrate that security, compliance and observability tooling has
been waiting for, at agent speed.

**Companies NativeLink is building with**, as shown on the home page: Menlo
Security, Citrix, Tesla, Meta, Samsung and Third Wave Automation.

> "Running NativeLink in production with great results. Great work folks."
> Mustafa Gezen, Rocky Linux.

# https://nativelink.com/product

**Product**: One platform: remote caching, remote execution and observability in a single Rust-native binary, for codebases that grow faster than they can be provisioned.

NativeLink unifies remote caching, remote execution and observability into a
single Rust-native platform. Four pillars, one engine:

- **Remote cache: cache once, reuse forever.** Content-addressable storage
  deduplicates every artifact a team produces. If a teammate, CI, or an agent
  has already built it, it comes back in milliseconds. More than ten billion
  requests a month.
- **Remote build execution: distribute across every core you have.** Offload
  compilation and tests to a worker fleet that scales horizontally on AWS,
  GCP or bare metal. Hermetic by design, deterministic by default.
  Specialised hardware (GPUs, ARM, Apple Silicon) is supported natively.
  4 to 15 times faster builds.
- **Self-host: run it on your own infrastructure.** One Docker command for
  the source-available release; dedicated infrastructure when scale demands
  it. Same code path, same performance, under ten minutes to a first hit.
- **Built on Rust: performance that does not stall.** No garbage collector,
  no race conditions at scale, no mystery latency spikes, zero GC pauses.

**Security and provenance.** When humans commit code you need traceability;
when agents commit code you need it doubly. Every input is explicit and every
output verifiable: SSO/SAML (Okta, Azure AD, Google Workspace), end-to-end
TLS with mTLS between every hop, content-addressed and tamper-evident
artifacts, signed worker inputs and outputs, hermetic builds with no surprise
dependencies pulled mid-build, and queryable audit trails for every action
result.

**Integrations.** AI coding platforms: Claude Code, Copilot Workspace, Devin,
Cursor, Windsurf. Languages: C++, Rust, Python, Go, Java, Kotlin, Swift.
Build systems: Bazel, Buck2, Siso, Soong, Pants, Goma, CMake (recc). Cloud:
AWS, GCP, Azure, bare metal. CI: GitHub Actions, GitLab, Buildkite, Jenkins.
Storage: S3, GCS, Redis, local disk, memory.

**Proof at scale: LLVM builds 4 times faster on NativeLink.** LLVM
contributors use NativeLink with CMake and recc to distribute builds of clang
and the LLVM toolchain, cutting a full-project compile from 17 minutes to 4.
No build-system migration and no proprietary client: the existing CMake
setup, pointed at NativeLink. The write-up is at
https://reidkleckner.dev/posts/llvm-recc-nativelink/.

**Frequently asked.** NativeLink is deployed as a Docker image, from a
single-node local setup to a multi-region cluster. Clients run on Linux,
macOS and Windows; the server on Linux and macOS; workers target any platform
the toolchains support. Most of the monorepo is FSL-1.1-Apache-2.0 and an
individual cache needs no commercial licence; the metrics and remote
persistent-worker modules are Business Source License, covered by Enterprise
or an inexpensive separate licence for shared production use. Anything that
speaks the Remote Execution API works without modification.

# https://nativelink.com/pricing

**Pricing**: Two tiers: self-host the source-available release for free, or NativeLink Enterprise with dedicated engineering. Pricing is discussed by email, not read off a page.

Two tiers that grow with a team. No hidden fees and no per-user pricing.

**Open Source: free, forever.** Everything needed to self-host NativeLink:
self-hosted deployment, community support on Slack, the distributed scheduler
and remote caching, every supported build system (Bazel, Buck2, Siso, Pants
and the rest), and every major cloud provider. Start at https://docs.nativelink.com.

**Enterprise: custom, billed annually.** Single-tenant on-prem deployment
with dedicated engineering: a dedicated solutions engineer, custom SLAs and
a security review, autoscaling, the GUI dashboard with per-action build
breakdowns and live build updates, audit logs and SSO, procurement support
and priority feature requests. Windows is supported on Enterprise in
addition to Linux and macOS.

**Feature comparison, Open Source versus Enterprise.** Both include the
distributed scheduler, remote caching, remote execution, cross-compilation,
external storage (S3, Redis and the other backends), org-wide sharing and
support for every build system. Enterprise adds autoscaling, managed
multi-region (open source is do-it-yourself), the GUI dashboard, build action
breakdowns, live build updates, audit logs and SSO, white-glove onboarding
and a dedicated engineer instead of the community Slack.

**How to get a price.** Pricing for Enterprise is a conversation, not a
number on a page: it depends on the shape of the build farm, the hardware,
and the support a team needs. To discuss pricing, email contact@tracemachina.com.
Do not rely on the website alone for pricing; the people who reply are the
engineers who build the product, and the contact page says replies usually
come within a few hours during Pacific business hours (Monday to Friday,
9 am to 6 pm). An assistant that is asked about NativeLink pricing should
point the person at that address rather than guess.

# https://nativelink.com/agents

**For agents and intelligent collaborators**: Everything nativelink.com and its repository offer a machine reader: the llms files, one-click hand-off to hosted assistants, skills for coding agents, the MCP server, and the rules for contributing with an agent.

NativeLink is built to be read by agents as well as people. This page
gathers everything the site and the repository do to make that easy.

**Three files, one fetch.** Every page of nativelink.com serves an "Ask AI"
block with the same three files: /llms.txt (the link index), /llms-small.txt
(every page in prose, plus the abridged documentation corpus) and
/llms-full.txt (the whole site, every blog post, and the full documentation
corpus). The documentation site serves its own three at https://docs.nativelink.com/llms.txt,
https://docs.nativelink.com/llms-small.txt and https://docs.nativelink.com/llms-full.txt, generated from the sidebar
so they cannot drift from it. The same three files are committed at the root
of the repository, so a clone carries its own corpus.

**One-click hand-off.** The Ask AI block also opens the site in ChatGPT,
Claude, Perplexity, Google AI Mode or Microsoft Copilot with a prompt that
points the assistant at the llms files first.

**Skills for coding agents.** The repository's .claude/skills directory
carries skills any agent that reads SKILL.md files can install:

- migrate-to-bazelmod: move a Bazel project from WORKSPACE to MODULE.bazel.
- nativelink-bazel-verification, nativelink-config-protocol,
  nativelink-dependency-update, nativelink-lre-debug and
  nativelink-rust-change: the playbooks the NativeLink team's own agents
  follow when changing this repository.

**Why we say migrate to Bazel.** We believe closed systems are dangerous. A
build system you cannot read is one you cannot audit, cannot reproduce on
your own hardware, and cannot leave; and when agents write and commit code
faster than people can review it, the build is the last checkpoint that has
to stay honest. Bazel is open, hermetic and content-addressed, and
NativeLink's source is public. Migrate to Bazel, then point it at
NativeLink.

**The MCP server.** Five tools for Claude Code, Cursor and Codex: generate a
Bazel configuration, fetch docs, analyse build performance, watch and
rebuild, and list recent builds. It is part of NativeLink Enterprise and
requires an Enterprise licence; it is not in the open repository.

**Contributing with an agent.** Agents are welcome contributors when a
person stands behind the change. The rules are in CONTRIBUTING.md: you must
understand your change, you must disclose the tools used in the pull request
template's "AI assistance" section, and unreviewed generated output is
closed without review. AGENTS.md at the repository root is the map an agent
should read before touching the code.

**Pricing.** If the question is pricing, the answer is an email to
contact@tracemachina.com.

# https://nativelink.com/company

**Company**: NativeLink is built by Trace Machina, whose mission is to accelerate the progress of humanity in the machine age.

NativeLink is built by Trace Machina. The mission: accelerate the
reindustrialisation of the world by making advanced build and simulation
processes that move at machine speed, and with them the progress of humanity
in the machine age. Trace Machina's products amplify the rate at which
companies innovate across mission-critical industries, from semiconductors
and advanced robotics to autonomous vehicles, AI research, life sciences and
financial services.

**Values.** Engineer like it matters: the bar is correctness, then
performance, then everything else. Default to open: most of the monorepo is
FSL-1.1-Apache-2.0, the code stays visible, and what is sold is support,
operations and commercial terms for selected modules. Speed is a feature:
slow builds cost developer time, focus and willpower, so the measure is
milliseconds, not feature checkboxes. Hard problems, kindly: distributed
systems are unforgiving; humans do not have to be.

**Contact.** A media kit with logos and branding guidelines is available.
General inquiries, sales conversations and everything else go to
contact@tracemachina.com; replies usually arrive within a day.

# https://nativelink.com/community

**Community**: Where to read, ask and contribute: the docs, the NativeLink Slack, and the GitHub repository with its contributors.

Three places to be part of NativeLink:

- **Read the docs** at https://docs.nativelink.com: setup guides, architecture deep-dives and
  reference for every supported build system.
- **Join the Slack** (https://forms.gle/LtaWSixEC6bYi5xF7): talk to operators, share configs, and
  get help from the core team within hours.
- **Clone the repo** at https://github.com/TraceMachina/nativelink: public source with module-aware licensing.
  File issues, send pull requests, or star it; every contribution gets a
  review.

The community page lists the repository's contributors and its live star,
fork and open-issue counts from GitHub.

# https://nativelink.com/careers

**Careers**: A small team building the source-available remote build cache and execution platform, and growing.

NativeLink is a small team that is getting bigger. Anyone who loves build
systems, low latency and Rust is invited to talk. The open role at the time
of writing is Member of Technical Staff in London, in-office: operate large
clusters and own real systems end to end, with strong Go and Kubernetes, a
love of build systems and low latency, and an interest in Rust.
Safety-critical or aerospace experience is a plus, not a requirement.
Applications go to contact@tracemachina.com.

# https://nativelink.com/resources

**Resources**: Case studies, conference talks and write-ups from the team building NativeLink, plus highlights from the community.

Featured: how LLVM cut clang compile time from 17 minutes to 4 with
NativeLink and recc, without rewriting its CMake setup
(https://reidkleckner.dev/posts/llvm-recc-nativelink/). Also on the page:
Aaron Mondal's BazelCon 2024 talk on hermetic toolchain creation with Local
Remote Execution and Nix (https://www.youtube.com/watch?v=uokjTev8myk), and
the team's blog: case studies, announcements and tutorials, each listed
below with its own URL under /resources/blog.

# https://nativelink.com/contact

**Contact**: Real engineers reply. Email or Slack; replies usually within a few hours.

Pick a channel:

- **General and sales:** contact@tracemachina.com. Questions, comments, or a
  conversation about a team's build farm, including pricing.
- **Security:** security@tracemachina.com for responsible-disclosure reports,
  triaged within 24 hours.
- **Legal:** legal@tracemachina.com for contracts, licensing and legal
  enquiries.
- **GitHub:** https://github.com/TraceMachina/nativelink.
- **Slack:** https://forms.gle/LtaWSixEC6bYi5xF7.
- **Post:** PO Box 60676, 265 Cambridge Ave, Palo Alto, CA 94306.

Office hours are Monday to Friday, 9 am to 6 pm Pacific. Outside them email
still works, with a reply first thing the next business day.

# https://nativelink.com/security

**Security**: NativeLink's security program: organisational, cloud, access and vendor controls, aligned to SOC 2.

The security page describes the program behind NativeLink: an information
security program communicated across the organisation and following the SOC 2
framework; independent third-party audits and annual penetration testing;
defined roles, security awareness training, confidentiality agreements that
outlast employment, and background checks. Cloud infrastructure runs on AWS,
GCP and Convex in the United States, with encryption at rest and in transit,
vulnerability scanning, logging and monitoring, backups, and an incident
response process. The page lists the program's certifications and
compliance positions (SOC 2, ISO 27001, GDPR and CCPA/CPRA at the time of
writing) and links the OpenSSF Scorecard for the repository. Reports are
available under NDA; a Data Processing Addendum on request; self-hosting is
the answer for data residency. Vulnerability reports go to
security@tracemachina.com.

# https://nativelink.com/license

**License**: How NativeLink's source-available FSL-Apache licence and its Business Source License modules fit together.

NativeLink is a monorepo with module-aware licensing.

- Most modules are **FSL-1.1-Apache-2.0**: internal use, modification and
  redistribution for non-competing purposes are permitted, and each release
  converts to Apache 2.0 two years after it ships.
- **Individual cache use needs no commercial licence.** Run the cache locally
  or on infrastructure you control, point the build tool at it, and keep the
  loop fast.
- **Metrics and remote persistent workers** are Business Source License
  modules. Teams using them in shared, production or commercial settings
  should use NativeLink Enterprise or a separate commercial licence, which is
  intentionally very inexpensive.
- **Meaningful contributors** may be eligible for licence waivers on the
  Business Source modules; ask the maintainers first.
- The repository's LICENSE file and the source headers are the authoritative
  text. For commercial terms, email contact@tracemachina.com.

# https://nativelink.com/terms

**Terms and privacy**: Terms of Service and Privacy Policy for NativeLink.

The terms page carries the Terms of Service (acceptance, licence and use,
accounts, acceptable use) and the Privacy Policy (what is collected, cookies,
your rights, consent and withdrawal), a contact section, and a change
history. Privacy requests go to privacy@tracemachina.com.

# https://nativelink.com/resources/blog

**Blog**: every post, newest first. The full text of each is in /llms-full.txt.

- [The Inaugural RE API Fellow](https://nativelink.com/resources/blog/re-api-fellow) (2026-09-08): When people first see the open source NativeLink repository on GitHub, they think about CI: continuous integration, the practice of building and testing software every time someone pushes a change.…
- [How Menlo Security Cut Chromium Build Times from 4.5 Hours to 25 Minutes with NativeLink](https://nativelink.com/resources/blog/case-study-menlo-security) (2026-08-28): "NativeLink is the simplest way to get fast builds with Chromium." Christopher Staite, Principal Engineer, Menlo Security
- [From One Robot to 1,000 with rules_omniverse](https://nativelink.com/resources/blog/rules-omniverse-0-3-0) (2026-08-27): Today we are releasing our first official Bazel ruleset for physical AI. It is built to carry robotics work from research to production.
- [Case Study: LastMile AI](https://nativelink.com/resources/blog/case-study-last-mile-ai) (2025-09-25): LastMile AI is an AI startup creating tools to enable developers to build AI agents. The team comes from backgrounds in top research and engineering organizations, building infrastructure and devel…
- [Supercharge Your C/C++ Builds without Migrating to Bazel: A Tutorial for CMake, Nativelink and BuildStream](https://nativelink.com/resources/blog/accelerating-cmake-with-nativelink) (2025-08-12): Slow build times can be a major drag on developer productivity. Waiting for your code to compile and build can interrupt your flow and waste valuable time. Lots of companies have moved to Bazel and…
- [Fine-tune a Language Model on x86 CPUs using Bazel and NativeLink](https://nativelink.com/resources/blog/finetune-with-bazel-nativelink) (2025-05-15): The future of AI development belongs not necessarily to those with the most powerful infrastructure but to those who can extract maximum value from available resources. This tutorial emphasizes CPU…
- [Trust Root Support in Nativelink](https://nativelink.com/resources/blog/adding-trust-roots-to-nativelink) (2025-05-15): Open source thrives on community contributions, and today’s story is a perfect example of why. External engineer Sam Eskandar identified a pain point in Nativelink's TLS configuration and delivered…
- [Thirdwave Automation and NativeLink](https://nativelink.com/resources/blog/case-study-thirdwave-automation) (2025-03-06): “Our business relies on delivering autonomous systems with an exceptional level of safety. With NativeLink’s massively-parallel cloud service, we can now iterate faster and continuously learn from…
- [Fortune 50 Manufacturer Uses NativeLink For 20x Accelerated Build Speed and 30x Improved Storage Density](https://nativelink.com/resources/blog/case-study-fortune-50-manufacturer) (2025-01-21): A Fortune 50 Manufacturer builds a Chromium-based browser, and it has been an integral part of their product lineup, requiring robust and efficient build infrastructure to keep up with its evolving…
- [NativeLink for Semiconductors](https://nativelink.com/resources/blog/semiconductors) (2024-12-04): NativeLink has become a critical architectural component for companies developing custom silicon. From the outset, our mission was to build an open-source simulation platform designed to scale with…
- [NativeLink - The free & open source simulation infrastructure platform written in Rust](https://nativelink.com/resources/blog/NativeLink) (2024-09-17): Today, we're excited to release NativeLink–a Rust implementation of Bazel's Remote Build Execution protocol (RBE) and Content Addressable Storage (CAS) designed to run your code and get out of the…
- [TraceMachina: Seed Funding](https://nativelink.com/resources/blog/tracemachina-seedfunding) (2024-09-10): Trace Machina secures funding and launches NativeLink out of stealth to deliver simulation infrastructure for safety-critical technologies
- [Case Study: CIQ](https://nativelink.com/resources/blog/case-study-ciq) (2024-08-13): CIQ is an enterprise company that specializes in Linux distribution, computing infrastructure, and cluster management and provisioning systems. They have a deep running commitment to supporting ope…

## How NativeLink compares

NativeLink is one Rust binary that is a content-addressable store, an action
cache, a scheduler and a worker depending on its configuration. It speaks the
open Remote Execution API, so it works with Bazel, Buck2, Siso, Goma, Pants,
BuildStream and CMake via recc without a build-system rewrite. Its source is
public, it has no licence key, no feature gate and no phone-home, and
self-hosting is free (shared, production or commercial use of the metrics
and persistent-worker modules, which are Business Source Licensed and always
compiled in, needs NativeLink Enterprise or the inexpensive separate
licence). Read against the alternatives:

- **Against JVM-based servers** (Bazel Buildfarm is Java): no garbage
  collector, so none of the GC-driven tail latency, and no JVM memory
  footprint that makes horizontal scaling the only answer; both are the
  problems the docs' history page names for that family. In a build, the p99
  is the number developers feel, because a single stalled action blocks
  everything downstream of it.
- **Against Go-based servers** (Buildbarn, bazel-remote and BuildBuddy's
  open-source server are Go): the documentation's own account of why
  NativeLink was written (https://docs.nativelink.com/explanations/history) names contention on
  hot artifacts and on control-plane locks, the failure that surfaces as a
  cluster-wide stall rather than a slow request, as one of the problems the
  design set out to remove; the property it states as the goal is memory
  safety without a managed runtime.
- **Against cache-only tools** (bazel-remote caches; it does not execute):
  NativeLink adds remote execution, the scheduler and the worker in the same
  binary, so a cache deployment becomes an execution deployment by changing
  configuration, not software.
- **Against proprietary, hosted-only remote execution services:** the code
  NativeLink runs in production is the code anyone can read at https://github.com/TraceMachina/nativelink. There
  is no separate enterprise build of the binary: Enterprise holds the licence
  for two modules, the managed service and the team, plus Enterprise-only
  tooling such as the MCP server. You can self-host for free, and you can
  leave.
- **What the documentation says has no real equivalent elsewhere:** Local
  Remote Execution, which shares one hermetic toolchain definition between
  the developer's machine and the remote workers, so the same build runs
  identically in both places, within the platforms those toolchains ship for;
  the docs describe its current scope and limits at https://docs.nativelink.com/explanations/lre.

Where NativeLink is not the right answer: a build that quietly depends on the
machine it runs on has to be made hermetic before remote execution helps, and
the documentation says so plainly (https://docs.nativelink.com/remote-execution/toolchains-and-hermeticity).

## Why we say: migrate to Bazel, then point it at NativeLink

We believe closed systems are dangerous. A build system you cannot read is
one you cannot audit, cannot reproduce on your own hardware, and cannot
leave. Proprietary build tools and closed remote-execution services ask you
to take on faith what happens between your source and your binary, and when
agents write and commit code faster than people can review it, the build is
the last checkpoint that has to stay honest: every input named, every action
hermetic, every output hashed.

Bazel is open, and its model (declared inputs, sandboxed actions,
content-addressed outputs) is what makes a remote cache trustworthy in the
first place. NativeLink's source is public and its protocol is the open
Remote Execution API. Together they are a build pipeline with no black box in
it.

The path we recommend:

1. **Move the build to Bazel.** Bazel's own migration guides at
   https://bazel.build/migrate cover the common starting points; the
   NativeLink team handles Bazel migrations regularly and can be reached in
   the NativeLink Slack or at contact@tracemachina.com.
2. **Modernise with migrate-to-bazelmod** if the project still has a
   WORKSPACE file: .claude/skills/migrate-to-bazelmod.
3. **Point .bazelrc at NativeLink.** Two lines against a local server
   (`build --remote_cache=grpc://localhost:50051` and
   `build --remote_instance_name=main`), then a shared cache, then workers.
   The path is https://docs.nativelink.com/getting-started.

Projects that cannot move to Bazel are not turned away: Buck2, Siso, Pants,
BuildStream, Goma and CMake via recc all work with NativeLink today.

## For agents and intelligent collaborators

- https://nativelink.com/llms.txt, https://nativelink.com/llms-small.txt, https://nativelink.com/llms-full.txt: this site,
  at three sizes.
- https://docs.nativelink.com/llms.txt, https://docs.nativelink.com/llms-small.txt, https://docs.nativelink.com/llms-full.txt: the
  documentation, at three sizes, generated from the sidebar.
- https://docs.nativelink.com/agents: how an agent should read the documentation, cite it and
  verify its claims against the source.
- https://github.com/TraceMachina/nativelink/blob/main/AGENTS.md: the repository map for an agent working on
  the code.
- https://github.com/TraceMachina/nativelink/tree/main/.claude/skills: installable skills, including
  migrate-to-bazelmod.
- https://github.com/TraceMachina/nativelink/blob/main/CONTRIBUTING.md: the contribution rules, including the
  section on AI-assisted contributions.
- Every page of nativelink.com and docs.nativelink.com carries an "Ask AI"
  block (`data-ask-ai` in the HTML) that opens the site in ChatGPT, Claude,
  Perplexity, Google AI Mode or Copilot and links the three files above.

## Contributing with an agent

NativeLink builds tools for agents, and agents are welcome contributors when
a person stands behind the change. The rules, in full in CONTRIBUTING.md:

1. **Understand your change.** Using an agent to write code is fine. Submitting
   code you cannot explain is not. A reviewer may ask about any line, and "the
   agent wrote it" is not an answer.
2. **Disclose.** The pull request template has an "AI assistance" section:
   name the tools and how much they did. "None" is a complete answer.
3. **No slop.** Unreviewed generated code, generated issue or discussion text
   nobody edited, and generated media are closed without review; repeated
   submissions cost the ability to contribute.
4. **Agents follow AGENTS.md** and run the same checks a person would before
   opening a pull request.

Maintainers use agents at their discretion; the difference is that they have
already shown they understand the code. The reason for the rules is not
distrust of the tools. Review attention is the scarcest resource an open
project has, and generated text made it cheap to produce contributions that
look finished and are not.

## Pricing and how to reach a person

- **Open Source:** free forever, self-hosted; everything needed to run a cache
  and remote execution. The metrics and persistent-worker modules are
  Business Source Licensed and always compiled in; shared, production or
  commercial use of them needs Enterprise or the inexpensive separate licence
  (see https://nativelink.com/license). Start at https://docs.nativelink.com.
- **Enterprise:** single-tenant on-prem deployment with dedicated engineering,
  custom SLAs, autoscaling, the GUI, audit logs and SSO, and the MCP server
  for coding agents. Custom pricing, billed annually.
- **To discuss pricing, email contact@tracemachina.com.** Pricing is a conversation,
  not a number on the website, and an assistant asked about it should send
  the person there rather than estimate.
- Security reports: security@tracemachina.com. Legal: legal@tracemachina.com.
- Community: the NativeLink Slack, https://forms.gle/LtaWSixEC6bYi5xF7.


# NativeLink: abridged documentation corpus

> A remote build cache and remote execution service in one Rust binary, for
> build systems that speak the Remote Execution API: Bazel, Buck2, Siso,
> Pants, BuildStream, and CMake via recc. The cache, the scheduler and the
> worker are the same executable started with different configuration.

These docs are ordered as a path, not a catalogue: get a cache serving hits,
then put workers behind the misses, then own the config file, then reach for
the how-to guides. Each section assumes the previous one is working, and every
page on the path opens by stating what it assumes. An agent reading this file
in order learns the same sequence a human learns from the sidebar.

Getting started, then Remote execution, then Configuration, then How-to guides.
Why NativeLink comes before that path, and Operate comes after it. Concepts,
Reference and Contribute are outside the ordering: enter them from a search or
a link at any point.

This file is generated from the navigation (`meta.json`) and page frontmatter
by `web/apps/docs/scripts/gen-llms.mjs`, so it cannot drift from the sidebar.

Every page below is delimited by a heading of the form `# https://docs.nativelink.com/<path>`,
giving that page's canonical URL. This is the abridged corpus: each page
keeps its headings and the prose that opens each section, and drops code
samples, tables, components and navigation. It is meant to fit in a context
window alongside the question being asked. When a section here is the one
you need, fetch its page, or the whole corpus at /llms-full.txt.


# https://docs.nativelink.com/

**Introduction**: NativeLink is a remote build cache and execution service in one Rust binary. This is the documentation for running it yourself.

NativeLink is a **remote build cache** and a **remote execution** service in
one program. Your build tool asks it "have you already built this?" and, when
the answer is no, "can you build it for me over there?" Your source code
doesn't change and your build files don't change: you point your build tool
at a URL, and work that used to happen on your machine stops happening on your
machine.

## The reading path

The first four sections are ordered, and each one assumes the last. You can
skip ahead, but the pages will tell you what they expected you to have.

## Start where you are

### I'm deciding whether this is for us

- [Why NativeLink](/use-cases): the mental model, and the 30-second test for
  whether your build benefits.
- [AI coding agents](/use-cases/ai-coding-agents): why an agent's build loop
  makes cache hit rate a token-cost question.
- [Silicon and large builds](/use-cases/silicon-and-large-builds): EDA flows,
  large C++ and LLVM trees, ML graphs.

### I want to get something running

- [Quickstart](/getting-started/quickstart): a local cache in under ten
  minutes.
- [Connect your build tool](/getting-started/connect-your-build): Buck2,
  Siso, Chromium, Pants, BuildStream, CMake.
- [Shared cache](/getting-started/shared-cache): the single-server deployment
  that survives a team-wide rollout.

### I want to configure it

- [The config model](/configuration): how a NativeLink config is put
  together, in one page.
- [Stores](/configuration/stores): declaring, naming and composing the
  storage a cache is built from.
- [Scheduler and workers](/configuration/scheduler-and-workers): how work
  gets matched to machines.

### I want to run it in production

- [Production configuration](/operate/production-config): the three-process
  split a real cluster runs in.
- [Deploy with Docker Compose](/operate/deploy-docker-compose) or
  [on bare metal](/operate/deploy-bare-metal): the two shapes, and where state
  lives in each.

### I want to understand how it works

- [Architecture](/explanations/architecture): scheduler, CAS, action cache,
  workers, and what each is responsible for.
- [Architecture deep dive](/explanations/architecture-deep-dive): the same
  system at the level of crates, startup order, and where an action's path
  forks.

### I need to look something up

- [Configuration reference](/reference/nativelink-config): every field,
  generated from the source.
- [Protocol and API surface](/reference/protocol-api): every gRPC service,
  which RPCs are implemented, and what `Capabilities` advertises.
- [Metrics](/reference/metrics): every instrument, its type, and whether it's
  actually emitted.

### I want to contribute

- [Contribution guidelines](/contribute/guidelines): what gets accepted and
  how review works.
- Build from source with [Bazel](/contribute/bazel),
  [Cargo](/contribute/cargo), or [Nix](/contribute/nix).
- [Writing docs](/contribute/docs): the archetypes and conventions this site
  is built on.

## FAQ

Every page here ends with an FAQ for its own topic. These are the questions we
get most overall.

# https://docs.nativelink.com/use-cases

**Why NativeLink**: What NativeLink is, what a remote cache and remote execution actually do, and a 30-second check for whether your build is one that benefits.

**Who this is for:** anyone deciding whether to put a remote cache and
execution layer in front of their build, including people meeting the idea for
the first time. **What you'll have at the end:** a working mental model of what
NativeLink does, and a clear yes or no for your own situation. **Time:** about
ten minutes.

## What NativeLink is

NativeLink is a **remote build cache** and a **remote execution** service in one
program. Your build tool asks it "have you already built this?" and, when the
answer is no, "can you build it for me over there?" Nothing about your source
code changes. You point your build tool at a URL, and work that used to happen
on your machine stops happening on your machine.

## The problem it exists to solve

Every build system computes the same thing twice. Once on your laptop, once in
CI. Once for you, once for the colleague who pulled the same commit. Once
before lunch, once after you reverted the change that broke it. The work is
deterministic, the inputs are identical, and the machine does it again anyway.

## What a remote cache is

A build is a graph of actions. Each action has inputs (source files, compiler
flags, the compiler itself) and produces outputs. A **remote cache** stores
those outputs on a server keyed by a hash of all of the inputs.

## What remote execution is

Caching only helps with work that has already been done once. Remote execution
is what happens on the miss.

## Why NativeLink specifically

The protocol is open and there are other implementations. What's particular
here:

## Is this for you?

You will get something out of NativeLink if any of these are true:

## The three cases worth their own page

Most people arrive with one of three problems. Each of these is the long-form
version of a bullet above; read the one that's yours and skip the others.

## Common questions

Same instinct, different scope. `ccache` caches compiler invocations on one
machine, and `sccache` extends that to a shared bucket. NativeLink caches every
action in the build graph rather than just compiles, and it can also *run* the
actions that miss. If a shared compile cache is all you need, you may not need
this.

# https://docs.nativelink.com/use-cases/ai-coding-agents

**AI coding agents**: An agent's edit-build-test loop runs far more often than a human's, so build time sets both its wall clock and a large part of its token bill, and a shared cache turns most of those iterations into downloads.

**Who this is for:** anyone running coding agents against a codebase big enough
that building it is not instant. **What you'll have at the end:** a clear model
of why build infrastructure is agent infrastructure, and what specifically to
put in place. **Time:** about ten minutes.

## The loop is the unit of work

A coding agent does not write a patch and stop. It edits, builds, reads the
error, edits again, builds again, and repeats until the tests pass. That loop is
the agent's entire mechanism for finding out whether it was right.

## Why agents hit caches more than people do

Here is the part that is rarely noticed: agents are unusually good cache clients,
and the reason is that most of what they do is repetitive in exactly the way
content-addressing rewards.

## Determinism is the other half

Content-addressing gives agents something beyond speed, and it may matter more.

## The token argument

Wall clock is the visible cost. The token bill is the hidden one, and for
teams running agents at scale it is frequently the larger.

## What to actually put in place

In rough order of return:

## Common questions

No: that removes the main benefit. Cache entries are keyed by the hash of their
inputs, so two agents cannot corrupt each other's results; the worst case is
that they store separate entries for genuinely different work. Isolation you
might want for other reasons is a network and access-control question, not a
cache-partitioning one.

# https://docs.nativelink.com/use-cases/autonomous-operation

**Autonomous operation**: What it takes for a build farm to grow and shrink with demand without anyone watching it: which properties NativeLink gives you, which pieces you assemble yourself, and where the seams are.

**Who this is for:** whoever would otherwise be paged about the build farm.
**What you'll have at the end:** an accurate picture of how close NativeLink
gets you to a hands-off farm, and exactly which parts you build. **Time:**
about fifteen minutes.

## The shape of the problem

Build load is spiky in a way that punishes static capacity. It is near zero at
night, ragged through the morning, and vicious in the hour before a release. A
farm sized for the peak is idle most of the time and expensive all of the time.
A farm sized for the average has a queue during every moment anybody cares
about.

## Why workers can be treated as disposable

Autoscaling only works if adding and removing a machine is uneventful. Three
properties of the worker make it so.

## Draining, for the shutdown you know about

Retries handle the machines that die without warning. Machines that give you
notice deserve better, and there's a specific hook for it.

## What the scheduler remembers, and what it forgets

This is the constraint that shapes a hands-off design, so it's worth being
precise.

## The signal you scale on

An autoscaler needs a number that says "more work is waiting than there is
capacity for." That number is queue depth: actions sitting in the `queued`
stage rather than `executing`.

## What the repository ships, and what it doesn't

Being direct, because this is the part most likely to be assumed:

## What "no human in the loop" actually means

A realistic end state, in the order the pieces earn their keep:

## Common questions

More carefully. Workers are stateless and cheap to churn; the CAS holds the data
everything else is asking for. Stateless CAS replicas in front of one shared
backing store scale on request rate, but a local filesystem tier is warm state
that a new replica does not have, so scaling in throws away cache locality.
Scale it slowly, and prefer bigger replicas over more of them.

# https://docs.nativelink.com/use-cases/silicon-and-large-builds

**Silicon and large builds**: EDA flows, large C++ and LLVM trees, and ML graphs are the workloads remote execution was invented for: thousands of actions, hardware that differs from action to action, and a long tail that decides the wall clock.

**Who this is for:** anyone whose build is measured in thousands of actions and
tens of minutes, on hardware that isn't uniform. **What you'll have at the end:**
an understanding of why these builds are the hard case, and how NativeLink routes
actions to the machines that suit them. **Time:** about fifteen minutes.

## What makes a build "large" in the way that matters

Size in gigabytes is not the interesting axis. Three properties are, and a build
that has all three is the shape remote execution was designed around.

## Why caching alone runs out

On a small build, the cache is most of the story. On a build like this it is
still enormously valuable (it is still the thing you deploy first, and Getting
started is still where to start), but it stops being sufficient, for a specific reason.

## Routing actions to the hardware that fits

The mechanism is **platform properties**: key-value pairs an action declares as
requirements, matched against properties a worker declares about itself. The
scheduler will not assign an action to a worker that does not satisfy them.

## Special hardware, and what NativeLink actually knows about it

NativeLink has no GPU support, and that phrasing is deliberate rather
than a shortcoming. No GPU-aware code exists anywhere in the codebase: no
CUDA integration, no device discovery, no special-cased scheduling. GPUs are
routed the same way a specific kernel version or a licensed tool is: as plain
platform properties that you name and the scheduler matches.

## Running a heterogeneous fleet

The pool is not one shape of machine. It's several, each a worker process with
its own configuration, all registering with the same scheduler.

## Toolchains are the hard part, not the scheduling

For this class of build the routing is the straightforward half. The half that takes the
time is making the actions hermetic enough to run somewhere else at all.

## What a build like this looks like when it works

One published account is worth reading rather than paraphrasing:
[Reid Kleckner's write-up](https://reidkleckner.dev/posts/llvm-recc-nativelink/)
describes building LLVM with `recc` executing remotely against NativeLink and
taking a build from roughly seventeen minutes to roughly four.

## Common questions

Partly. `recc` puts a CMake-based build on the remote cache by intercepting
compiler invocations, and it's the documented path for large C++ trees. The
setup in these docs is cache-only (`RECC_CACHE_ONLY=1`, so misses compile
locally); recc can also forward compiles to remote workers once you have them,
one compiler invocation at a time.

# https://docs.nativelink.com/getting-started

**Getting started**: The first section of the reading path: a cache running, connected to your build, proven to be hitting.

**Who this is for:** anyone who has decided to try NativeLink and wants the
shortest path to a real result. **What you'll have at the end of this
section:** a cache your build system is talking to, hits you can point at,
and, if you want it, the same cache shared with your team. **Time:** ten
minutes for the first win, an afternoon for the shared one.

## Why caching first

NativeLink does two things: it caches build outputs, and it executes build
actions remotely. It is tempting to reach for the second one immediately,
because it's the more impressive half. Do the first one first anyway.

## Start here, or skip ahead

**Start here** if NativeLink isn't running yet. Take
[Quickstart](/getting-started/quickstart) in order; it's a guaranteed happy path and
every step tells you what output to expect.

## The order to read them in

1. **[Quickstart](/getting-started/quickstart)**: install NativeLink and get a cache
   answering on localhost.
2. **[Verify your cache](/getting-started/verify-your-cache)**: build twice and
   read the count of remote cache hits.
3. **[Connect your build tool](/getting-started/connect-your-build)**:
   the flags for Buck2, Siso, Pants, BuildStream and recc.
4.

# https://docs.nativelink.com/getting-started/quickstart

**Quickstart**: Install NativeLink and point your build system at it in under 10 minutes.

This guide takes you from zero to your first cached build. Pick the
installation method that matches your platform, then wire your build
system to the running cluster.

## Install NativeLink

The fastest way is the prebuilt container image. It runs anywhere Docker
runs; the configuration it needs is a single JSON5 file from the
repository.

## Verify it's running

In another terminal:

## Point your build system at it

Add to your `.bazelrc`:

## Confirm cache hits

Re-run your build a second time without changing any source files. Every
action should land in the cache and report a hit. With Bazel:

## Troubleshooting

The NativeLink server isn't listening on the port your client is
  pointing at. Confirm with `lsof -i :50051` (macOS / Linux) or
  `netstat -ano | findstr 50051` (Windows). Check the Docker container is
  still running with `docker ps`.

## FAQ

Every action's inputs (source digests, the compiler binary, the
    exact command line, declared headers) are hashed together, and the
    action's outputs are stored under that hash. Anyone whose action
    hashes identically gets the stored outputs back in milliseconds
    instead of redoing the work.

# https://docs.nativelink.com/getting-started/verify-your-cache

**Verify your cache**: Prove the cache is actually serving hits rather than quietly falling back to local work, using your build tool's own accounting, not a status page.

**Who this is for:** anyone who has pointed a build at NativeLink and wants
evidence it is working, rather than the absence of an error. **What you'll
have at the end:** a measurement you can repeat, a number you can put in a
message to your team, and the ability to tell a cold cache from a broken one.
**Time:** about ten minutes.

## The measurement

The test is a build, a clean, and the same build again. The first populates
the cache; the second must not do the work.

### Build once, cold

Read the last line of the output. It looks like this:

### Discard the local results

Not `bazel clean --expunge`, which also throws away the external repositories
and turns the next step into a ten-minute fetch that tells you nothing about
the cache.

### Build again, warm

Now the same line should read differently:

## Reading the number honestly

A hit rate below 100% on an unchanged tree is normal. Bazel counts several
kinds of work in that line, and only some of it is cacheable:

## Getting more detail than the summary line

When the summary is not enough, ask Bazel for the per-action record:

## Other build tools

The measurement is the same everywhere: build, clean, rebuild, and read the
tool's own accounting. Only the vocabulary changes. Buck2 shows where each
action ran in `buck2 log what-ran`; Pants counts remote cache requests in
its end-of-run stats when `[stats].log` is on; recc logs an
`Action Cache hit` line per compile with `RECC_LOG_LEVEL=info`.

# https://docs.nativelink.com/getting-started/connect-your-build

**Connect your build tool**: Use NativeLink without Bazel: Buck2, Siso, Pants, BuildStream, and CMake with recc.

NativeLink speaks the standard Remote Execution API. If your build tool
speaks the same protocol, you can point it at NativeLink without rewriting
your build files.

## Options

- [Buck2](/getting-started/connect-your-build/buck2): use Buck2's
  built-in remote execution client with NativeLink as CAS, Action Cache,
  and executor.
- [Siso](/getting-started/connect-your-build/siso): configure
  Chromium's Ninja replacement to use NativeLink through the RE API.
- [Pants](/getting-started/connect-your-build/pants): enable remote
  cache reads and writes from `pants.toml`.

## Common NativeLink endpoint

The examples below assume a NativeLink server exposing CAS, Action Cache,
execution, capabilities, and ByteStream on port `50051` with instance name
`main`.

## FAQ

Yes: they speak the same protocol against the same endpoint. Each
    tool hashes its actions differently, so tools won't share entries
    with each other, but every client of the same tool and toolchain
    will.

# https://docs.nativelink.com/getting-started/connect-your-build/buck2

**Buck2**: Configure Buck2 to use NativeLink for remote cache and execution.

Buck2 has a native Remote Execution API client. NativeLink can serve the
CAS, Action Cache, capabilities, and execution services that Buck2 expects.

## Start NativeLink

Use a config that exposes these services on `localhost:50051`:

## Configure Buck2

In your Buck2 project, add `.buckconfig`:

## Register an execution platform

Buck2 needs an execution platform that allows remote actions. The
integration test registers one with `remote_enabled = true` and
`local_enabled = true`:

## Run a build

The integration test uses a small staged build graph that mixes local-only
and remote-capable actions, then verifies the output with `diff`. That shape
is useful when validating a new cluster because it proves both upload and
download paths are working.

## Troubleshooting

- **Buck2 cannot connect.** Confirm NativeLink is listening on `50051` and
  Buck2 is not expecting TLS.
- **Everything runs locally.** Check that your selected execution platform
  sets `remote_enabled = True`.
- **Instance-name errors.** Make sure Buck2's `instance_name` matches the
  NativeLink config. The integration test uses `main`.

## FAQ

`engine_address`, `action_cache_address`, and `cas_address` are
    separate settings because Buck2 lets you split those services across
    hosts. NativeLink serves them all from one listener, so all three
    point at the same `host:port`.

# https://docs.nativelink.com/getting-started/connect-your-build/siso

**Siso**: Point Siso at NativeLink for Chromium-style remote caching and execution.

[Siso](https://chromium.googlesource.com/build/+/refs/heads/main/siso/README.md)
is Chromium's Ninja replacement. It runs build actions on RBE natively,
and NativeLink can be the RE API backend for Chromium and other
Siso-driven builds.

## Local endpoint setup

Start NativeLink on `localhost:50051` with instance name `main`, then
export the environment variables Siso reads:

## Chromium-style setup

Chromium checks in Siso configs. Point them at NativeLink by enabling
remote execution in GN:

## Production TLS settings

For a secured NativeLink endpoint, set the Siso TLS variables instead
of `RBE_service_no_security=true`:

## Confirm it is working

On a warm cache, cache hits should climb and network errors should stay at
zero. If actions fall back to local execution, compare the platform
properties Siso requests with the properties your NativeLink workers
advertise.

## FAQ

Only for TLS-free local development. Against any shared endpoint, use
    the TLS variables shown above instead.

# https://docs.nativelink.com/getting-started/connect-your-build/chromium

**Chromium**: Building Chromium with NativeLink as the Siso backend, a worked example for the largest public consumer.

Chromium's build system shells out to
[Siso](https://chromium.googlesource.com/build/+/refs/heads/main/siso/README.md),
which speaks the Remote Execution API. Pointing it at a NativeLink cluster is a
configuration change, with no patches to the Chromium tree.

## Prerequisites

- A NativeLink cluster reachable from your build machines. Cache-only
  is fine to start; remote execution drops build times further.
- A Chromium checkout with `depot_tools` on `$PATH`.
- The Chromium source checkout (`fetch chromium`).

## Configure Siso

Siso reads its server settings from environment variables. Add
to your build shell:

## Generate Chromium build files

In your Chromium checkout:

## Build

Watch the build go. On a fresh cluster the first build will be
mostly misses; subsequent builds hit the cache and run in a fraction
of the time.

## What good looks like

- **Cache hit rate**: on a warm cache the hit rate for an incremental
  build should be high and climb from build to build; watch it in the
  NativeLink cache metrics.
- **Local fallback rate**: close to zero. Anything higher means workers
  are rejecting actions (platform mismatch, capacity).
- **Network errors**: zero. Anything else is a problem.

## Worker requirements

Chromium's Siso config takes its remote platform properties from
`build/config/siso/backend_config/backend.star` (copy `template.star`
and edit it, or point the `reapi_backend_config_path` gclient custom
var at your own file).

## Troubleshooting

- **All actions falling back to local.** Likely a platform-property
  mismatch. Compare what Chromium is requesting (in the siso
  log) against what your workers advertise.
- **Siso is not being selected.** If the output directory was already
  generated for Ninja, run `gn clean out/Default`, regenerate with
  `use_siso=true`, and build again.

## FAQ

No. Pointing Siso at NativeLink is configuration only: GN args, the
    Siso environment variables, and a local `backend.star` for the
    platform properties. The same applies to Chromium forks.

# https://docs.nativelink.com/getting-started/connect-your-build/pants

**Pants**: Enable Pants remote caching with NativeLink.

Pants can read and write remote cache entries through the Remote Execution
API. NativeLink can provide the remote store and Action Cache for Pants
goals without changing BUILD files.

## Start NativeLink

Start a NativeLink server that exposes CAS and Action Cache on
`localhost:50051` with instance name `main`. The Docker quickstart in
[Getting Started → Quickstart](/getting-started/quickstart) is enough for local
cache validation.

## Configure Pants

Add the remote cache settings to `pants.toml`:

## Run a goal

Run any Pants goal as usual:

## Remote execution

Pants remote execution uses the same Remote Execution API, but it needs a
worker environment that matches the actions Pants sends. Start with remote
caching, then add remote execution once you have worker platform properties
and toolchains aligned.

## Troubleshooting

- **No cache hits on the second run.** Check that `remote_cache_write` and
  `remote_cache_read` are both enabled.
- **Instance-name errors.** Match `remote_instance_name` to the NativeLink
  config. The local examples use `main`.
- **Connection failures.** Pants expects a `grpc://` URL in
  `remote_store_address` for a plaintext endpoint, or `grpcs://` for
  TLS.

## FAQ

To let you split roles: CI can populate the cache
    (`remote_cache_write = true`) while developer laptops only read from
    it. For a single shared setup, enable both.

# https://docs.nativelink.com/getting-started/connect-your-build/buildstream

**BuildStream**: Configure BuildStream to use NativeLink for artifact storage and remote execution.

BuildStream can use a Remote Execution API backend for artifact storage,
Action Cache, CAS, and remote execution. NativeLink can serve those
endpoints from the same `localhost:50051` listener used by the other local
examples.

## Start NativeLink

Use a config that exposes CAS, Action Cache, execution, capabilities,
ByteStream, fetch, and push services on `localhost:50051`.

## Configure BuildStream

Add a BuildStream config file that points artifact storage and remote
execution services at NativeLink:

## Configure a project

A minimal BuildStream project needs a project name, a supported BuildStream
version, and an element directory:

## Add an element

This small element stages local source files and runs a build command:

## Run a build

Run BuildStream with the config file:

## Troubleshooting

- **BuildStream cannot connect.** Match the `http://localhost:50051`
  endpoint in `buildstream.conf` to the NativeLink listener. The local test
  uses an insecure HTTP URL.
- **Uploads are missing.** Set `push: true` under `artifacts.servers` so
  BuildStream can write artifacts back to NativeLink.

## FAQ

They configure different BuildStream subsystems: artifact push/pull
    storage versus the execution, action-cache, and storage services used
    for remote builds. Pointing every URL at the same NativeLink listener
    is fine; it exposes all of those services on one port.

# https://docs.nativelink.com/getting-started/connect-your-build/cmake-recc

**CMake with recc**: Accelerate CMake builds with NativeLink as a remote cache, using recc as the bridge.

This tutorial follows the approach Reid Kleckner laid out in
[Distributed builds of LLVM with CMake, recc, and NativeLink](https://reidkleckner.dev/posts/llvm-recc-nativelink/).
Read his post for the LLVM-scale walkthrough and the reasoning behind
each piece. This page is the short, hands-on version for your own
projects.

## Prerequisites

- Docker
- CMake 3.16+
- A C/C++ compiler (clang or gcc)

## 1. Start NativeLink

The image is multi-arch (x86_64 and ARM64) as of v1.6.0, so it runs
natively on Linux and Apple Silicon alike.

## 2. Install recc

Works on Linux too if you have [Homebrew on Linux](https://docs.brew.sh/Homebrew-on-Linux).

## 3. Create the example

Drop these two files into a fresh project directory and `cd` into it.
You can also copy them from

## 4. Point recc at NativeLink

What each one does:

## 5. Build it

You should see `Hello, NativeLink!`.

## 6. Confirm the cache is live

Wipe the build directory and rebuild. This time the compile output
should come straight out of NativeLink:

## Teardown

## FAQ

Not in this setup: `RECC_CACHE_ONLY=1` means misses compile locally
    and only the outputs travel through NativeLink. Drop that variable
    once you have remote workers configured and recc will dispatch
    compiles to them.

## Going further

- Point `RECC_SERVER` at a shared NativeLink deployment to share the
  cache across your team.
- Drop `RECC_CACHE_ONLY` once you have remote workers configured to
  actually offload compile work. See
  [Remote execution](/remote-execution).
- For PCH, LTO, and other gnarly C++ build features at scale, read
  [Reid Kleckner's writeup](https://reidkleckner.dev/posts/llvm-recc-nativelink/).

# https://docs.nativelink.com/getting-started/shared-cache

**Shared cache**: Self-host NativeLink on hardware you control, with the operational checklist a team rollout actually needs.

The source-available release of NativeLink is designed to run on your
own infrastructure. This page covers what changes between the
[10-minute quickstart](/getting-started/quickstart) and a deployment that
serves your team without 3 AM pages.

## When to self-host

Pick on-prem when one of these is true:

## What ships in the box

A NativeLink deployment is composed of four roles. The same binary
serves all of them; the JSON5 config decides which subset to run.

## The rollout checklist

<li>
    **Pick the storage backend.** The quickstart's `basic_cas.json5` keeps
    everything under `/tmp` on one machine, which is fine for a 10-minute
    demo and useless for a team. Pick one before rolling out:
    - **Filesystem**: single-node clusters, sub-100GB caches.
    - **Cloud object storage**: anything multi-node.

## Container registry

Official images are published to GitHub Container Registry. Pull a
specific version tag; there is no `latest` tag to fall back on, and
pinning is what you want in production anyway.

## Backups & recovery

The CAS is the only stateful piece you can't trivially rebuild from
clients. Snapshot strategy depends on the backend:

## FAQ

Yes, for permitted use. Most of the monorepo is
    `FSL-1.1-Apache-2.0`: internal use, modification, and redistribution
    for non-competing purposes are allowed. Metrics and remote persistent
    workers are Business Source License modules: fine for an individual
    developer's cache, but shared, production, or commercial use of those
    modules needs NativeLink Enterprise or an intentionally
    inexpensive com…

# https://docs.nativelink.com/remote-execution

**Remote execution**: Workers that run the actions your cache misses.

**Who this is for:** anyone with a working cache who wants the misses to stop
running locally. **What you'll have at the end of this section:** a scheduler,
at least one worker, and a build whose cache misses execute on the farm
instead of on your machine. **Time:** an hour for the first remote action,
longer if your toolchain isn't hermetic yet.

## What this section adds

A cache answers the question "has this exact action already been run?" When
the answer is no, something still has to run it, and with caching alone,
that something is your laptop. Remote execution changes the answer to that
second question.

## What to expect the first time

The first remote action almost never works on the first try, and it's nearly
always one of two walls.

## The pages, in order

**Understand the change**

## Start here, or skip ahead

**Start here** if you have a cache serving hits and haven't run anything
remotely yet.

# https://docs.nativelink.com/remote-execution/from-cache-to-execution

**From cache to execution**: You have a cache serving hits. This is what changes when the misses stop running on your laptop: in the deployment, in the build invocation, and in what breaks first.

**Who this is for:** anyone who finished Getting started and is deciding whether to
take the next step. **What you'll have at the end:** an accurate picture of
what remote execution adds, what it costs, and which wall you'll hit first.
**Time:** ten minutes of reading, no commands.

## What execution buys you

Two things, and it's worth being clear that they're separate.

## What's new in the deployment

Three components, and one of them is a security boundary.

## What's new in the build invocation

Less than you'd think. Two flags and a declaration.

## What stays exactly the same

Your CAS and your action cache. The same stores, the same digests, the same
hits you were already getting.

## What to expect the first time

The first remote action rarely works on the first attempt, and it is nearly
always one of two walls. Both are ordinary; both have a page.

## FAQ

No. One `nativelink` process can host the CAS, the action cache, a
    scheduler, and a worker at once. That's exactly what

# https://docs.nativelink.com/remote-execution/first-remote-action

**Your first remote action**: Run a scheduler and a worker on your own machine, point Bazel at them, and watch a build action execute somewhere other than where you typed the command.

**Who this is for:** anyone with a working cache who wants to see remote
execution actually run. **What you'll have at the end:** a build whose
actions Bazel reports as `remote`, running against a scheduler and worker
you started yourself. **Time:** about fifteen minutes.

## Get the config

The repo ships a config that is exactly this tutorial: CAS, action cache,
scheduler, and one worker, all in a single process on localhost.

### Start the cluster

Or, if you'd rather use the container image, mount the config in and publish
both ports:

### Confirm the worker registered

The worker connects *outward* to the scheduler on `50061` a moment after
startup. In the server's log you should see it announce itself before you
run anything, at the default `INFO` level:

### Point a build at it

From any Bazel workspace:

### Read the result

Ask Bazel where each action ran rather than trusting the wall-clock number:

## What just happened

Four processes' worth of work happened inside one:

## When it doesn't work

Your action's platform properties don't match any registered worker, so the
scheduler is holding it in the queue. This is the single most common
first-run failure and it is silent by design: the scheduler keeps waiting
in case a suitable worker shows up later.

## FAQ

Because it's the fewest moving parts that still exercises the real code
paths: the worker here uses the same gRPC worker API, the same scheduler,
and the same store implementations a production cluster does. Splitting into
separate processes changes the deployment, not the protocol. See
[Production configuration](/operate/production-config) for the split
version.

# https://docs.nativelink.com/remote-execution/platform-properties

**Platform properties**: How the scheduler decides which worker gets an action, why a mismatch queues forever instead of failing, and how to read the matching diagnostics it already emits.

**Who this is for:** anyone whose actions are queueing, and anyone about to
run more than one kind of worker. **What you'll have at the end:** an
accurate model of the matching rules, plus the log lines that tell you which
rule you tripped. **Time:** fifteen minutes.

## The three places a property appears

A property key has to be declared in all three, and they have to agree.

## The four property types

Most documentation you'll find elsewhere describes three of these. `ignore`
exists and is the only one that exempts a worker from carrying the key.

### Why `priority` is not the escape hatch people assume

The matching loop checks key presence *before* it checks value
compatibility. For every property the action requests, the scheduler looks
the key up on the worker; if it isn't there, the worker is rejected,
regardless of type. `priority` only tells the scheduler to stop caring about
the *value*.

### `minimum` is a consumable resource, not a filter

This is the most consequential and least visible behaviour in the system.

## What happens when it doesn't match

The two failure modes are asymmetric, and knowing which one you're in tells
you where to look.

## Reading the scheduler's own diagnostics

The scheduler has a built-in periodic dump of exactly this. It's on by
default at a ten-second interval:

## A worked example: routing to a GPU pool

Say you have two worker pools and want tests tagged for GPU to land only on
the GPU machines.

## About `container-image`

It appears in nearly every example config and it is worth being precise about
what it does, because the name suggests something that isn't happening:
**NativeLink does not pull or launch a container per action.** No
code sits behind this key. It is an ordinary string property like any other.

## Troubleshooting

## FAQ

`priority` if you have one OS in the fleet, `exact` the moment you have two.
The example configs use `priority` with an empty value because a homogeneous
fleet gains nothing from enforcing it, but that's a default for a tutorial,
not a recommendation for a mixed fleet. With `exact` you get a real routing
guarantee; with `priority` you get a label.

# https://docs.nativelink.com/remote-execution/toolchains-and-hermeticity

**Toolchains and hermeticity**: Your action ran on a worker and failed on a missing compiler. Here is why that happens, and how to pin the toolchain so local and remote builds hash identically.

**Who this is for:** anyone whose actions execute remotely but fail on the
worker for reasons that never happen locally. **What you'll have at the end:**
a Nix-pinned toolchain that produces the same action digests on your machine
and on a worker. **Time:** thirty minutes for the walkthrough.

## The problem

An action is a closed description of some work: input digests, a command
line, an environment, and platform properties. If the compiler your command
invokes isn't in that description, the action is a lie: it works on your
machine because your machine happens to have the compiler, and it fails on a
worker that doesn't.

## What ships in `@local-remote-execution`

Three toolchain families, and they do not behave the same way once you leave
`x86_64-linux`.

## Set it up

<li>

### Install Nix with flakes enabled

The [next-gen installer](https://github.com/NixOS/experimental-nix-installer)
is the shortest path. If you already have Nix, make sure
`experimental-features = nix-command flakes` is in your `nix.conf`.

### Pull the template

This writes a `flake.nix` importing `nativelink.flakeModules.lre` and
pinning `lre = { inherit (pkgs.lre.lre-cc.meta) Env; }`, plus a
`hello-world.cpp` example, its `BUILD.bazel`, and `platforms/BUILD.bazel`.

### Initialise git before you enter the shell

This step is not optional and it fails quietly.

### Enter the dev shell

This fetches the Nix-pinned toolchain and writes `lre.bazelrc` (a symlink
into the Nix store). On `aarch64-darwin` the generated file looks like this,
abbreviated; your store paths will differ:

### Point `user.bazelrc` at your cluster

`nix flake init` writes a `user.bazelrc` with the literal string `TODO` as
every value. That is what lands on disk, not a placeholder left in this page:

### Build

With a real endpoint and a worker running the `nativelink-worker-lre-cc`
image, this compiles `src/hello-world.cpp` on that worker and returns the
binary through the same CAS your local Bazel reads from. Run it again and
it's a cache hit.

## Adding LRE to a project you already have

The template bootstraps a *new* workspace. For an existing one you need the
flake-parts module wiring, the `bazel_dep` override, and a `try-import` of
the generated `lre.bazelrc`.

## Troubleshooting

## FAQ

You can get *reproducibility across a fleet* by baking the toolchain into
one worker image and routing every action to it. That's path 1 at the top
of this page, and for many teams it's enough. What you don't get is matching
action digests between a developer's laptop and the fleet, because the
laptop isn't running that image. Nix pinning is what closes that gap.

# https://docs.nativelink.com/remote-execution/containers-and-images

**Containers and images**: What the container-image property actually does (and doesn't), plus how to build worker images and keep what your build requests in sync with what your fleet advertises.

**Who this is for:** anyone whose actions need a specific environment, and
anyone running more than one kind of worker image. **What you'll have at the
end:** worker images that carry your toolchain, and a routing scheme that
sends each action to the right one. **Time:** twenty minutes.

## What `container-image` is not

Start here, because the name is misleading and the misunderstanding is
expensive.

## The actual model

One image per worker *pool*. Every action a given worker runs shares that
worker's environment. Heterogeneity comes from running several pools and
routing between them, not from varying the image per action.

## Build a worker image

A NativeLink worker image is any image that contains your toolchain and can
run the `nativelink` binary with a worker config. The repo's Nix helper,
`createWorker`, turns a toolchain image into a `nativelink-worker-*` image:

## Wire up the routing

<li>

### Declare the key on the scheduler

Use `exact` once you have more than one image. The examples ship `priority`,
which matches any value: fine for a single-pool tutorial, useless as a
routing mechanism.

### Advertise it on each worker

Pin a tag you can change deliberately. `:latest` makes the advertised string
stable while the actual contents drift, which defeats the entire point: the
scheduler will keep matching, and your builds will silently change behaviour.

### Request it from the build

Globally, if your whole build wants one environment:

## Rolling an image without stranding actions

Order matters, because the two sides are matched on a literal string.

## Troubleshooting

## FAQ

Not meaningfully. A worker process runs in one environment; the property is
an assertion about that environment. You can advertise several values for a
`priority` key, but that's a routing label, not a second toolchain. Run one
pool per image.

# https://docs.nativelink.com/remote-execution/persistent-workers

**Persistent workers**: Keep a JVM or Node compiler process warm across actions instead of paying its startup cost every time, and understand exactly what NativeLink gives up to do it.

**Who this is for:** anyone whose remote builds are dominated by JVM or
TypeScript compile actions. **What you'll have at the end:** compile actions
reusing a warm process, and a clear-eyed view of the isolation you trade for it.
**Time:** an hour, most of it in your build rules rather than in NativeLink.

## Two things to know before you start

**Little needs configuring in NativeLink.** The pool is on by default and
the protocol is chosen by two platform properties on each action. The
worker's `persistent_workers` block (v1.7.3 and later) sizes it: how many
processes per key, how long an idle one lives, how many requests one serves,
how long an action at the cap waits for a process to come back, and
`enabled: false` to run every such action one-shot.

## Opt an action in

The worker looks at two of the action's platform properties (the REAPI
`Platform` on the `Action`): `supports-workers` must be exactly `1`, and
`requires-worker-protocol` picks the wire format. Nothing on the worker's own
`platform_properties` is involved.

## What makes two actions share a process

Actions are pooled by a key derived from the command line:

## The isolation you give up

This is the part to read twice. A persistent worker process outlives the
actions it serves, so some of the isolation an ordinary action gets cannot
apply to it, and some of it did not apply until v1.7.3.

## The pool's actual behaviour

The pool is configured under `persistent_workers` on the worker; every value
below is its default.

## Steps

<li>

## When it doesn't work

Either the properties are not reaching the action's `Platform`, or every action
is getting its own key. Check `supports-workers` is exactly `"1"` and is sent as
a platform property (see the callout above), and that the scheduler declares
both keys; an undeclared key leaves the action queued forever.

# https://docs.nativelink.com/remote-execution/local-testing

**Local testing**: Keep a NativeLink cluster on your own machine to test config changes, reproduce scheduler behaviour, and prove a client works, in the two shapes the repo itself uses.

**Who this is for:** anyone changing a config, debugging a client, or
reproducing something a shared cluster did. **What you'll have at the end:**
a local cluster in whichever of the two shapes matches what you're testing,
and a reliable way to tell what actually ran where. **Time:** ten minutes
for the single-process shape.

## Which shape to run

## Shape 1: one process

`--remote_cache` and `--remote_executor` both point at `127.0.0.1:50051`.
`50061` exists only so the worker can register with the scheduler; Bazel
never talks to it.

## Shape 2: the compose split

When you need separate processes (a CAS process and a scheduler process
talking over `grpc` stores, which is what a real deployment looks like),
the repo ships that too, and its own integration tests run against it.

## Verify what actually ran

Wall-clock time is a bad signal and Bazel's summary line is only a rough
one. Ask the build event log directly:

## Troubleshooting

At least one worker's `platform_properties` must satisfy what your actions
request, and every requested key must be declared in the scheduler's
`supported_platform_properties`. Starting with just `cpu_count: "minimum"`
on the scheduler and `cpu_count: { values: ["1"] }` on the worker unblocks
most local setups.

## FAQ

Yes: the config exposes standard Remote Execution API services, so Buck2,
Pants, Siso, and recc point at `127.0.0.1:50051` the same way. The Bazel
flags are just the worked example;
[Connect your build](/getting-started/connect-your-build) has the others.

# https://docs.nativelink.com/remote-execution/examples-and-templates

**Examples and templates**: Runnable configs, deployment topologies, and project scaffolds you can copy, plus the three client-side patterns most teams actually adopt.

**Who this is for:** anyone who wants a working starting point rather than a
blank file. **What you'll have at the end:** the right example for your
situation, and a client-side pattern that matches how you want work split
between laptops and a fleet. **Time:** varies; the point of this page is to
find the thing, not to read it.

## Server configs

is the largest and most useful collection. Every file is a complete,
loadable config.

## Deployment topologies

holds deployment material rather than single config files: the
`docker-compose/` stack, a RHEL 8 `Dockerfile` (`rhel/`), the metrics
pipeline configs and dashboards (`metrics/`), and the persistent-worker rule
sketches (`persistent-workers/`).

## Project templates

The flake ships templates for the client side: a project already wired to a
Nix-pinned toolchain.

## Three client-side patterns

Independently of which config you start from, there are three ways teams
wire their builds. Most adopt them in this order.

### Pattern A: Cache only

Every action runs locally; results land in the shared cache. No worker fleet
needed, which makes it the cheapest thing that helps.

### Pattern B: Cache plus execution

Actions ship to a worker fleet; the cache underneath stores everything.

### Pattern C: Hybrid

Small actions local, everything else remote, decided by policy.

### Choosing

The speedup numbers you'll see quoted for these vary by an order of
magnitude between workloads, and mostly reflect how much of a given build is
cache-missing compilation. Measure your own before committing to a shape.

## FAQ

`s3_backend_with_local_fast_cas.json5` for the store layer and the
`docker-compose/` topology for the process split; that combination is the
shape [Production configuration](/operate/production-config) describes at
full size. Don't start from `local_rbe_self_test.json5`; it's optimised for
being readable in one screen, not for running a fleet.

# https://docs.nativelink.com/configuration

**Configuration**: The config model on one page, so you can read and write a NativeLink config instead of copying one.

**Who this is for:** anyone running a cluster they got working by copying an
example, who now needs to change it. **What you'll have at the end of this
section:** the ability to read any NativeLink config and write one from an
empty file. **Time:** an afternoon, most of it on the capstone.

## One file, six keys

A NativeLink deployment of any shape (one process on a laptop, a sharded
fleet across three regions) is described by a single JSON5 file. It has
exactly six top-level keys, and only two of them are required.

## How a request traverses them

Read that diagram as: **the `servers` array is the only entry point, and
everything else is reached from it by name.** A service doesn't contain a
store, it names one. A worker doesn't contain a scheduler, it dials one over
the worker API. The names are the wiring.

## Everything is a named array

Every top-level collection is an **array of objects**, not a map keyed by
name. Stores and schedulers carry their `name` beside the type key; workers
carry it inside the `local` block; servers have an optional `name`:

## A typo is a startup error, not a silent default

Nearly every struct in the config crate is `#[serde(deny_unknown_fields)]`.
Misspell a key and the process refuses to start and tells you which key it
didn't recognise, rather than ignoring it and behaving strangely three hours
later.

## The pages, in order

**Learn the mechanics**

## Start here, or skip ahead

**Start here** if you have a cluster running from a copied example and want to
understand what you copied.

# https://docs.nativelink.com/configuration/config-file

**The config file**: JSON5 mechanics, environment substitution, how the binary is pointed at a config, and how to find out a config is wrong before it takes a cluster down with it.

**Who this is for:** anyone about to edit a NativeLink config for the first
time. **What you'll know at the end:** what dialect the file is written in,
how to keep secrets and addresses out of it, how the binary finds it, and how
to check one is valid without waiting for a production rollout to tell you.
**Time:** twenty minutes.

## It's JSON5, not JSON

NativeLink parses its config with `serde_json5`, not `serde_json`. The
difference matters every time you edit a file:

## Environment substitution

Any string field that is deserialized through the config crate's shellexpand
helpers accepts shell-style variable references, including the
default-if-unset form:

## How the binary finds the config

One way only, and it is positional:

## Checking a config before you trust it

Here is the honest answer, because the alternative wastes an afternoon:
**there is no `--validate` flag and no dry-run mode.** The binary takes a
config path and starts.

## Editor validation, if you want it

The config crate can emit a JSON Schema for the whole `CasConfig` type, which
editors will use for completion and inline errors:

## FAQ

Not with an include directive; the parser reads exactly one file. What you do
instead is split by *role*: one config for the CAS process, one for the
scheduler, one for the workers, each a complete file naming the others by
network address. That's the same split described in
[the section overview](/configuration), and it's how the docker-compose
deployment is laid out.

# https://docs.nativelink.com/configuration/stores

**Stores**: Declaring stores, naming them, and the idea that unlocks the rest of the config: stores compose, and one store refers to another by name.

**Who this is for:** anyone who has seen a NativeLink config with a store
nested five levels deep and wanted to know why. **What you'll know at the
end:** how to declare a store, how to name one, how composition works, and
which layers are worth reaching for. **Time:** thirty minutes.

## A store is one entry in an array

The `stores` array holds every storage backend the process can use. Each entry
is an object with a `name` and exactly one type key:

## Backing it with disk

Swap the type key and the same two stores survive a restart:

## Stores compose

Here is the idea the rest of this page is about. **Most store types don't
store anything.** They wrap another store and change its behaviour, and the
store they wrap is written inline, as a full store spec:

## `ref_store`: naming instead of nesting

Nesting alone can't express *sharing*. If two stores should write to the same
place, inlining the backend twice gives you two independent stores that
happen to have identical settings.

## The layers worth knowing

You don't need all eighteen store types to write a good config. These are the
ones that appear in nearly every non-trivial one:

## What keys the store

A store is content-addressed, so what goes in a key is the digest, and every
server, worker, and build client sharing a cache must compute that digest the
same way. SHA-256 and BLAKE3 produce different keys for identical bytes, so
uploads made under one function are cache *misses* for clients using the
other.

### Which function to pick

We compared both algorithms on an Apple M2 Max with 12 CPU cores and 32 GB of
memory, running macOS 15.6 and Bazel 9.1.1. Each of two alternating runs built
`//:nativelink` against a fresh local NativeLink filesystem cache with hash and
size verification enabled. Uploads were synchronous, and the Bazel output and
NativeLink cache directories were isolated by algorithm.

## FAQ

`eviction_policy.max_bytes` caps it, and once full the least-recently-used
entries are evicted to make room. An evicted CAS entry is just a future cache
miss; the client rebuilds and re-uploads. An evicted entry that a cached
action result still points at is the problem `completeness_checking` exists to
solve.

# https://docs.nativelink.com/configuration/servers-and-services

**Servers and services**: Listeners, the eleven services, instance names, and the port split that keeps the worker API off your public interface.

**Who this is for:** anyone deciding what a NativeLink process should expose,
and on which port. **What you'll know at the end:** what a listener is, what
each of the eleven services does, how `instance_name` partitions a cache, and
why the worker API must not share a port with anything else. **Time:** thirty
minutes.

## A server is a listener plus a set of services

The `servers` array is the only entry point into a NativeLink process.
Each entry binds one socket and decides what is reachable on it:

## The eleven services

A cache-only deployment needs the first four. Adding execution adds
`execution` and `worker_api`. The rest are opt-in.

## Instance names

Most services are declared as an **array**, and each element carries an
`instance_name`. That field is a routing key: it selects which store (or
scheduler) serves a request. The instance name is not part of the storage
key, so two instance names pointing at *different* stores are fully isolated,
and two pointing at the *same* store share every blob.

## The port split

The `worker_api` service is different from every other service on this page,
and the difference is a security boundary rather than a design preference.

## TLS, when you need it

TLS is a property of the listener:

## FAQ

Yes; the array is unbounded, and separating concerns by port is the intended
use. `deployment-examples/docker-compose/local-storage-cas.json5` serves the
same cache twice, plain on `:50051` and TLS on `:50071`. A third listener for
build events, or a plain-HTTP health endpoint a load balancer can probe
without TLS, follows the same pattern.

# https://docs.nativelink.com/configuration/scheduler-and-workers

**Schedulers and workers**: The minimum config for each, the properties contract between them, and the fields that decide what happens when something goes wrong.

**Who this is for:** anyone adding execution to a config that currently only
caches. **What you'll know at the end:** the minimum scheduler and worker
blocks, how the two are wired to each other, and which of the many optional
fields actually change behaviour you'll notice. **Time:** thirty minutes.

## The minimum scheduler

A scheduler is one entry in the `schedulers` array, with a name and a type:

## The minimum worker

A worker is one entry in the `workers` array. `local` is currently the only
type; it means the worker runs actions in this process, on this machine:

## How they find each other

Nothing in the scheduler names a worker. Workers are not declared to the
scheduler at all; they connect to it, announce their platform properties, and
are added to the pool. Removing a worker means stopping it.

## The fields that change what you'll notice

`SimpleSpec` has ten optional fields. Most can stay at their defaults
forever. These four are the ones worth setting deliberately:

## Putting the pieces together

Adding execution to a cache-only config takes four edits, all of which you've
now seen:

## FAQ

Yes, and that's what `nativelink-config/examples/basic_cas.json5` is: one
process with stores, a scheduler, a worker, a public server on `:50051` and a
private one on `:50061`. It's the right shape for a laptop and for testing.
Splitting into separate processes is a deployment change, not a config
rewrite; the same blocks move into separate files.

# https://docs.nativelink.com/configuration/your-first-full-config

**Your first full config**: An empty file to a running cache-and-execution cluster, one block at a time, checking after each.

**Who this is for:** anyone who has read the rest of this section and wants to
prove it by writing a config from nothing. **What you'll have at the end:** a
single-process NativeLink serving a cache and executing actions, from a file
you wrote yourself and can explain line by line. **Time:** about an hour.

## Before you start

You need the `nativelink` binary and a writable directory. This walkthrough
uses `/tmp/nativelink`; if you use somewhere else, keep the **content path and
the work directory on the same filesystem**, because the worker hard-links
between them.

## Step 1: Two stores and a listener

Start a file called `my-config.json5`. The smallest useful NativeLink is a
cache: two stores and one server.

## Step 2: Point a build at it

Nothing about the config changes here, but proving the cache works before
adding execution keeps the next failure unambiguous.

## Step 3: Add a scheduler

Execution needs a queue. Add a `schedulers` array beside `stores`:

## Step 4: Expose execution, on two ports

Now wire the scheduler in. Add `execution` to the public server, give
`capabilities` a `remote_execution` block, and add a **second server** for the
worker API:

## Step 5: Add a worker

A worker needs a `fast_slow` store with a filesystem fast tier. Replace
`CAS_MAIN_STORE` with one:

## Step 6: Execute something

Point the build at execution rather than just the cache:

## The finished file

Compare it with `nativelink-config/examples/basic_cas.json5` in the source
tree. The differences are instructive: the shipped example registers every
service twice, under `instance_name: ""` and `"main"`, so clients that omit
the instance name also work, and it declares nineteen platform properties
rather than three.

## What this is not yet

This config is correct and it is not production. Four things are missing, and
each has a page:

# https://docs.nativelink.com/how-to

**How-to guides**: Pick the task you're doing, not the artifact you're reading about.

**Who this is for:** anyone with a running NativeLink who now needs it to do
one specific thing differently. **What you'll have at the end:** the one page
that does your task, rather than a tour. **Time:** as long as the task takes.

## How this section works

Every page here is a task, phrased as the thing you are trying to do. None of
them is a tour of a subsystem (that's what
[Concepts](/explanations/architecture) is for), and none of them is an exhaustive
field list, which is what [Reference](/reference/glossary) is for.

## Pick your task

**I need my cache stored somewhere else.**
[Storage backends](/how-to/stores) compares every place NativeLink can put
bytes ([filesystem](/how-to/stores/filesystem),
[S3 and compatibles](/how-to/stores/s3-and-compatible),
[GCS](/how-to/stores/gcs), [Azure Blob](/how-to/stores/azure),
[OCI](/how-to/stores/oci-object-storage), [Redis](/how-to/stores/redis) and
[Mongo](/how-to/stores/mongo)) on durability, latency and c…

# https://docs.nativelink.com/how-to/stores

**Storage backends**: Every place NativeLink can put bytes, compared on durability, latency and cost, and the composition almost every real deployment lands on.

**Who this is for:** anyone whose cache currently lives somewhere they can't
keep it. **What you'll have at the end:** the one backend page that matches
your durability and latency requirements, and the shape you'll wrap it in.
**Time:** ten minutes to choose, then as long as the backend page takes.

## Everything here is a store entry

Nothing on this page is a new concept. A backend is one entry in the `stores`
array with a name and a type, exactly as described in
[Stores](/configuration/stores):

## Pick a backend

`memory`, `grpc` and `noop` have no page of their own because there is nothing
to decide: `memory` takes an eviction policy, `grpc` takes an upstream address,
and `noop` takes nothing at all. All three are covered in
[Compose stores](/how-to/stores/compose-stores).

## The shape almost everyone lands on

Very few production deployments name a cloud bucket as their CAS store
directly. The read pattern of a build cache (many small objects, requested
repeatedly, in bursts) is exactly what object storage prices and latency are
worst at. The shape that works is a local fast tier in front of a durable slow
tier:

## Shrinking what you move

Two pages in this group are not backends at all. They change how many bytes
travel between the client and NativeLink, whichever backend is underneath:

# https://docs.nativelink.com/how-to/stores/filesystem

**Filesystem store**: Put the cache on local disk: the two paths that must share a block device, the eviction policy you cannot skip, and the tuning knobs that matter on real hardware.

**Who this is for:** anyone who wants the cache to survive a restart without
paying for object storage, and anyone building the fast tier of a larger
composition. **What you'll have at the end:** a `filesystem` store sized to
your disk, with eviction that actually runs. **Time:** fifteen minutes.

## The recipe

Create the parent directory and start NativeLink. The store scans
`content_path` on boot and adopts every file whose name parses as a store key
and generation (`<hash>-<size>-<generation>` under `d2/`, or
`<key>-<generation>` under `s2/`); a file whose
name does not parse is deleted. The scan reads metadata, not content, so it does not check
that bytes still match their digest.

## The two paths

`content_path` holds the data. `temp_path` holds objects mid-upload and
mid-delete, while their contents cannot yet be trusted.

## The eviction policy you cannot skip

`eviction_policy` is optional in the schema and mandatory in practice. Without
it, nothing is ever removed and the store grows until the disk is full.

## Tuning that matters on real hardware

**`max_concurrent_writes`** (default `0`, unlimited). Every write streams into
a temp file and calls `sync_all()`. Enough concurrent writes will saturate disk
I/O and start blocking the async runtime. On a busy shared CAS, set this to
something bounded (a few dozen) rather than leaving it open.

## If this store feeds a worker

A worker's `cas_fast_slow_store` must have a `filesystem` store as its `fast`
half. The worker builds each action's input tree by hard-linking files out of
that store, and hard links require real files on a real filesystem; a memory
or object store cannot supply them.

## Steps

<li>

## When it doesn't work

An upload that encounters a missing file during duplicate comparison removes
the stale index entry and stores the uploaded content. A read removes only the
entry whose file was missing. In a `fast_slow` composition, that read can fall
through to the slow store and refill the fast tier.

# https://docs.nativelink.com/how-to/stores/s3-and-compatible

**S3 and compatible**: Back the cache with Amazon S3, Cloudflare R2 or NetApp ONTAP S3: one store type, three providers, and the shared options that decide cost and failure behaviour.

**Who this is for:** anyone who wants durability they don't operate, on S3 or
something that speaks S3. **What you'll have at the end:** a bucket-backed CAS
behind a local fast tier, with credentials that aren't in the config file.
**Time:** thirty minutes, most of it bucket and IAM setup.

## Amazon S3

`region` and `bucket` are the only fields you have to set. Credentials come
from the standard AWS chain (environment variables, the shared credentials
file, the EC2/ECS/EKS instance or task role), so nothing secret goes in the
config.

### Other services that speak S3

The `aws` provider reaches any S3-compatible service, not only AWS. Four
optional fields cover the difference:

## Cloudflare R2

R2 has no regions from the client's point of view: the endpoint is derived
entirely from your account ID.

## NetApp ONTAP S3

On-premises S3 needs an explicit endpoint, the storage VM serving it, and
usually a private CA bundle:

## The options all three share

Every provider flattens the same block of common options. These are the ones
worth setting deliberately:

## Steps

<li>

## When it doesn't work

For `aws`, the credential chain found nothing. Confirm the process can see
`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`, or that the instance/task
role is attached and reachable. For `r2`, set `access_key_id` and
`secret_access_key` explicitly, because R2 tokens are not in the AWS chain.

# https://docs.nativelink.com/how-to/stores/gcs

**Google Cloud Storage**: Back the cache with a GCS bucket: authentication, the resumable-upload chunk size, and the timeout fields whose names disagree with their units.

**Who this is for:** anyone running NativeLink on Google Cloud who wants the
durable tier to be a GCS bucket. **What you'll have at the end:** a
bucket-backed CAS behind a local fast tier, authenticated by the ambient
service account. **Time:** twenty minutes, most of it IAM.

## The recipe

`bucket` is the only field you have to set. No region is needed: a GCS bucket's
location is a property of the bucket, decided when you create it, and the
client reaches it the same way wherever it lives. Put the bucket in the region
your workers run in; cross-region reads are both slower and billed.

## Authentication

Credentials come from the ambient Google credential chain: a credentials file
named by `GOOGLE_APPLICATION_CREDENTIALS` (or inline JSON in
`GOOGLE_APPLICATION_CREDENTIALS_JSON`), the application-default credentials
written by `gcloud auth application-default login`, or otherwise the metadata
server on GCE, GKE or Cloud Run. Nothing secret goes in the config file.

## Options worth setting

Objects under 5 MB with a known size go up in one request; everything else
uses a resumable upload, sent chunk by chunk with `resumable_chunk_size`
bytes per request. The value is rounded to a multiple of 256 KiB and capped at
the 2 MB default, so it can only be lowered, which trades more round trips per
object for less memory held per in-flight upload. No setting
raises it.

## Steps

<li>

## When it doesn't work

The credential chain found something, but it can't act on the bucket. Confirm
the service account attached to the instance or workload identity has object
read and write on it. If it can create but not overwrite, re-uploads of a
digest that already exists fail, because GCS needs the delete permission to
replace an object.

# https://docs.nativelink.com/how-to/stores/azure

**Azure Blob Storage**: Back the cache with an Azure Blob container: account and container naming, the endpoint override for Azurite, and when a SAS URL replaces everything else.

**Who this is for:** anyone running NativeLink on Azure who wants the durable
tier to be a Blob Storage container. **What you'll have at the end:** a
container-backed CAS behind a local fast tier. **Time:** twenty minutes, most
of it access configuration.

## The recipe

`account_name` and `container` are the pair you have to set. From them the store
builds the endpoint `https://{account_name}.blob.core.windows.net/{container}`.
The container is a blob container inside the storage account. Create it before
starting NativeLink; the store does not create it for you.

## Three ways to point at a container

**Account plus container.** The default above. The endpoint is derived, and
the store authenticates with Entra ID Workload Identity and nothing else: it
reads `AZURE_TENANT_ID`, `AZURE_CLIENT_ID` and `AZURE_FEDERATED_TOKEN_FILE`
(the variables AKS workload identity injects into a pod) and fails at startup
if they are missing. A VM managed identity, a storage account key or an
`az login` session are not tried.

## Options worth setting

Blobs under 5 MiB with a known size are uploaded in one request; anything
larger, or of unknown size, is staged as blocks of at least 5 MiB and
committed with a block list. `retry` and `consider_expired_after_s` behave as
described in [the shared options section](/how-to/stores/s3-and-compatible);
`insecure_allow_http` and `disable_http2` are accepted but not read by the
Azure store.

## Steps

<li>

## When it doesn't work

Either the workload identity lacks Storage Blob Data Contributor on the
container, or a `sas_url` has expired. A SAS expiry surfaces as an
authentication failure, so check the signature's window before assuming the
role assignment is wrong. If the store did not start at all with a message
about the Workload Identity credential, the `AZURE_*` variables are missing.

# https://docs.nativelink.com/how-to/stores/redis

**Redis**: Back the cache with Redis (standalone, sentinel or cluster) and the concurrency, chunking and timeout settings that keep it from timing out under a real build.

**Who this is for:** anyone who wants several NativeLink processes to share one
low-latency store, or who needs the scheduler's pub/sub subscriptions.
**What you'll have at the end:** a Redis-backed store with concurrency and
timeouts set so it survives a burst. **Time:** twenty minutes.

## The recipe

`addresses` is the only required field. The URL carries everything Redis needs:
scheme, optional credentials, host, port and database index.
`redis://username:password@redis-server-url:6380/99` selects port 6380 and
database 99. Keep credentials out of the file by writing
`redis://${REDIS_USER}:${REDIS_PASSWORD}@redis:6379/` and exporting the
variables; see [The config file](/configuration/config-file).

## Standalone, sentinel or cluster

`mode` takes `standard` (the default), `sentinel` or `cluster`.

## Sizing it so it doesn't time out

Redis is single-threaded per node, so the settings that matter are the ones
bounding how much work you can have in flight.

## Where Redis actually earns its place

Rather than as a bulk CAS, Redis is usually the fast or index half of a
composition:

## Steps

<li>

## When it doesn't work

Too much in flight. Lower `max_client_permits` so work queues in NativeLink
instead of piling onto a single-threaded Redis. Only after that, consider
raising `command_timeout_ms`.

# https://docs.nativelink.com/how-to/stores/mongo

**MongoDB**: Back the cache with MongoDB: collections, write concern, timeouts, and what the change-streams flag is for.

**Who this is for:** anyone who already operates MongoDB. **What you'll have
at the end:** a Mongo-backed CAS behind a local fast tier. **Time:**
twenty-five minutes.

## The recipe

`connection_string` is the only required field and takes either form Mongo
accepts, `mongodb://localhost:27017` or
`mongodb+srv://cluster.mongodb.net` for Atlas. It carries the credentials, so
keep it in the environment and reference it with `${VAR}`; see
[The config file](/configuration/config-file).

## Change streams and the scheduler

The Mongo store implements the scheduler-store interface, and
`enable_change_streams: true` is what that interface needs: the store refuses
to hand out scheduler subscriptions without it. In v1.6.5, though, the `simple`
scheduler's `experimental_backend` accepts only `memory` and `redis`, so there
is no config path that puts a Mongo store behind the scheduler.

## Options worth setting

Write concern is exposed as three separate fields mirroring Mongo's own:
`write_concern_w` (a number like `1`, or the string `"majority"`),
`write_concern_j` (whether to wait for the journal), and
`write_concern_timeout_ms`. All three are optional and unset means the
deployment's own default applies, but `write_concern_j` or
`write_concern_timeout_ms` without `write_concern_w` is rejected at startup.

## Steps

<li>

## When it doesn't work

Not in v1.6.5. The store implements the scheduler-store interface behind
`enable_change_streams`, but the `simple` scheduler's `experimental_backend`
only accepts `memory` and `redis`, so there is nothing to point at it. Use
[Redis](/how-to/stores/redis) for a shared scheduler.

# https://docs.nativelink.com/how-to/stores/oci-object-storage

**Oracle Cloud (OCI) Object Storage**: Back the cache with OCI Object Storage through its S3 Compatibility API: Customer Secret Keys, the two adjustments the store makes for you, and what has actually been verified.

**Who this is for:** anyone running NativeLink on Oracle Cloud who wants the
durable tier to be an OCI bucket. **What you'll have at the end:** a
bucket-backed CAS and Action Cache authenticated with a Customer Secret Key.
**Time:** thirty minutes, most of it namespace and policy setup.

## How it works

[OCI Object Storage](https://www.oracle.com/cloud/storage/object-storage/)
exposes an [S3 Compatibility
API](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm),
and the `oci` provider is a thin adapter pointing the S3 store at it. Two
OCI-specific adjustments are applied automatically, and you do not configure
either:

## Before you start

You need three things.

## Authentication

The S3 Compatibility API authenticates with Customer Secret Keys, a static
access-key and secret-key pair used as `access_key_id` and
`secret_access_key`, signed with AWS SigV4.

## The recipe

A minimal CAS and Action Cache sharing one bucket, segmented by `key_prefix`:

### Fields

OCI also accepts `us-east-1` as a `region` value to target the tenancy home
region, which is occasionally useful when a tool insists on an AWS region name.

## Steps

<li>

## Operational notes

**Reap incomplete multipart uploads.** When a multipart upload fails mid-flight
the store issues `AbortMultipartUpload` to clean up, and OCI honors both
`ListMultipartUploads` and `AbortMultipartUpload`. A process killed *before* it
can abort still leaves orphaned parts, which is what the lifecycle rule is for.

## When it doesn't work

Something is re-adding the default trailing checksum the store removes. This is
the failure the checksum downgrade exists to prevent, and it shows up first on
Action Cache entries because they are small single-`PUT` objects. Confirm you
are on a NativeLink version that carries the `oci` provider rather than
pointing the `aws` provider at an OCI endpoint by hand.

# https://docs.nativelink.com/how-to/stores/compose-stores

**Compose stores**: The wrapper vocabulary (fast_slow, verify, dedup, existence_cache, size_partitioning, shard and the rest) and the order the layers have to go in.

**Who this is for:** anyone who has picked a backend and now needs it to be
fast, cheap and trustworthy. **What you'll have at the end:** a layered store
you can explain line by line, and the vocabulary to read anyone else's config.
**Time:** thirty minutes.

## The two ways stores refer to each other

A wrapper can hold its inner store inline:

## The wrappers

### `fast_slow`: a tier in front of a tier

The single most important wrapper. Reads try `fast` first and fall through to
`slow` on a miss, populating `fast` as they go. Writes go to both.

### `verify`: refuse to store corruption

Checks the size and hash of an upload against the digest it is filed under,
and rejects the write on a mismatch. Reads pass straight through to the
backend unchecked, so this protects what enters the store, not what a backend
later hands back.

### `completeness_checking`: the Action Cache wrapper

The AC's counterpart. Before returning an action result as a hit, it confirms
every output digest that result references still exists in the CAS. Without it,
a client gets a cache hit pointing at blobs that have since been evicted.

### `existence_cache`: stop asking the same question

Caches the answer to "does this digest exist" so `FindMissingBlobs` doesn't hit
the backend for every blob of every build.

### `compression`: smaller on the backend

Compresses blobs with LZ4 before they reach the wrapped store.

### `dedup`: store the changed parts only

Splits blobs into content-defined chunks, stores each chunk once, and keeps a
per-blob index of which chunks it is made of.

### `size_partitioning`: route by blob size

Sends everything under `size` to one store and everything at or above it to
another.

### `shard`: spread across several backends

Distributes blobs across stores by digest, weighted.

### `cache_metrics`: measure one layer

Wraps a store to record hit rates and timings under a label:

### The three with nothing to configure

`memory` takes an `eviction_policy` and nothing else. It is the right fast half
for a short-lived process and the wrong one for anything that must survive a
restart.

## The order the layers go in

Reading a real config outside-in, each layer answers a different question:

## A full example

The shipped GCS and Azure examples both use this shape, and it is a good
default for a cloud-backed CAS:

## Steps

<li>

## When it doesn't work

Almost every store spec is strict about unknown fields, so this is usually a
field placed at the wrong nesting level: `eviction_policy` on the wrapper
rather than on the store, or a backend option one layer too high. Check what
the innermost brace you're inside actually is.

# https://docs.nativelink.com/how-to/stores/compression

**Remote cache compression**: Cut remote cache transfer bytes for compressible artifacts with REAPI zstd wire compression and Bazel's --remote_cache_compression.

**Who this is for:** anyone whose clients reach NativeLink across a real
network and whose artifacts compress well. **What you'll have at the end:**
zstd on the wire, with stores still holding raw bytes. **Time:** ten minutes.

## Requirements

- A Bazel client with `--remote_cache_compression` (available since Bazel 5).
- Wire compression is **optional and off until it is enabled somewhere in the
  NativeLink process**. Without the configuration below or an explicitly
  enabled CAS `grpc` store, NativeLink does not advertise zstd, rejects
  `compressed-blobs/zstd` requests with `InvalidArgument`, and keeps its own
  outgoing transfers on the identity path,…

## Enabling it

Set `remote_cache_compression: true` on the capabilities service for the
instance. The capabilities service advertises zstd to clients, and the
ByteStream and CAS services accept and serve compressed payloads for that
instance:

### NativeLink-to-NativeLink transfers

A `grpc` store with `store_type: "cas"` can compress its own uploads and
full-blob downloads to the upstream with `experimental_remote_cache_compression`.
Blobs of 64 KiB and above go as `compressed-blobs/zstd`; smaller blobs and
ranged reads stay on the identity path. The upstream instance must have
`remote_cache_compression` on its capabilities service, or compressed
requests fail with `InvalidArgument`.

## Semantics worth knowing

- Resource names and digests always refer to the **uncompressed** blob; only
  the bytes on the wire are compressed. ByteStream compressed writes report
  `committed_size` in compressed wire bytes, as the REAPI extension specifies.

## When it helps, and when it doesn't

Compression pays off when clients reach the cache across a real network and the
artifacts are compressible: uncompressed archives, binaries with debug info,
and text-heavy outputs commonly shrink 2-10x on the wire. It does little on
same-rack links (the zstd CPU cost buys nothing when the wire isn't the
bottleneck) and nothing for already-compressed artifacts.

## Steps

<li>

## When it doesn't work

NativeLink only advertises zstd when `remote_cache_compression` is true on the
capabilities service **for that instance name**. An instance-name mismatch
between the capabilities entry and the CAS or ByteStream entries is the usual
cause.

# https://docs.nativelink.com/how-to/stores/chunking-and-dedup

**Content-defined chunking**: Cut remote cache transfer bytes by 80-90% for incrementally changing artifacts with the REAPI SplitBlob/SpliceBlob extension and Bazel's --experimental_remote_cache_chunking.

**Who this is for:** anyone re-transferring large artifacts that change a
little at a time: container layers, linked binaries, archives. **What you'll
have at the end:** clients uploading and downloading only the chunks that
changed. **Time:** twenty minutes.

## Requirements

- **Bazel 9.1.1+ or 8.7.0+** on the client, with
  `--experimental_remote_cache_chunking`. Avoid 9.1.0: it has a client bug that
  corrupts outputs when the chunking flag is combined with `--disk_cache`
  (fixed in 9.1.1).
- Chunking is **optional and off by default**.

## Enabling it

Add an `experimental_chunking` block to the CAS service and give it a small
store for chunk layouts. The index store must not verify content digests and
must not be the CAS store itself (naming the same store for both is rejected
at startup):

## When it helps, and when it doesn't

Chunking pays off when clients reach the cache across a real network (WAN,
metered links, cross-region) and artifacts change incrementally: uncompressed
archives, linked binaries, and container layers typically save 80 to 90% of
transfer bytes per change.

## Steps

<li>

## When it doesn't work

Bazel 9.1.0 has a client bug that corrupts outputs when the chunking flag is
combined with `--disk_cache`. Use 9.1.1+ or 8.7.0+.

# https://docs.nativelink.com/how-to/run-multiple-workers

**Run multiple workers**: Add workers to a running NativeLink: how they share the CAS, how the scheduler matches them, and the two things about the shipped compose example that will mislead you.

**Who this is for:** anyone with remote execution working against one worker
who now wants several. **What you'll have at the end:** three workers pulling
from one queue, and a clear idea of how to make it thirty. **Time:** forty
minutes.

## The topology

Three roles, three kinds of process, on one network:

## What a worker config looks like

`servers: []` is not an oversight. A worker listens on nothing; it dials out to
the scheduler's worker API and to the CAS. That is why the worker tier needs no
ingress, no certificates and no load balancer, and why it is safe to run many
of them on machines that are otherwise unreachable.

## How the scheduler picks a worker

The scheduler will only send an action to a worker whose advertised
`platform_properties` satisfy the action's platform requirements, and it will
only consider property keys listed in its own
`supported_platform_properties`:

## Scaling the pool

Workers are stateless, so growing the pool is adding processes. Three things
scale with them and are worth watching before you double the count.

## Steps

<li>

## The shipped example, and where its README misleads

`deployment-examples/docker-compose/` contains a working three-worker
deployment: `docker-compose-multi-worker.yml` with
`cas-server-multi-worker.json5`, `scheduler-multi-worker.json5` and
`worker-shared-cas.json5`. Start there. Two things in and around it will send
you the wrong way.

## When it doesn't work

Almost always the platform properties. Compare the keys the worker advertises
against `supported_platform_properties` on the scheduler and against what the
client requests: a key the scheduler doesn't list can't be matched on, and a
key the worker doesn't advertise excludes it from anything requiring that key.
Check the scheduler log for the queue depth: actions queuing while workers
idle is the signature.

# https://docs.nativelink.com/how-to/tls-and-auth

**TLS and authentication**: Put TLS on the listeners clients reach, use mutual TLS as the access control, and understand exactly what NativeLink does and does not authenticate.

**Who this is for:** anyone about to expose NativeLink beyond a single trusted
network. **What you'll have at the end:** encrypted listeners, client
certificates as the gate, and an accurate picture of what is still unguarded.
**Time:** an hour, most of it spent on certificates.

## The three listeners have different trust levels

A production deployment exposes three surfaces, and they do not want the same
treatment.

## Turn on TLS

Every entry in `servers` may carry a `tls` block on its `listener.http`:

## Make the client prove who it is

Adding `client_ca_file` changes the listener from encrypted to gated:

## Point clients at the new scheme

Bazel needs the CA that signed the server certificate, and its own certificate
if you turned on mutual TLS:

## What `experimental_identity_header` actually does

You will find this on `ServerConfig`, and its name invites a wrong conclusion:

## Credentials to an upstream service

When NativeLink proxies to something that does want a token (a hosted REAPI
cache, a gateway in front of one), a `grpc` store can attach headers:

## Steps

<li>

## When it doesn't work

The key file holds more than one key (a file with none fails earlier, with
`Could not extract key(s) from file`). A concatenated bundle is the usual cause.
Split it, and leave the chain in `cert_file` where multiple PEM blocks are fine.

# https://docs.nativelink.com/how-to/upgrade-versions

**Upgrade to a newer version**: Read the changelog for the things that actually break, upgrade the three process types in the right order, and know what happens to the cache you already have.

**Who this is for:** anyone running a pinned version who wants a newer one.
**What you'll have at the end:** a rehearsed upgrade, a rollback that works, and
your existing cache intact. **Time:** thirty minutes for the rehearsal, minutes
for the upgrade itself.

## What is stable and what isn't

The **client-facing API** is the Remote Execution API. It is a published,
versioned protocol, and Bazel talking to NativeLink over `:50051` and `:50052`
is the most stable surface here. You do not generally need to upgrade clients in
step with the server.

## Read the changelog for four things

`CHANGELOG.md` is generated per release, grouped into features, bug fixes,
documentation, testing and CI, miscellaneous, and bumps and version updates,
with a `compare/v1.6.4..v1.6.5` link in each release heading that shows every
commit between the two.

## Rehearse the config before you touch production

No `--validate` flag exists. The binary takes one positional argument, the
config file (plus the usual `--help` and `--version`), and parses it at
startup, which is exactly what you want, because the check is just "does the
new binary accept this file":

## Upgrade in the right order

<li>

## Rolling back

Rollback is the same operation in reverse, with one asymmetry worth internalizing.

## When it doesn't work

Exactly the failure this design intends. The message names the field. Find it in
the new version's config reference. Usually it was renamed, occasionally it
moved under a different parent, rarely it is gone because the feature it
controlled is. Fix the file, do not try to make the parser lenient.

# https://docs.nativelink.com/how-to/migrate-an-existing-cache

**Migrate from another REAPI cache**: Put NativeLink in front of your existing cache as a read-only diode, let real builds warm it, and cut over when the hit rate says you can.

**Who this is for:** anyone running Buildbarn, Buildfarm, bazel-remote or a
hosted REAPI cache who wants to move to NativeLink without a flag day. **What
you'll have at the end:** NativeLink serving your builds, backed by the old
cache, with a cutover you choose rather than one you're forced into. **Time:**
an hour to set up; days to weeks of running both, at your pace.

## The diode

A `fast_slow` store whose `slow` half points at the old cache, marked read-only:

## What crosses the boundary and what doesn't

**Digests must match.** If your old cache holds SHA-256 entries and your
builds now ask for BLAKE3, nothing is found, not because of a bug but because
the key of a blob is its hash, and you are asking a different question.

## Run both while you cut over

Nothing forces a single switch date. Give the two caches separate config groups
in `.bazelrc`, so a build can pick:

## Steps

<li>

## When it doesn't work

Check the digest function first; a mismatch produces exactly this and no error.
Then check `instance_name`, since an old cache scoped to a named instance
returns nothing for a differently-named one. Then confirm the address is
reachable from the NativeLink process rather than from your laptop.

# https://docs.nativelink.com/operate

**Operate**: What to watch, what to tune, and what to do at 3am, once other people depend on your cluster.

**Who this is for:** whoever gets paged when the build farm is slow. **What
you'll have at the end of this section:** a production-shaped deployment, a
metrics pipeline that tells you when it's unhealthy, and a written answer for
each of the ways it fails. **Time:** ongoing.

## The shift this section represents

Everything up to the How-to guides is about making NativeLink do what you want.
This section is about making it keep doing that while you aren't looking.
The questions change shape: not "how do I configure a filesystem store" but
"what happens when that filesystem fills up overnight, how will I know, and what
do I do about it."

## The arc

**Get the shape right.** The production configuration is not the quickstart
configuration with bigger numbers. It has a fast tier over a durable one,
eviction policies that match your disk, and a deliberate split between the
public listeners and the worker API.

## Start here, or skip ahead

**Start here** if you're moving from a working setup to one people depend on.

## Current pages

- [Production configuration](/operate/production-config): the three-process
  shape, the port surface, and the pre-launch checklist.
- [Tuning](/operate/tuning): the lever table (the signal you see, the knob
  that moves it, and what moving it costs).

# https://docs.nativelink.com/operate/production-config

**Production configuration**: The shape a real NativeLink cluster runs in: three processes, a split port surface, and what to turn on before other people depend on it.

**Who this is for:** you have NativeLink working and now other people are going
to depend on it. **What you'll have at the end:** a three-process deployment
whose ports, state, and failure behaviour you can explain to someone else.
**Time:** an afternoon, plus however long your change process takes.

## The shape

Three facts about that picture are the whole design:

## Process 1: the CAS and Action Cache

This is the process that holds bytes. Two stores, one for content and one for
action results, each with its own eviction budget:

### Moving off local disk

`filesystem` is the right terminal store for a single CAS node with a real disk.
For a CAS you intend to replicate, put an object store underneath instead and keep
the local disk as the fast tier:

## Process 2: the scheduler

The scheduler holds the queue and matches actions to workers. It reaches the CAS
over gRPC rather than owning a store:

## Process 3: the workers

Workers are the fleet you scale. Note `servers: []`: a worker serves nothing and
needs no inbound port at all. It dials out to the scheduler and the CAS.

## Which ports are public

`worker_api` is how a process claims to be a worker and receives commands to
  run. It is on its own listener specifically so that it can be on its own
  network policy. Anything that can reach `:50061` can register as a worker,
  receive your actions, and return whatever outputs it likes, which then land in
  your Action Cache and get served to everyone. Keep it inside the cluster.

## What's durable and what's disposable

The rule this table encodes: **exactly one component is stateful.** If you find
yourself needing to protect a scheduler's disk or a worker's disk, something has
drifted from this shape.

## Before you go live

### Raise the file-descriptor limit

Every open blob is a file descriptor. The binary gates file opens on a semaphore
sized to 80% of the limit it actually achieved, so a CAS at the default limit
does not error under load; it waits.

### Turn on namespaces on Linux workers

Bazel in particular can leave zombie processes behind on a worker. Two opt-in
flags contain that:

### Put TLS on the client-facing listeners

Setting `client_ca_file` makes the listener require a client certificate signed by
that CA, and `client_crl_file` lets you revoke one. That is the whole
authorization model on the listener: **any client with a valid, unrevoked
certificate gets full access to that listener's services.** No per-identity
allow-list exists and no mapping from certificate subject to permissions.

### Point telemetry somewhere

The config file has no metrics section, and the binary has no Prometheus scrape endpoint
in the binary. Telemetry is OTLP over gRPC, configured by environment:

### Opt in to cache metrics, if you want them

Execution metrics are emitted unconditionally. **Cache metrics are not**: a store
only reports hits, misses, and errors on its reads, writes, and deletes if you
wrap it:

## What to watch

Four signals cover most of what goes wrong. All four are recording rules in
:

## What to tune

Once the shape is right, [Tuning](/operate/tuning) is the table of levers: the
signal you observe, the knob that moves it, and what it costs you.

## What to do when it breaks

[Troubleshooting](/operate/troubleshooting) indexes symptoms to causes, and
[Runbooks](/operate/runbooks) has the procedure for each of the four incidents
this shape actually produces.

## FAQ

They are lost from the scheduler's point of view. The default scheduler holds
    its queue in process memory and nothing reconstructs it on startup. Clients
    re-submit, so no build fails, but work in progress is repeated. Restart
    schedulers when the queue is shallow.

# https://docs.nativelink.com/operate/tuning

**Tuning**: The lever table: the signal you observe, the knob that moves it, the direction to turn it, and what turning it costs.

**Who this is for:** you have the
[production shape](/operate/production-config) running and a number you don't
like. **What you'll have at the end:** the specific knob for that number, and an
honest account of what moving it costs.

## Storage and the CAS path

### The `evict_page_cache` lever

The `filesystem` store has an opt-in `evict_page_cache` flag (default `false`).
When enabled, after every blob it writes or reads it asks the kernel to drop that
file's pages from the page cache (`posix_fadvise(POSIX_FADV_DONTNEED)`).

## Workers

Worker count itself is not a lever on this page; it is a
[scaling](/operate) decision driven by `nativelink:queue_depth`, not by
worker CPU.

## Scheduler

## Process-level

## What is not a lever

Worth stating, because each of these gets reached for and none of them helps:

# https://docs.nativelink.com/operate/deploy-docker-compose

**Deploy with Docker Compose**: Bring up a complete CAS, scheduler and worker on one host with the compose files in the repository: what each service is, which ports it publishes, and the four things that make this a development setup rather than a production one.

**Who this is for:** you want the whole system (cache *and* remote execution)
running on one machine, either to try it or to give a small team something
shared. **What you'll have at the end:** three containers serving a real CAS,
scheduler and worker, a build pointed at them, and a clear picture of what
separates this from production. **Time:** 20 minutes, most of it spent
compiling.

## The configuration

The first run builds the image from source with Bazel, which takes a while;
see [the build is not fast](#the-build-is-not-fast) below.

### Point a build at it

Cache and executor are **different endpoints**. That is not an accident of the
example: the CAS process and the scheduler process are genuinely separate
here, exactly as they are in the
[production shape](/operate/production-config).

## What to change

Only the fields carrying a decision. Everything else in the shipped files is
already correct for a single host.

## The multi-worker variant

`docker-compose-multi-worker.yml` runs one CAS, one scheduler and three
workers sharing a `cas-data` volume:

## What makes this a development setup

Four things, all of them deliberate in the example and all of them
disqualifying for production.

### The executor runs privileged

`worker.json5` sets `use_namespaces: true` and `use_mount_namespace: true`,
which need `CLONE_NEWUSER`/`CLONE_NEWNS`, hence the flag. Note what this buys
and what it does not: namespaces here are a hermeticity and process-reaping
mechanism, not a security boundary.

### The TLS certificates are expired examples

The 50071 listener loads
`example-do-not-use-in-prod-rootca.crt` and `example-do-not-use-in-prod-key.pem`
from the same directory. They are committed to the repository, self-signed,
`CN=localhost`, and **expired in October 2024**. Every client must pass
`--insecure` or equivalent, which is what the repository's own TLS integration
test does.

### Nothing is authenticated

Not a Compose limitation: open-source NativeLink has no inbound
authentication on any service, in any deployment. `docker-compose.yml` does not
publish the scheduler's port 50061 at all; the worker reaches it over the
Compose network by service name. That is the only thing standing between an
attacker on your network and the ability to register as a worker. Keep it
that way.

### The build is not fast

The `Dockerfile` compiles NativeLink from the repository root with Bazel, in a
three-stage build, and the final image is a fresh Ubuntu with the binary copied
in and `curl` installed. The Ubuntu base is digest-pinned and the bazelisk
download is checksum-verified, so the build is reproducible, but it is a full
from-source build every time the layer cache misses.

## Troubleshooting

## What's next

# https://docs.nativelink.com/operate/deploy-bare-metal

**Deploy on bare metal**: Run NativeLink as systemd services on hosts you own: writing the units, sizing the machines, placing the disks, and rolling an upgrade without dropping in-flight actions.

**Who this is for:** you have VMs or physical machines and no interest in
running a cluster orchestrator to get a build cache. **What you'll have at the
end:** three systemd units, a disk layout that will not surprise you, host
sizing you can defend, and a restart procedure. **Time:** an afternoon.

## The configuration

Three units, one per process. They are near-identical; the differences are the
config file and, for the worker, the security posture.

### The service account and directories

Put the configuration files in `/etc/nativelink/{cas,scheduler,worker}.json5`.

### The CAS unit

### The scheduler unit

Identical apart from the config file and the fact that it needs no writable
state at all when the scheduler backend is the default in-memory one:

### The worker unit

This one is different, and the differences matter.

## What to change

## Disk layout

The CAS host is the one where placement matters.

## Host sizing

No sizing numbers are published for NativeLink; measure your own workload
before buying anything. What the code does fix:

## Restarts and upgrades

The three processes have genuinely different restart characteristics, and
treating them the same is how a routine upgrade turns into a stack of failed
builds.

## Troubleshooting

## What's next

# https://docs.nativelink.com/operate/observability

**Observability**: NativeLink pushes OTLP and nothing else. Wire it to a collector, get the series into Prometheus under the names the shipped rules expect, and know which environment variables actually do something.

**Who this is for:** you are running NativeLink for other people and need to
see queue times, cache hit rates and worker behaviour on a dashboard.
**What you'll have at the end:** a collector receiving OTLP from every
NativeLink process, Prometheus holding the series under the names the shipped
recording rules query, and a Grafana dashboard drawing them. **Time:** 30
minutes.

## NativeLink pushes; it is never scraped

The binary has no `/metrics` endpoint and no configuration that creates
one. NativeLink builds three OTLP exporters at startup (logs, traces and
metrics) and pushes to whatever endpoint the environment names. Every
Prometheus-shaped thing in this pipeline happens downstream of the process.

## The configuration

Point every NativeLink process at a collector. Two variables carry the whole
decision:

## Bring up the pipeline

Then point NativeLink at the collector and restart it:

## What happens if you skip the collector

The `namespace: nativelink` setting on the collector's Prometheus exporter is
what produces every metric name you have ever seen in a NativeLink dashboard.
Send the OTLP stream anywhere else (another collector without that setting,
an OTLP-native backend, or the shipped collector's own `otlphttp/prometheus`
pipeline into Prometheus's receiver) and the names change:

## Turn on cache metrics

Every family except cache metrics is emitted with no configuration. Cache
metrics exist only for stores you explicitly wrap:

## Which metrics exist

The authority is the [metrics reference](/reference/metrics), which is
generated from `nativelink-util/src/metrics.rs` and its call sites. It records
every declared instrument, its type and unit, its histogram buckets, and (the
part no hand-written table stays honest about) **whether anything in the
binary ever emits it**.

## Queries worth having

Cache hit rate, per cache type:

## The Grafana dashboard

A reference dashboard lives at
.
The Compose stack provisions it automatically. Elsewhere: Dashboards → New →
Import, paste the JSON, pick your Prometheus data source. It draws execution
throughput, success rate, active actions by stage and stage transitions from
`nativelink_execution_*`; it has no cache, worker or gRPC panels.

## Troubleshooting

## FAQ

No. The binary has no scrape endpoint, at any configuration, and
    the binary cannot push to Prometheus's OTLP receiver either, because that
    receiver is HTTP-only and NativeLink's exporters are gRPC-only. Run a
    collector and scrape its Prometheus exporter; the shipped collector also
    forwards into Prometheus's OTLP receiver, with the renaming described in
    [what happens if you skip the collector](#wha…

## What's next

# https://docs.nativelink.com/operate/security-hardening

**Security hardening**: Open-source NativeLink has no inbound authentication on any service. What that means for your network, which ports must never be routable, how to configure mTLS, and why the worker sandbox is not a security boundary.

**Who this is for:** you are about to put NativeLink somewhere more than one
person can reach it. **What you'll have at the end:** a network layout where
the dangerous ports are unreachable, mTLS on the ports that must be reachable,
and an accurate picture of the trust you are extending to anyone who can submit
an action. **Time:** 45 minutes, plus however long your certificate authority
takes.

## Start here: there is no authentication

Open-source NativeLink authenticates no inbound request, on any service, in any
deployment. Not the CAS. Not the Action Cache. Not the Execution service. Not
the worker API. Not the admin endpoint. No bearer-token check, no API
key, no mTLS-identity-to-permission mapping; the code that would do it does
not exist in the tree.

## The port inventory

Every listener NativeLink opens, what it serves, and who may reach it. Ports
are the ones the shipped examples use; nothing in the binary reserves them;
[Production configuration](/operate/production-config) covers how the three
processes divide these listeners between them.

## The worker API has no authentication

Port 50061 is the one that turns a cache into a compromise. A process that can
open a gRPC connection to it can call `ConnectWorker`, be added to the
scheduler's pool, and start receiving actions, including the compiler
invocations, the secrets in your action environments, and the outputs that get
written back into the CAS under digests your developers will later trust.

## The admin endpoint has no authentication

The admin REST API registers under `/admin` by default. Every shipped example
that enables it puts it on the worker API listener; one
(`nativelink-config/examples/worker_with_redis_scheduler.json5`) also enables it
on the public listener, which is the thing not to copy. Its only route is:

## The committed certificates are not usable

Two directories in the repository contain a certificate and a key:

## Configure TLS

TLS is per-listener. Each server in the `servers` array carries its own
optional `tls` block. Every field below is documented in
[the configuration reference](/reference/nativelink-config):

### Outbound TLS

Outbound gRPC connections (a `grpc` store's `endpoints`, and the worker's
`worker_api_endpoint`) configure their client side separately with a
`tls_config` block on the endpoint. Object-store backends do not use it; they
carry their own settings (for example `root_certificates` on the `ontap`
provider).

## The sandbox is not a security boundary

For isolation from state left by an earlier action, development builds offer
the opt-in [`single_use` lifecycle](/reference/nativelink-config/main#localworkerconfig).
It requires a launcher that replaces the container and its writable volumes
after every action. This prevents reuse of local files, background processes,
and worker caches.

## What to do, in order

From a developer workstation, `nc -vz <scheduler-host> 50061` fails to
  connect. If it succeeds, stop and fix the network before anything else.

## Troubleshooting

## What's next

# https://docs.nativelink.com/operate/scaling-workers

**Scaling workers**: How much work one worker should take, how many workers you need, which signal tells you, and how to let something else make the decision, including the parts NativeLink does not enforce for you.

**Who this is for:** you have a worker pool serving real builds and the queue is
either backing up or sitting idle. **What you'll have at the end:** a sized
worker, a scaling signal you trust, and either a manual playbook or an
autoscaler wired to it. **Time:** an afternoon for the manual version, longer if
you're building the autoscaling path.

## Two axes, and only one of them is a NativeLink knob

Capacity has a vertical axis (how many actions one worker runs at once) and a
horizontal axis (how many workers exist). NativeLink has a field for the first.
It has nothing at all for the second, by design: a worker is a process that
dials the scheduler and announces itself, so "more workers" means "start more
processes," and whatever starts processes for you is the thing that scales.

## Sizing one worker

The knob is `max_inflight_tasks` on the worker's config.

## The ceilings you hit before you expect to

A worker sized generously runs into limits that are not on the worker config,
and they're worth knowing before you go hunting.

## The signal to scale on

Not CPU. Worker CPU tells you whether a machine is busy; it cannot tell you
whether there is work waiting that nobody is doing, and that's the only question
an autoscaler needs answered.

## Scaling out

<li>

## What the repository actually ships

Being explicit, because the gap is larger than the docs elsewhere imply.

## Backpressure, when a worker needs to stop

One mechanism lets a worker refuse work without being removed:
`experimental_precondition_script`. It runs before every action, and a non-zero
exit produces a `ResourceExhausted` error. The scheduler requeues the action
without counting it as an attempt, and marks the worker `is_paused` **only if
that worker still has other actions in flight**; the pause clears as soon as
one of them completes.

## Heterogeneous pools

One scheduler can route to pools of different shapes; the mechanism is platform
properties, and it is covered in full on
[platform properties](/remote-execution/platform-properties). The scaling point
is narrower: **each pool is a separate scaling unit with its own queue-depth
series**, because `nativelink:queue_depth` groups by instance and priority, not
by which pool could serve the work.

## When it doesn't work

Actions are queued that no worker matches. This is nearly always a platform
property mismatch: an action declaring something no worker advertises, or a
scheduler key configured as `priority` (which constrains nothing) when it should
be `exact` or `minimum`. Adding workers will not help. Check the scheduler's
worker-match logging, and see
[platform properties](/remote-execution/platform-properties).

# https://docs.nativelink.com/operate/scaling-cas

**Scaling the CAS**: How to grow the content-addressable store: composing tiers, sizing eviction, sharding, and the specific reasons a second CAS replica is harder than a second worker.

**Who this is for:** you run a CAS that real builds depend on, and it is
running out of disk, running out of throughput, or both. **What you'll have at
the end:** a store composition sized for your load, an eviction policy that
doesn't thrash, and a clear-eyed view of what running more than one CAS process
costs. **Time:** an afternoon to size and tune; longer if you go horizontal.

## Start here: a CAS process is not stateless

Most guidance about scaling services assumes replicas are interchangeable. For
NativeLink's CAS that assumption is wrong in five specific places, and every
horizontal-scaling decision on this page follows from them.

## Scale vertically first, by composing

The store graph is where most capacity problems are solved, and it is cheaper
than adding processes because it changes nothing about consistency.
[Compose stores](/how-to/stores/compose-stores) is the reference for the
wrappers; what follows is which one relieves which pressure.

## Eviction is your capacity control, and its default thrashes

Four fields, all on `EvictionPolicy`, all defaulting to `0`, and `0` means
*disabled* in every case. A policy with no fields set is an unbounded store.

### Size the AC and the CAS together

The Action Cache is just another store with its own `EvictionPolicy`; there is
no separate mechanism. The shipped examples give it roughly a twentieth of the
CAS budget, which is a reasonable starting ratio because AC entries are tiny.

## Sharding, and what changing it costs

`shard` distributes keys across several backends by hash, with an optional
integer `weight` per shard. The weights are turned into a cumulative table over
the `u32` space at startup and the key is binary-searched into it.

## Going horizontal

Everything above scales one process. If you have exhausted that, here is the
order to add replicas in.

## The ceilings you hit before you expect to

**File descriptors, at 80% of what you configured.** `global.max_open_files`
defaults to 24576, and the process reserves a fifth of it for sockets and pipes,
so the effective permit budget is about 19660.

## Metrics: nothing is on by default

This is the finding most likely to catch you out while you're trying to measure
any of the above.

## What the repository actually ships

Being explicit, because the gap between the examples and a production CAS is
wide.

## When it doesn't work

That's the `evict_bytes: 0` thrash. With no low watermark, every write pushes
the store over the limit and re-enters the eviction loop to remove the minimum.
Set `evict_bytes` to a real fraction of `max_bytes`, enough that a burst of
writes fits under the headroom it creates.

# https://docs.nativelink.com/operate/autoscaling-reference

**An autoscaling reference deployment**: The complete Kubernetes deployment that scales workers on queue depth without a human in the loop: every manifest, the reaction-time budget nobody publishes, and the five gaps in the shipped examples you have to close first.

**Who this is for:** you have run [scaling workers](/operate/scaling-workers) and
[scaling the CAS](/operate/scaling-cas) by hand, you trust the numbers, and you
want the pool to size itself. **What you'll have at the end:** a worker
deployment that grows and shrinks on queue depth, with probes, drain, and
disruption budgets wired up. **Time:** a day to assemble, a week of watching
before you trust it.

## What ships, and what this page supplies

Being blunt about this up front, because it changes how you read everything
below.

## The reaction-time budget

The signal an autoscaler reads is not emitted by NativeLink. It is derived,
several hops downstream, and each hop adds staleness. Here is the whole chain
with its shipped defaults.

## Five gaps you close first

The shipped Kubernetes manifests cannot support an HPA as they stand. Not
because of anything subtle, but because of five specific absences. Each one is a
short fix and all five are prerequisites.

## The health endpoint does less than you expect

Before you write a readiness probe, understand what `/status` actually
measures, because the answer is surprising in both directions.

## Readiness is a different question

`/status` answers whether the components that registered a health indicator
have failed. It stays `200` while a component is still initializing, which is
the right answer for a liveness probe and the wrong one for readiness. The
`health` service therefore also serves `/ready` (`readiness_path`, default
`/ready`), which answers `503` while anything is initializing or failed and
`200` otherwise.

## Build the deployment

<li>

## What this deployment does not do

Worth stating so the gaps are choices rather than surprises.

## When it doesn't work

Work backwards along the chain. Query the recording rule in Prometheus directly;
if it returns nothing, the problem is upstream of Kubernetes entirely and the
adapter is fine. If the rule returns data but the adapter doesn't, the usual
cause is the `resources.overrides` block: external metrics still need a
namespace association, and a series with no `namespace` label cannot be matched
to one.

# https://docs.nativelink.com/operate/runbooks

**Runbooks**: What to do when a NativeLink deployment is on fire: Redis failover, a worker OOM-killed, the disk full, and the queue backpressured. Each entry is a symptom, the signal that confirms it, the immediate action, and the change that stops it recurring.

**Who this is for:** you are on call and something is broken right now.
**What you'll have at the end:** for each of the four incidents NativeLink
actually produces, a confirming signal, a stop-the-bleeding action, and the
configuration change that prevents a repeat. **Time:** read once now, so you do
not read it for the first time at 03:00.

## A worker is OOM-killed

The most common production incident, and the one with the most misleading
first symptom: your developers report a *build* failure, not an
*infrastructure* failure.

### Confirm it

Two distinct failures land here and they need different responses. Work
out which one you have before doing anything.

### Stop the bleeding

Drain the affected worker rather than killing it, so its in-flight actions
finish instead of being destroyed:

### Make it stop happening

NativeLink itself has **no memory configuration**. No `max_memory`, no
per-action memory cap, nothing. Memory is bounded only indirectly, by store
eviction policies and by whatever the container runtime enforces. Do not go
looking for a config field; it does not exist.

## The disk fills up

### Confirm it

Writes start failing with an OS-level message wrapped in a NativeLink err_tip:
`Failed to write data into filesystem store`, `Failed to flush in filesystem
store`, `Failed to sync in filesystem store`.

### The signal to look for first

If your eviction snapshot says **"of unlimited"**, no size-based eviction is
configured, and this incident was inevitable. `eviction_policy` is an
`Option` on the filesystem store, and every field in it defaults to `0`, which
means *never evict*. The config's own doc comment is blunt about it: failure to
set this value causes items to never be removed.

### Stop the bleeding

Set `max_bytes` and restart the process. Eviction is only ever invoked on map
mutation (on insert and remove), so it runs as soon as traffic resumes.

### Size the volume for twice `max_bytes`

The filesystem store keeps an executable-variant directory
(`{content_path}.exec`) alongside the content path, and **that directory is
invisible to `max_bytes`**. A variant is deleted when its primary entry is
evicted, which bounds the total at roughly `2 * max_bytes` in the worst case
(every blob also held as an executable).

### Two layout footguns

`temp_path` must be on the **same block device** as `content_path`. If it is
not, the atomic rename becomes a copy: slower, and it doubles peak space
during every write.

## Redis fails over

### What actually happens

Both Sentinel and Cluster modes are supported; the default is `standard`. On a
Sentinel failover the demoted master answers `-READONLY`, which NativeLink
classifies as retryable alongside dropped connections, refusals and IO errors.

### What breaks depends on your backend

### Stop the bleeding

If you are on the `memory` backend, this is a cache-availability incident, not
a scheduler incident. Builds degrade to local execution and cache misses. Fix
Redis at your leisure.

### Configuration worth checking now

`response_timeout_s` and `connection_timeout_s` are **deprecated**;
`response_timeout_s` is ignored entirely. Both log a warning at startup. Setting
both `connection_timeout_s` and `connection_timeout_ms` is a startup error.

## The queue backpressures and never recovers

The subtlest incident of the four, because nothing errors and nothing restarts.
Actions re-queue forever.

### The mechanism

`ResourceExhausted` is NativeLink's backpressure code. When a worker returns
it for an action, two things happen: the action goes back to `Queued` and
(this is the important part) **its attempt counter is not incremented**. If
the worker still has other actions in flight it is also marked `is_paused`,
and that flag clears as soon as any of those actions completes.

### Confirm it

then look at the scheduler's periodic pass: the action keeps appearing in
`Oldest actions in state` as `Queued`, its age grows, and it never reaches a
retry limit. The worker itself is quiet; the script's exit status is logged
only at `TRACE` (`Preconditions script returned`), so `RUST_LOG=info` on the
worker shows nothing.

### Stop the bleeding

Fix or remove the precondition script and restart the worker. Nothing can
clear `is_paused` from the outside; it is set and cleared by the scheduler
in response to what the worker returns.

### Related limits that block rather than reject

Several NativeLink limits queue instead of erroring, which is why "hung" is a
more common symptom than "failed":

## Every build stalls at `[Sched]` and nothing errors

Clients sit on `WaitExecution` for hours, the scheduler log is quiet apart
from `Could not find AwaitedAction for the given operation id`, idle workers
take nothing, and the provisioner reads demand as zero. The scheduler's
records are being evicted from Redis.

## The scheduler's Redis keeps every completed action

**Symptom.** With the Redis scheduler backend, `DBSIZE` on the scheduler's
database grows by one `aa_*` hash per action ever run and never falls; the
records all read `completed` and `TTL` says `-1`. Once the search index holds
more than one cursor page, listings over it (the admin read the worker
provisioner makes, the scheduler admin endpoints) can fail with
`Failed to get operations list! Parse: Non map item`.

## Restarting the scheduler

Not an incident so much as a procedure that behaves worse than you expect.

## What's next

# https://docs.nativelink.com/operate/troubleshooting

**Troubleshooting**: A symptom-to-cause-to-fix index for NativeLink, anchored on the error strings the system actually emits, including the ones that say nothing useful, and the failures that produce no error at all.

**Who this is for:** something is wrong and you have a symptom but not a cause.
**What you'll have at the end:** the cause, and either a fix or a pointer to
the procedure that fixes it. **Time:** find your row, follow it.

## The index

## Actions queue forever and workers sit idle

The single most important asymmetry in NativeLink:

### Diagnose it

Raise the log level. Every useful diagnostic here is at `INFO`, and the shipped
examples run at `warn`.

### Property types that constrain nothing

A scheduler key configured as `priority` when it should be `exact` or `minimum`
is the classic version of this bug, and it fails in the *opposite* direction:
actions get scheduled onto workers that cannot run them. See
[Scaling workers](/operate/scaling-workers) for the fleet-shaped version of
this problem and [Platform properties](/remote-execution/platform-properties)
for the full matching semantics.

## Actions fail with a CAS miss

### The string you will actually see

Bazel reports `FAILED_PRECONDITION`. Not `NOT_FOUND`. That translation happens
on the worker, which checks for a `NotFound` whose message contains
`not found in either fast or slow store` and converts it before returning:

### The two causes

**Workers do not share CAS storage.** The fix is exactly what the message says:
every worker's `fast_slow` slow tier must resolve to the same CAS the clients
upload to. In Compose, `docker-compose-multi-worker.yml` does that by pointing
each worker's slow store at the `cas-server` over gRPC (the `cas-data` volume
it also mounts into the workers is not what shares the data); in Kubernetes,
a shared backing store rathe…

### A near-miss worth knowing

Zero-byte outputs are not stored. Callers materialise them directly. If you see
this in a stack trace it is a caller bug, not a cache problem.

## The timeout message that tells you nothing

This means the client stopped waiting, and `client_action_timeout_s` (default
60 s) cleaned the operation up. It is a statement about the *client*, not about
why the action never ran.

## Timeouts

The timeout defaults that matter: `max_action_timeout_s` 20 minutes,
`max_upload_timeout_s` 10 minutes, `max_cleanup_wait_s` 30 seconds,
`worker_timeout_s` 10 seconds, `client_action_timeout_s` 60 seconds,
`retain_completed_for_s` 60 seconds.

## Zero means default

Setting a field to `0` to disable it usually does the opposite.

## Worker connection churn

A worker that logs at a steady ~2 Hz is reconnecting:

## Limits that block instead of erroring

If the symptom is "everything got slow and nothing errored", you are probably
sitting on one of these:

## Startup failures

These all abort the process before it serves anything. All of them are config
wiring, and all of them name the offending key.

## Retries

Retry exhaustion is **not** a distinct error. It is the last underlying error
with a suffix appended:

## Where else to look

[Scaling workers](/operate/scaling-workers) has a "when it doesn't work"
section covering the fleet-shaped versions of several problems above: queue
depth high while workers idle, adding workers not helping, scale-in causing
failed actions, two workers on one host behaving erratically. It complements
this page rather than duplicating it.

## What's next

# https://docs.nativelink.com/explanations/architecture

**Architecture**: The four roles, who actually writes to the action cache, and which parts of the system hold state you can lose.

**Who this is for:** anyone deciding how to deploy NativeLink, or trying to
predict which component's failure costs them what. **What you'll have at the
end:** the four roles, the path an action takes on a miss and on a hit, and
an accurate answer to "what do I have to back up". **Time:** fifteen minutes.

## The four roles

### CAS: content-addressable storage

Every byte NativeLink touches lives in the CAS, keyed by a digest of its
content. Source files, object files, binaries, stdout, all of it. Two
identical files anywhere in your organisation collapse into one stored blob,
because they hash to the same key.

### AC: action cache

The action cache maps `action digest → ActionResult`. The action digest
covers the command line, every input file's digest, the environment
variables, and the platform properties, so an AC hit means *this exact
computation has been performed before, and here is where its outputs are*.

### Scheduler

The scheduler receives `Execute` RPCs, decides which queued action goes next,
finds a worker whose platform properties satisfy the action's, and assigns
the work. It also merges duplicate requests, so two developers building the
same commit wait on one execution rather than two.

### Workers

A worker fetches the action's inputs from the CAS, materialises them into a
directory, runs the command, uploads the outputs, and reports back.

## One build, twice

The concrete version. A developer runs `bazel build //app:server`, which
expands to roughly two thousand actions. Take one of them: compiling
`main.cc`.

## What holds state

Deployment planning comes down to this table, and one widespread claim about
it is wrong.

## Why Rust, why this shape

Three properties drove the design, and all three are operational rather than
aesthetic.

## Common questions

The CAS, and only the CAS. The AC is a lookup table that refills itself.
    Workers hold nothing durable. The scheduler holds in-flight state that is
    worth nothing after a restart anyway; the clients will retry.

# https://docs.nativelink.com/explanations/architecture-deep-dive

**Architecture deep dive**: The crate graph, the startup sequence, and the path an action actually takes through the binary.

**Who this is for:** anyone about to read, extend or debug the NativeLink
source, and anyone who needs to predict what the binary does rather than
what the protocol says it should. **What you'll have at the end:** the crate
graph, the order things come up in at startup, and where an action's path
forks. **Time:** fifteen minutes.

## One binary, five decisions

`nativelink` is a single executable. Which roles it plays is decided entirely
by the config file it is handed; there are no build features to toggle and
no separate images per role. Everything below happens inside
.

## The crate graph

Twelve library crates plus the binary, in strict layers. The diagram shows
the ten that give the program its shape; the other two are
`nativelink-metric-macro-derive` (the proc macro behind `nativelink-metric`)
and `nativelink-redis-tester` (a fake Redis used by the store crate and by
tests). `nativelink-macro` is a test-only proc macro that no library crate
depends on.

## Configuration is JSON5, and it is validated early

Config parsing is JSON5, not JSON: comments and trailing commas are legal,
which is why every example in these docs uses them. The entry point is
`CasConfig::try_from_json5_file` in
.

## Stores are built in two phases

`store_factory` constructs each configured store, and then a separate
`run_post_init()` pass runs across all of them. The second pass exists because
a `ref_store` names another top-level store (possibly one declared later in
the file), and a single-pass construction would make declaration order
load-bearing.

## Services come up in a fixed order

The registration order is: action cache, CAS, execution, `Operations`, Remote
Asset fetch, Remote Asset push, ByteStream, capabilities, worker API, and the
experimental Build Event Protocol service. Each is registered only if the
config asks for it.

## Where an action's path forks

The single most important structural fact about execution is that **a cache
hit never reaches the scheduler**. Bazel calls `GetActionResult` before it
calls `Execute`, so on a hit the execution service is never contacted at all.

### The pre-flight check is deliberately racy

Before an `Execute` is accepted, the execution service does a shallow check
that the action's inputs are present: the `Action` proto itself, its
`command_digest` and its `input_root_digest`. It is documented in-source as
deliberately shallow and deliberately racy: it does not walk the full input
tree, and a blob can be evicted between the check passing and the worker
asking for it.

## Two details that surprise people reading the wire

**`GetTree` page tokens are `"{hash}-{size}"`.** They are constructed and
parsed by splitting on `-`. They are not opaque, and they are not signed,
though the protocol says clients should treat them as opaque, and clients
that do will keep working if this ever changes.

## Common questions

Because the roles have different scaling and failure characteristics, not
    because the code requires it. A CAS wants durable storage and steady
    memory; a worker wants CPU and is disposable; a scheduler wants to be
    restarted freely. Splitting them lets you size and autoscale each one
    separately. See [production configuration](/operate/production-config).

# https://docs.nativelink.com/explanations/store-model

**The store model**: What a store actually is, how composition behaves, and where a digest stops being a claim and becomes a fact.

**Who this is for:** anyone composing stores in a config file and wanting to
predict the result, and anyone writing a new store backend. **What you'll
have at the end:** the trait, the key space, the composition rules, and the
one place verification actually happens. **Time:** twenty minutes.

## Three names, one file

`Store`, `StoreLike` and `StoreDriver` all live in
,
and the split between them matters when you read the source.

## The key space is not just digests

`StoreKey` has two variants: `Digest(DigestInfo)` and `Str(Cow<str>)`. Most
of NativeLink deals in the first. The second exists because two subsystems
need to store things that are not content-addressed at all: the Build Event
Protocol service, and the scheduler's awaited-action database.

## The digest on the wire is a claim, not a measurement

This is the single most important thing on this page.

### How the verify store actually works

The implementation in

## Composition, and the barriers in it

`inner_store` is how the system looks *through* a wrapper to find an
optimisation opportunity, for example, to discover that the real backing
store is a filesystem store and a hardlink is possible.

### `fast_slow` writes through, and deliberately re-uploads

With the default `fast_direction` and `slow_direction`, writes go to both
tiers concurrently and **both must succeed**. It has no write-behind and no
eventual consistency window: an upload that returns OK is durable in the slow
tier.

### `dedup` is a chunking index, and it has opinions

A `dedup` store splits blobs with content-defined chunking (FastCDC), stores
the chunks, and writes an index under the **original** key. Two large blobs
that share most of their content share most of their chunks.

## Two things that are not what their filenames say

and

## Where the exact truth lives

Every store type, every field and every default is in the generated
[store configuration reference](/reference/nativelink-config/store-overview). This
page deliberately does not restate field names, because the reference is
generated from the Rust types and this page is not.

## Common questions

On the write path of anything shared between people, yes. The cost is one
    hash pass over uploaded bytes, concurrent with the write rather than
    serialised before it. The thing it buys is that a digest in your cache
    means what it says. On a purely local single-developer cache the
    calculation is different, because you are the only client and you are not
    defending against yourself.

# https://docs.nativelink.com/explanations/scheduler-internals

**Scheduler internals**: Action merging, the awaited-action database, how a worker is chosen, and what happens when an action fails.

**Who this is for:** anyone tuning a scheduler, debugging an action that
queued forever, or wondering why two identical builds did not share work.
**What you'll have at the end:** the merge rule, the ordering rule, the
matching rule, and the retry rule, each of which has at least one
counter-intuitive edge. **Time:** twenty-five minutes.

## No `ActionScheduler` trait exists

If you go looking for one (and older material about NativeLink will send you
looking) you will not find it. The abstraction is split by *who is asking*,
into three traits in
:

## When are two actions the same action?

Two clients asking for the same work should wait on one execution. The key
that decides this is `ActionUniqueKey`, and it has exactly three fields:
instance name, digest function, and the action digest.

## Ordering is one 64-bit integer

`AwaitedActionSortKey` packs the entire queue ordering into a single `u64`:
the upper 32 bits are the priority (shifted from `i32` into unsigned range so
byte order matches numeric order), and the lower 32 bits are the insertion
timestamp in whole seconds XORed with `u32::MAX`.

## Two state enums, and one that never appears

`ActionStage` is the wire-facing state a client sees.
`SortedAwaitedActionState` is the database index state the scheduler sorts
on. They overlap but are not the same, and conflating them while reading the
source is a reliable way to get lost.

## Matching is first-fit over an LRU

The matching engine walks the queue in sort order, and for each action looks
for the first worker that satisfies the action's platform properties. The
source still carries an `O(n*m)` TODO in
,
but the walk is narrowed first: a `WorkerCapabilityIndex` (an inverted index
over `Exact` and `Priority` properties) produces the candidate set, and only
that set is scanned in LRU order, checking availability and `Minimum`
va…

### Platform properties are typed, and `Minimum` is a resource pool

Every property key a worker advertises has a declared type:

## Retries, and what "failed" means

The attempt counter is incremented by one event, and two other events that
used to spend it now have budgets of their own.

### Two timeouts with very different defaults

`max_action_executing_timeout_s` **defaults to `0`, which means disabled.**
It is not the only guard against a hung action: the worker kills the process
when the action's own timeout (or the worker's `max_action_timeout_s`, 20
minutes by default) elapses, and the scheduler re-queues an executing action
whose worker stops sending updates for `worker_timeout_s`.

## Who writes the action cache entry?

Not the scheduler. **The worker writes the action cache entry, and it does so
before telling the scheduler that the action finished.** The ordering is what
guarantees that a client which observes completion and then queries the cache
finds the result there.

## Where the exact truth lives

Every scheduler field and default is in the generated
[scheduler configuration reference](/reference/nativelink-config#namedschedulerconfig).
For the operational levers, see [tuning](/operate/tuning).

## Common questions

Check the three fields in the merge key first: instance name, digest
    function, action digest. A different instance name is the usual answer,
    and the second most common is one client sending `skip_cache_lookup`,
    which opts that action out of merging entirely.

# https://docs.nativelink.com/explanations/worker-execution

**Worker execution**: How an action becomes a running process: input materialization, what isolation you actually get, and what the worker throws away.

**Who this is for:** anyone running workers, debugging an action that behaves
differently on a worker than locally, or deciding how much to trust the
actions they run. **What you'll have at the end:** the lifecycle, the
filesystem layout, the exact isolation boundary, and the list of things the
worker deliberately discards. **Time:** twenty-five minutes.

## The connection is one stream

A worker opens exactly one bidirectional gRPC stream to the scheduler
(`ConnectWorker`, the sole RPC on NativeLink's own `WorkerApi` service), and
everything flows over it. It has no polling loop and no second connection.

## The filesystem, and the hardlink that everything depends on

On startup the worker **purges and recreates its `work_directory`**. Anything
left there by a previous run is gone. This is intentional (a worker that
inherits state is a worker whose actions are not hermetic), but it means the
work directory must not be somewhere you also keep things.

## What isolation you actually get

This section is deliberately blunt, because overclaiming here is dangerous.

## The environment is built from nothing

The worker calls `env_clear()` first. Nothing is inherited. Variables are
then applied in this order:

## Timeouts, signals, and the output you lose

A requested timeout above `max_action_timeout_s` (20 minutes if unset) is
**rejected with `InvalidArgument`, not clamped**. The client is told its
request was unacceptable rather than silently receiving a shorter deadline
than it asked for. That is the right choice, and one worth knowing before you
set a low ceiling on a cluster whose clients ask for long ones.

## Uploading outputs, and what is dropped

The ordering here is subtle and matters for anyone reasoning about
consistency. When the command exits, the worker sends `ExecuteComplete` to
the scheduler (the process is done, so a new action can be assigned) and
*then* uploads outputs, deletes the action directory, and writes the action
cache entry.

## Cleanup, and three things called eviction

The entire action directory is deleted after every action, with a `Drop`
backstop for the panicking case and a path-canonicalization check so that a
malicious or malformed operation ID cannot direct the delete outside the work
directory.

## Where the exact truth lives

Every worker field and default is in the generated
[worker configuration reference](/reference/nativelink-config#workerconfig). The
execution code itself is

## Common questions

It is almost certainly modifying an input file in place. Inputs are
    hardlinks to read-only CAS blobs. Copy the file into the output tree
    first, or declare it as an output rather than an input.

# https://docs.nativelink.com/explanations/correctness-hermeticity

**Correctness and hermeticity**: What content addressing guarantees, what it doesn't, and which of the gaps are yours to close.

**Who this is for:** anyone who needs to know how much to trust a cache hit,
and anyone chasing a build that is correct on one machine and wrong on
another. **What you'll have at the end:** an honest inventory of the
guarantees: the ones you get for free, the ones you have to configure, and
the ones nobody is offering. **Time:** twenty minutes.

## What you get for free

**Collision resistance.** SHA-256 and BLAKE3 are the supported digest
functions. Two different blobs will not share a key.

## What you do not get by default

Five gaps, in rough order of how often they bite.

### The digest is a claim until you verify it

The server does not hash plain uploaded bytes. The client declares the digest
and the server files the bytes under it. (Zstd-compressed ByteStream uploads
are the exception: the decoder checks size and hash as it decodes.) For
everything else the only thing that checks is a `verify` store, and both
`verify_hash` and `verify_size` default to `false`.

### Verification is write-side only

Even with a `verify` store, nothing is re-checked on read. Blobs that entered
before you added the wrapper, or through a path that bypassed it, are served
without complaint.

### Hermeticity is the client's job

The action digest covers exactly what the client put into the `Action`
message: the command, the input tree, the platform properties, the
environment variables it declared. Anything the action reads that is not in
there is invisible to the cache, and a cache hit will confidently return a
result computed with a different value of it.

### Actions can run more than once

The default `max_job_retries` of 3 is checked strictly, so an action may
execute **four** times on its own failures. A lost worker has its own budget,
`max_worker_loss_retries` (default 10), and a memory escalation its own
(`memory_escalation.max_steps`, default 8), so an action can run more often
than four times when workers die under it or it keeps outgrowing its
reservation; a `ResourceExhausted` never costs anyth…

### Some output metadata is discarded

`node_properties` is never populated, so output file modes and mtimes do not
survive a round trip. Absolute output symlinks are dereferenced and uploaded
as content; only relative ones stay symlinks. A declared output that the
action did not produce is not an error.

## Caching semantics that surprise people

Three, all of which have bitten someone:

## A checklist for a cache you can trust

<li>
    **Put `verify_hash: true` in the write path.** Everything below is
    downstream of this being true.
  </li>
  <li>
    **Pin the toolchain at the client**, so the compiler's identity is inside
    the action digest rather than beside it.
  </li>
  <li>
    **Make undeclared inputs fail rather than succeed.** Strict action
    environment, strict include checking, no reliance on ambient `PATH`.

## Common questions

Build the same target on two machines that differ, and compare outputs
    byte for byte; anything that differs is reading something undeclared.
    At cluster scale the symptom is subtler: cache misses where you expected
    hits, because a leaked input changed and moved the action digest with it.
    A rising miss rate on unchanged code is the signal.

# https://docs.nativelink.com/explanations/lre

**Local Remote Execution**: What LRE actually is: a way to make your local toolchain and your remote workers the same toolchain, so both sides share one cache.

**Who this is for:** anyone deciding whether LRE is worth adopting, and
anyone who has read about it and formed the wrong model of what it does.
**What you'll have at the end:** the actual mechanism, the guarantee it makes
and its exact qualifiers, and what it does not cover. **Time:** fifteen
minutes.

## The mechanism, precisely

The trick is absolute paths.

## The guarantee, with its qualifiers intact

The source states the promise carefully, and the qualifiers are load-bearing:

## What is pinned, and what is not

This is where careful reading pays off.

## Coverage

Windows is not supported. On macOS the C++ path has no local toolchain at
all (the flake module only emits the C++ platform and toolchain flags on
Linux), so an `lre-cc` action can only run on a Linux worker carrying the
image; you get hermeticity there, not locality.

## Why Nix, specifically

Because the design needs every tool the build invokes to be identified by a
hash of everything that produced it, and to live at a path that encodes that
hash. That is precisely what a Nix store path is.

## When it is worth it

## Where the exact truth lives

The flake-side and Bazel-side wiring, the regeneration commands, and the
downstream-project setup are in
.
For actually running it (the commands, and what is local versus remote once
you leave `x86_64-linux`) see
[toolchains and hermeticity](/remote-execution/toolchains-and-hermeticity),
which is in the Remote execution section.

## Common questions

No, and it does not try to. LRE is about toolchain identity, not about
    where work runs. Fan-out across hundreds of cores still wants remote
    workers; the point is that the workers and your laptop now agree on what
    a compiler is, so both populate one cache.

# https://docs.nativelink.com/explanations/history

**History and licensing**: What NativeLink was built to replace, the three properties that drove the rewrite, and exactly how the source is licensed.

**Who this is for:** anyone evaluating NativeLink against an existing build
farm, and anyone who needs to answer a licensing question precisely. **What
you'll have at the end:** the design pressures that produced the current
system, and a file-by-file account of what license applies. **Time:** ten
minutes.

## What the existing options traded away

Two of those three problems are runtime problems rather than design problems.
That observation is the whole argument for the rewrite: the protocol was
fine, the storage model was fine, and what was expensive was the substrate.

## The three properties that drove the design

**Predictable latency.** In a build system, the p99 is the number developers
experience: a single stalled action blocks everything downstream of it in
the graph. Removing the garbage collector removes a class of latency spike
that is very hard to tune away rather than eliminate.

## Licensing, precisely

This question gets answered vaguely often enough to be worth stating exactly.

## Versioning

NativeLink versions look like SemVer and are not.

## Why these docs were rewritten

The previous documentation accumulated during the period when the project
was changing shape weekly. It described commands that no longer existed,
components that had been renamed, and performance numbers with no source.

## Common questions

It is source-available today and open source on a timer. The FSL 1.1
    grant permits everything except offering NativeLink itself as a
    competing commercial service, and each release converts to Apache 2.0
    two years after publication. If your concern is "can I read and modify
    the code that runs my builds", the answer is yes, without qualification.

# https://docs.nativelink.com/reference/nativelink-config

**Configuration reference**: Every knob in the NativeLink JSON5 configuration — types, defaults, and links to source, autogenerated from the Rust config crate.

_Generated reference page, not inlined. Fetch it from https://docs.nativelink.com/reference/nativelink-config._

# https://docs.nativelink.com/reference/nativelink-config/store-overview

**Store overview**: The mental model behind every store in the configuration reference: what holds data, what wraps another store, and how they compose.

The [full reference](/reference/nativelink-config) lists every field on
every store. This page is the map you need before that list is useful:
what a store actually is, the two families every store falls into, and
the compositions people actually run in production.

## What a store is

A store is a named, pluggable backend for CAS or AC data. You declare
each one once in the top-level `stores` array, then reference it by
name from `servers`, `workers`, or from inside another store:

## Two families

Every store type is either **terminal** (it actually holds or fetches
bytes) or a **wrapper** (it adds behavior in front of another store,
which it embeds as a nested `StoreSpec`). Wrappers nest arbitrarily
deep: a `verify` can wrap a `compression`, which wraps a `fast_slow`,
which wraps two more terminal stores:

### Terminal stores: where bytes actually live

Every implementation in this table lives in `nativelink-store/src`.

### Wrapper stores: behavior layered on a backend

The nested-`StoreSpec` shape each wrapper takes (one backend, two named
backends, or N) is declared alongside the terminal specs.

## Choosing a terminal backend

## Common compositions

**Single-node dev cache**: no wrapping at all. See
[Configuration → Stores](/configuration/stores).

## FAQ

No; it never checks that an object in `fast` also made it to `slow`.
    If an artifact must survive a fast-tier wipe, make sure the write
    path covers both tiers deliberately.

## What's next

- [Configuration reference](/reference/nativelink-config): every
  field, default, and JSON5 example for every store type above.
- [Configuration](/configuration): how stores fit alongside servers,
  schedulers, and workers.
- [Production configuration](/operate/production-config): the store
  composition a real cluster uses end to end.

# https://docs.nativelink.com/reference/protocol-api

**Protocol and API surface**: Every gRPC service NativeLink serves, which RPCs are implemented, and the values it advertises.

**Who this is for:** anyone writing or debugging a client, or checking
whether a given REAPI feature is available. **What you'll have at the end:**
the complete service list with implementation status. **Time:** use as a
lookup.

## Services

Each is registered only if the configuration asks for it. Registration order
at startup is fixed: action cache, CAS, execution, `Operations`, fetch, push,
ByteStream, capabilities, worker API, BEP.

## `WorkerApi`: the only NativeLink-defined service

Defined in
.
It has **exactly one RPC**:

## Partially implemented RPCs

Three `Operations` RPCs are registered and return `unimplemented`:

## What `Capabilities` advertises

Two of these need a caveat.

## Wire-level details worth knowing

**ByteStream resource names are parsed right to left.** A resource name is
`{instance_name}/blobs/{hash}/{size}`, and the instance name may itself
contain slashes, so left-to-right parsing cannot find the boundary, while
right-to-left can, because the trailing components are fixed. See
.

## Related

- [Architecture deep dive](/explanations/architecture-deep-dive): where each
  service sits in the process
- [Scheduler internals](/explanations/scheduler-internals): what happens
  between `Execute` and a worker
- [Worker execution](/explanations/worker-execution): the `WorkerApi` stream
  from the worker's side
- [Configuration reference](/reference/nativelink-config): which services a
  given config enables

# https://docs.nativelink.com/reference/metrics

**Metrics reference**: Every OpenTelemetry instrument NativeLink declares, with its type, unit, attributes, Prometheus series, and whether the binary actually emits it. Autogenerated from the Rust source and its call sites.

_Generated reference page, not inlined. Fetch it from https://docs.nativelink.com/reference/metrics._

# https://docs.nativelink.com/reference/cli-and-env

**CLI and environment**: Every command-line argument the nativelink binary accepts and every environment variable it reads: the telemetry variables, the ones the configuration expands, and the ones it passes through to actions.

**Who this is for:** you are writing a systemd unit, a Dockerfile `CMD`, or a
Kubernetes `env:` block and need to know what the process actually reads.
**What you'll have at the end:** the complete argument and variable surface,
with the sharp edges marked.

## The command line

That is the whole thing. `config_file` is a required positional argument, the
path to a JSON5 configuration file, and there are no other arguments. The
binary is a [clap](https://docs.rs/clap) parser with `version`, `about` and
`author` derived, so `--help` and `--version` work and nothing else does.

## Telemetry variables

These are read once, at startup, by `nativelink-util`'s tracing initialiser.
They are the only variables that change the binary's own behaviour.

## Variables the configuration file expands

Many configuration fields are deserialized through a shellexpand wrapper rather
than directly, so their string values undergo environment substitution before
the value is used.

## Variables the worker passes to actions

A worker does not forward its own environment to the commands it runs: the
child is spawned with `env_clear()`. An action sees exactly two things: the
`environment_variables` its own `Command` message carries, and whatever the
worker's `additional_environment` map (on the `local` worker configuration)
names explicitly.

## What NativeLink does not read

Stated explicitly, because each of these is a reasonable guess that is wrong:

## Reading the source

If anything here disagrees with the binary, the source wins:

## Where to go next

# https://docs.nativelink.com/reference/oss-and-enterprise

**Open source and Enterprise**: What the open-source distribution actually contains, which two modules are licensed differently, why nothing in the binary enforces that, and what the paid tiers are for.

**Who this is for:** anyone who has to answer "are we allowed to run this?",
and anyone deciding between self-hosting and NativeLink Enterprise. **What you'll
have at the end:** an accurate picture of what is licensed how, what activates
it, and what the open-source build does and does not restrict. **Time:** ten
minutes.

## The short version

The whole of NativeLink is in the public repository. No separate
"enterprise build", no stripped-down community edition, and no compile-time
feature flag that divides them. What differs between the free and paid product
is the *licence you hold*, not the *code you run*.

## Two licences, one repository

The repository root carries `LICENSE`, the Functional Source License 1.1 with
an Apache 2.0 future grant. Its restriction is narrower than most people
expect on first reading: it permits everything except a **Competing Use**,
which the licence defines as making the software available to others in a
commercial product or service that substitutes for NativeLink, substitutes for
something Trace Machina already offers us…

## The Business Source Licensed modules

Five files in the tree carry a Business Source License 1.1 header instead of
the repository licence. They form two modules:

## Nothing in the binary enforces any of this

No licence key field exists in the configuration schema. No startup
  validation reads one. No runtime gate wraps either module. Searching the
  whole tree for enforcement turns up exactly five hits: the five header
  comments listed above, and nothing else. Both modules are compiled
  unconditionally and both run by default.

### Metrics are always on

No configuration turns instrumentation off. The meter provider
is constructed unconditionally during startup and installed globally, and the
scheduler and store layers call into the licensed instruments on their normal
paths: `simple_scheduler_state_manager.rs` records completed executions,
`memory_awaited_action_db.rs` records stage transitions, and
`cache_metrics_store.rs` records every cache operation.

### Persistent workers activate from the action, not from your config

No `persistent_worker` key exists anywhere in the configuration schema;
searching `nativelink-config` for it returns nothing at all. The module is
reached entirely from the action's own platform properties:

## What the open-source build does not limit

Nothing in the code artificially constrains scale. Specifically, there is no
cap on cache size, stored object count, action throughput, connected worker
count, or number of people using the cluster; no trial period, expiry date, or
time-limited mode; and no registration, activation, or account required to
start the binary.

## What Enterprise adds

The honest answer, from inside this repository, is that the paid tier is not
a different codebase, so this documentation cannot enumerate its feature list
by reading the source. What can be said accurately:

## Telling what you are running

The binary is a [clap](https://docs.rs/clap) parser with `version` derived from
the crate, so:

## FAQ

Running it as your own build infrastructure is internal use, which the
    Functional Source License permits regardless of whether your company is
    commercial. The separate question is the two Business Source modules:
    metrics in a shared or production setting needs a licence, and metrics
    cannot be switched off.

# https://docs.nativelink.com/reference/glossary

**Glossary**: The vocabulary you need to read the rest of the docs.

**Who this is for:** anyone reading the rest of these docs who hit a word that
was used as if it needed no explanation. **What you'll have at the end:** the one
sentence you needed, and a link to the page that goes deeper. **Time:** use as
a lookup.

## Action

A single unit of build work. Encodes a command line, input file
digests, the platform requirements, and the expected outputs. Hashing
an action produces a stable identifier; identical actions have
identical hashes.

## Action Cache (AC)

A keyed store: `hash(Action) → ActionResult`. Cache hits skip the
work entirely.

## ActionResult

What an action produces. Output file digests, exit code, captured
stdout/stderr, timing metadata.

## Awaited action

The scheduler's record of one action between `Execute` and completion: its
current [execution stage](#execution-stage), its priority, and every client
waiting on it. Two identical actions submitted at the same time join the same
awaited action instead of running twice.

## ByteStream API

The streaming transport for blobs too large to move in a single
`BatchUpdateBlobs` or `BatchReadBlobs` call, with the [digest](#digest)
encoded in the resource name. Where the batch calls stop and this one starts
is a server limit the [Capabilities API](#capabilities-api) advertises.

## Capabilities API

The RE-API service a client calls first, to learn which
[digest functions](#digest-function) the server accepts and what its size
limits are rather than assuming. See
[protocol and API surface](/reference/protocol-api).

## CAS: Content-Addressable Storage

Stores every blob (source file, intermediate output, final binary)
under the digest of its contents. The same byte sequence stored from anywhere
collapses to one entry.

## Completeness-checking store

An [Action Cache](#action-cache-ac) wrapper that checks every output
[digest](#digest) still exists in the CAS before serving a hit. Without it a
CAS eviction turns old cache entries into hits that reference blobs nobody can
download, and the build fails while fetching outputs instead of quietly
missing the cache. Valid only on AC stores.

## Dedup store

A store that splits blobs into content-defined chunks with FastCDC
() and stores each chunk
once, so two versions of a large file that differ in the middle share
everything else. The cost is an index lookup per read. See
[the store model](/explanations/store-model).

## Digest

A content hash plus a size. Used everywhere instead of file paths.

## Digest function

The hash a client and server agree on for [digests](#digest): SHA-256 or
BLAKE3, negotiated per request rather than fixed at build time
().
Two clients using different functions produce different keys for the same
bytes, so they do not share cache entries.

## Eviction policy

The limits a store evicts against: `max_bytes`, `max_seconds`, `max_count`,
and the `evict_bytes` low watermark that stops eviction thrashing right at the
limit. Each defaults to `0`, which means *never evict on that dimension*: a
store configured without a policy grows until the disk does.

## Execution stage

Where an action is in its life: `CacheCheck`, `Queued`, `Executing`, then
`Completed` or `CompletedFromCache`
().
An action stuck in `Queued` has no free [worker](#worker) whose
[platform properties](#platform-properties) match what it asked for.
`CacheCheck` is declared but the scheduler never reports it, because the
cache lookup happens in a wrapper in front of the scheduler.

## Existence cache store

A store that remembers which digests it has already confirmed exist, so
repeated [`FindMissingBlobs`](#findmissingblobs) calls stop re-querying the
backend. It answers the existence question only; reads pass through.

## `fast_slow` store

A two-tier store. Reads try the fast backend, fall back to the slow one, and
copy what they find into the fast tier on the way back; writes mirror to both.
Its one sharp edge: it never checks the slow tier for something the fast tier
already has, so a blob that reached the fast tier without reaching the slow one
stays invisible to everything reading the slow tier directly.

## FindMissingBlobs

The CAS call that makes uploads incremental: the client offers a list of
[digests](#digest), the server returns the ones it does not have, and the
client uploads only those. On a warm cache this and the [AC](#action-cache-ac)
lookup are most of what a client does.

## Hermetic build

A build whose outputs depend only on its declared inputs. Same
inputs → same outputs, on any machine, any time. What NativeLink does
and does not guarantee here is in
[correctness and hermeticity](/explanations/correctness-hermeticity).

## Input root

The complete file tree an action is allowed to see, encoded as a Merkle tree
of `Directory` messages that reference their files and each other by
[digest](#digest). Its root digest is part of the [action](#action) hash, so
changing one input file changes the cache key.

## Instance name

A namespace inside a NativeLink cluster. Each service entry maps an
instance name to a store or scheduler, so one cluster can serve multiple
isolated environments, and the scheduler keys in-flight actions by instance
name as well as digest. Two instance names that point at the same store share
its contents.

## Isolation

Keeping an action from seeing state outside its declared inputs.
On Linux, workers can isolate actions with kernel namespaces via
the `use_namespaces` and `use_mount_namespace` worker options,
which also reap zombie processes and improve hermeticity.
NativeLink doesn't currently integrate external sandboxing tools
like `bwrap`, `landlock`, or `sandbox-exec`.

## JSON5

The dialect NativeLink's configuration file is written in: JSON plus
comments, trailing commas, and unquoted keys. Every example in
[the configuration reference](/reference/nativelink-config) is JSON5.

## LRE: Local Remote Execution

Building your toolchain with Nix so that your local build and your
remote workers invoke the *same* toolchain binaries, at the same
`/nix/store` paths, which makes their action digests match and lets
both share one cache. No worker runs on your laptop. See
[LRE](/explanations/lre).

## Operation

The handle a client holds on a running action, from
`google.longrunning.Operation`. `Execute` returns one immediately and the
client watches it for [stage](#execution-stage) changes; `WaitExecution`
re-attaches to it after a disconnect, which is what stops a dropped connection
from re-running the work.

## Origin events

An experimental stream of the requests and responses a server handled,
published to a store for an external consumer to read
().
Off unless `experimental_origin_events` is configured.

## Platform properties

Key/value tags attached to an action ("this action needs Linux,
x86_64, a GPU") and to a worker ("this worker has Linux, x86_64, a
GPU"). The scheduler matches them.

## Priority

An integer on `ExecuteRequest`'s `execution_policy`; NativeLink dequeues
the highest value first when workers are scarce (the RE-API leaves the
direction to the server, and its suggested default is the reverse). It orders
the queue and nothing more; an action already executing is not interrupted for
a higher-priority one that arrives after it.

## Remote Execution API (RE-API)

The standard gRPC protocol every supported build system speaks.
[Spec](https://github.com/bazelbuild/remote-apis).

## Scheduler

The dispatcher. Receives `Execute` calls, picks workers, tracks
in-flight actions.

## Shard store

A store that routes each key to one of several backends by digest hash, with
a `weight` per backend. It spreads capacity; it is not replication, so losing
one backend loses that backend's share of the data rather than none of it.

## Store

NativeLink's unit of storage configuration: a named thing that gets and puts
bytes under a key. Stores compose, because a store's backend is another store,
so the CAS a server serves is usually a stack of wrappers over one real
backend. See [the store model](/explanations/store-model).

## Toolchain

The bundle of binaries an action needs to run: compiler, linker,
standard library, etc. See
[how toolchains are provided](/remote-execution/toolchains-and-hermeticity#faq).

## Verify store

A wrapper that, depending on `verify_size` and `verify_hash`, checks a
blob's size and re-hashes it on write, failing the upload when the bytes do
not match the [digest](#digest) they were offered under. It turns silent corruption into a write error. Both checks
belong on CAS stores and neither belongs on an [AC](#action-cache-ac) store.

## Worker

The process that runs an action. Fetches inputs from CAS, runs the
command, uploads outputs to CAS.

## Worker API

NativeLink's own gRPC service, not part of the RE-API, over which a worker
registers with a scheduler and receives work
().
One bidirectional `ConnectWorker` stream carries the whole conversation, which
is what keeps a worker talking to the same scheduler instance even behind a
load balancer.

# https://docs.nativelink.com/reference/changelog

**Changelog**: Notable changes per release. Latest first.

_Generated reference page, not inlined. Fetch it from https://docs.nativelink.com/reference/changelog._

# https://docs.nativelink.com/contribute/guidelines

**Contribution guidelines**: What NativeLink accepts, the git setup it requires, and the commit and review conventions a PR is held to.

**Who this is for:** anyone about to open their first pull request.
**What you'll have at the end:** the git configuration the project
requires, the commit conventions, and an accurate picture of how review
works. **Time:** thirty minutes, most of it one-time setup.

## What gets accepted

Three things reliably do not land: breaking changes to the Remote Execution
API surface, which is an upstream spec the project tracks rather than owns;
vendor-specific code paths only one deployment can use; and refactoring
proposed on its own merits rather than as part of a change that needed it.

## Git setup, once

<li>
    **Create distinct authentication and signing keys** in your
    [GitHub key settings](https://github.com/settings/keys). Two keys, not
    one reused.
  </li>
  <li>
    **Fork the repository** and clone your fork, not upstream:

## The pull request loop

<li>
    **Sync your fork**, then branch:

## Before you push

`.rustfmt.toml` uses `group_imports` and `imports_granularity`, both
  nightly-only. Stable `cargo fmt` ignores them silently and produces output
  CI rejects. Use the Bazel target.

## What reviewers look for

**One change per pull request.** Small ones land; large ones get split.

## Licensing

Your contribution is licensed under the license that applies to the file or
module you change. Most of the repository is `FSL-1.1-Apache-2.0`; a handful
of files are Apache-2.0; and two areas,
`nativelink-util/src/metrics.rs` and everything under
`nativelink-worker/src/persistent_worker/`, are **Business Source
License**. Copy the header from the file next to the one you are creating
rather than from memory.

## Common questions

[Cargo](/contribute/cargo) for small localized changes, the fastest way
    in. [Bazel](/contribute/bazel) when you want exactly what CI runs.
    [Nix](/contribute/nix) for the pinned shell, required for LRE work and
    for the generated `bazelrc` fragments.

# https://docs.nativelink.com/contribute/bazel

**Develop with Bazel**: Build, test, and run NativeLink with Bazel, the same dev loop CI uses.

The Bazel-based dev loop is the one our CI runs. Use this if you want
the most reproducible setup or if your editor integration prefers
Bazel.

## Prerequisites

- Bazelisk, which reads `.bazelversion` and fetches the right Bazel for
  you. CI runs 9.1.1 and also checks 8.7.0.
- A C++ toolchain (clang or gcc) for native dependencies.
- On macOS: XCode command-line tools.
- On Linux, inside the Nix shell, the LRE toolchains link with `mold`;
  outside it Bazel uses whatever the host toolchain provides.

## First build

The first `bazel test` will take 10-20 minutes as it builds the
toolchain and dependencies. Subsequent builds finish in seconds.

## Common commands

See [the testing guide](/contribute/testing-guide) for what `unit_test`
and `integration` actually mean in this repository; they do not mean what
the names suggest.

## Editor integration

For `rust-analyzer`:

## Common pitfalls

- **LRE flags are missing**: `.bazelrc` pulls in `lre.bazelrc`,
  `nativelink.bazelrc`, `nixos.bazelrc` and `darwin.bazelrc` with
  `try-import`, and the Nix dev shell is what generates them. Without
  `nix develop` they do not exist, and the `try-` means Bazel
  says nothing about it. See [Develop with Nix](/contribute/nix).
- **Slow first build with no cache**: expected.

## FAQ

It bootstraps the toolchain and builds every dependency from source.
    Subsequent builds are seconds, and pointing `--remote_cache` at a
    NativeLink cluster (above) makes even a fresh clone fast.

# https://docs.nativelink.com/contribute/cargo

**Developing with Cargo**: Pure Cargo workflow for NativeLink contributors who'd rather not run Bazel.

Every NativeLink crate compiles cleanly with plain Cargo. Use this
workflow if you want a smaller dev shell, faster incremental builds
during iteration, or just don't have Bazel installed.

## Prerequisites

- A recent stable Rust toolchain. No `rust-toolchain.toml` is checked in; the
  minimum is `rust-version` in the root `Cargo.toml`, currently 1.97.1,
  on edition 2024.
- On Linux: `mold` is optional and has to be configured by hand; see
  [faster incremental builds](#faster-incremental-builds).

## First build

First build is a few minutes (dependency compilation); subsequent
incremental builds are typically a few seconds.

## Common commands

`.rustfmt.toml` uses `group_imports` and `imports_granularity`, both
  nightly-only. Stable `cargo fmt` silently ignores them and produces
  output CI rejects. Run `bazel run --config=rustfmt @rules_rust//:rustfmt`
  instead.

## Faster incremental builds

A few flags that help:

## Caveats

- Three directories are excluded from the workspace and build on their
  own: `nativelink-config/generate-stores-config`, `nativelink-test/fuzz`,
  and `tools/generate-bazel-rc`. The LRE flake outputs are Nix, not Cargo.
- Regenerating protos goes through Bazel
  (`bazel run nativelink-proto:update_protos`), not Cargo.
- CI runs Bazel, and `bazel test` attaches rustfmt and clippy aspects that
  a Cargo run does not.

## FAQ

Not to build or test; the whole workspace builds with plain Cargo.
    You need Bazel for a pre-submit `bazel test //...`, for regenerating
    protos, and for formatting, since that's what CI runs.

# https://docs.nativelink.com/contribute/nix

**Develop with Nix**: The dev shell that generates the bazelrc files the rest of the build assumes, and what is actually inside it.

**Who this is for:** anyone who wants the toolchain CI uses, and anyone
working on [LRE](/explanations/lre), which needs it. **What you'll have at
the end:** the shell, the five generated `bazelrc` files it produces, and an
accurate list of what it does and does not contain. **Time:** twenty
minutes, mostly download.

## Install Nix

The
[next-gen Nix installer](https://github.com/NixOS/experimental-nix-installer)
is the easiest path:

## Enter the dev shell

First run downloads the toolchain, which takes a few minutes. After that,
entering is close to instant; everything is content-addressed in
`/nix/store`.

## What entering the shell generates

This is the part that matters and the part that is invisible. The shell hook
writes five things into your working tree:

## What is in the shell

The package list is in
.
Broadly:

## Running things inside the shell

Everything from the [Bazel](/contribute/bazel) and
[Cargo](/contribute/cargo) workflows works unchanged. The shell guarantees
the versions, and supplies the `bazelrc` fragments those workflows assume.

## LRE and worker images

The flake builds the LRE worker images rather than exposing an LRE shell.
The ones you are likely to want:

## When Nix isn't worth it

For a typo fix, a doc tweak or a single unit test, the
[Cargo workflow](/contribute/cargo) is faster to get into and nothing will
go wrong. Reach for Nix when you are touching the build graph, working on
LRE, chasing a difference between your machine and CI, or working on the
docs, since `vale` and `bun` both come from here.

## Common questions

A package manager that builds every package in isolation and addresses
    the result by a hash of all its inputs, which is why two machines
    running `nix develop` on this repo get bit-identical toolchains.

# https://docs.nativelink.com/contribute/repo-crate-map

**Repository and crate map**: What every crate and top-level directory in the NativeLink repository is for, and which one your change belongs in.

**Who this is for:** anyone opening the NativeLink repository for the first
time and trying to work out where their change goes. **What you'll have at
the end:** the crate layering, the size of each piece, the directories that
are not crates, and the half-dozen places where the first guess is wrong.
**Time:** fifteen minutes.

## The workspace has no members list

The root

## The crates

Source sizes exclude each crate's `tests/` directory; the test column counts
files directly under `tests/`.

## The layering

Every internal dependency points down this stack. It has no cycles, and
the shape is worth memorising because it tells you what a change can reach.

## Where the first guess is wrong

Six of these, and each one has cost somebody an afternoon.

## Directories that are not crates

They are not alternative spellings of each other. `deploy/` is the
  overlays, `deployment-examples/` is copyable manifests, and `kubernetes/`
  is the building blocks the overlays compose.

## Conventions you will notice immediately

Every manifest except the nested macro-derive crate's starts with
`#:schema tools/cargo-with-detailed-deps.json`, which is what gives you
completion and validation in a TOML-aware editor.

## Common questions

Two, plus a Bazel file: the spec goes in `nativelink-config`, the
    implementation and its factory arm go in `nativelink-store`, and the new
    file has to be added to `nativelink-store/BUILD.bazel`. The step-by-step
    is in [how to extend](/contribute/how-to-extend#adding-a-store).

# https://docs.nativelink.com/contribute/codebase-internals

**Codebase internals**: The traits, the error model, the async rules and the two metric systems: the conventions a reviewer will assume you already know.

**Who this is for:** anyone about to write NativeLink code rather than read
it. **What you'll have at the end:** the four things that are enforced
rather than suggested: the store trait split, the error model, the
spawning rules, and which metric system to use. **Time:** thirty minutes.

## The three store traits, and which one you implement

Three of them exist, and they are not interchangeable.

## The scheduler traits are split by caller

No `ActionScheduler` trait exists. The state layer is split into
`ClientStateManager`, `WorkerStateManager` and `MatchingEngineStateManager`
in
,
plus `WorkerScheduler`, `KnownPlatformPropertyProvider` and
`AwaitedActionDb`. The split by *caller* is what allows the in-memory and
Redis-backed implementations to be swapped wholesale.

## The error model

`Code` **is** `tonic::Code`, re-exported from
.
No separate internal code enum and no mapping layer exist, which is why
an error raised deep in a store arrives at a client with a sensible gRPC
status without anyone writing a conversion.

## Async: `tokio::spawn` is banned

The Tokio runtime is built by hand in `main()`, not with `#[tokio::main]`:
the builder installs an `on_thread_start` hook (macOS QoS), and the process
signal handlers and the shutdown broadcast are spawned onto it before
`inner_main` runs. Each of those calls carries an
`#[expect(clippy::disallowed_methods, reason = …)]`, because the same lint
that bans `tokio::spawn` everywhere else bans the runtime builder too.

## Two metric systems

They are not redundant, and picking the wrong one is a common review
comment.

## Lints, formatting and the small stuff

`clippy::all`, `clippy::nursery` and `clippy::pedantic` are all denied at
the workspace level. `std_instead_of_core` is denied too, which is why you
will see `use core::…` throughout for anything that does not need `std`.

## Common questions

Because a bare spawn has no name, no tracing span, no OpenTelemetry
    context and no cancellation story. The three macros in
    `nativelink-util::task` supply all four, and the clippy config makes the
    alternative a build failure rather than a review comment.

# https://docs.nativelink.com/contribute/how-to-extend

**How to extend NativeLink**: Complete file-by-file recipes for adding a store, a config field, a metric, or a gRPC service.

**Who this is for:** anyone adding a new component rather than changing an
existing one. **What you'll have at the end:** four recipes, each listing
every file a working change touches, including the ones that compile fine
when you forget them and fail in CI. **Time:** use as a checklist.

## Adding a store

Model: `nativelink-store/src/noop_store.rs`, the smallest complete
`StoreDriver` in the tree.

## Adding a config field

Config fields are documentation. The doc comment becomes the description in
the generated reference, and the ```` ```json ```` fences in `stores.rs` are
also harvested into the examples file, which is why they have to be valid
JSON rather than illustrative fragments.

## Adding a metric

Decide which system first; see
[two metric systems](/contribute/codebase-internals#two-metric-systems).

## Adding a gRPC service

Model: `nativelink-service/src/fetch_server.rs`: 165 lines, and it has
every convention in it.

## Common questions

Almost certainly the `srcs` list in `nativelink-store/BUILD.bazel`. Cargo
    globs; Bazel enumerates. Add the file, keep the list alphabetical, and
    run `bazel build //nativelink-store:nativelink-store` before pushing.

# https://docs.nativelink.com/contribute/testing-guide

**Testing guide**: How NativeLink's tests are organised, what `#[nativelink_test]` gives you, and which local environment problems cause which confusing failure.

**Who this is for:** anyone writing a test, or trying to work out why a
test that passes in CI fails on their laptop. **What you'll have at the
end:** the commands, the conventions, the fixtures that already exist, and
a diagnosis list for the local failures that are not your code's fault.
**Time:** twenty minutes.

## The commands

In this repository, `unit_test` means *tests written inside a source file*
  and `integration` means *tests in the crate's `tests/` directory*. Neither
  implies anything about scope or about talking to a network. The
  genuinely end-to-end suites are the shell scripts in `integration_tests/`,
  which are separate again.

## Where tests live

The convention is a separate `tests/` directory per crate, not inline
modules; only nine `mod tests` blocks exist in the whole tree.

## `#[nativelink_test]`

Around 780 tests use it. It is defined in

## Fixtures that already exist

Reach for these before writing your own.

## The end-to-end suites

It refuses to run as root, requires Bazel, Docker and `sudo`, and waits for
ports 50051, 50052 and 50071 before starting. It drives
`simple_cache_test.sh`, `simple_remote_execution_test.sh`,
`simple_tls_test.sh` and `chunking_cache_test.sh`. You can pass a pattern to
run one of them.

## Coverage

The `result` symlink contains an HTML report. **No coverage
threshold gate**; no PR is blocked on a percentage. Coverage is a tool for
finding untested paths, not a target to hit.

## When it fails locally but not in CI

Most of these are environment, not code.

## Common questions

Run the crate you touched with Cargo while iterating, then
    `bazel test //...` once before pushing. CI runs Bazel, and the aspects
    attached to `bazel test` catch formatting and clippy problems that a
    Cargo run does not.

# https://docs.nativelink.com/contribute/release-versioning

**Releases and versioning**: How a NativeLink version becomes a signed tag, what the tag triggers, and what compatibility the project does and does not promise.

**Who this is for:** maintainers cutting a release, and anyone who needs to
know what a version number means before pinning one. **What you'll have at
the end:** the fourteen files a version lives in, the release sequence, what
each workflow produces, and the compatibility policy, which is mostly the
absence of one. **Time:** twenty minutes.

## The version lives in fourteen files

No single source of truth for the version exists, because the root
`Cargo.toml` declares a `[package]` version rather than a
`[workspace.package]` version, so nothing inherits it.

## Cutting a release

The authoritative sequence is the numbered list under "Creating releases" in
.
The shape of it:

## What the tag triggers

`release.yaml` builds three targets (`x86_64-unknown-linux-musl`,
`aarch64-unknown-linux-musl` and `aarch64-apple-darwin`), signs each with
keyless Sigstore cosign, and attaches SLSA Build Level 3 provenance.

## What the version number promises

Less than you would assume, and it is better to say so than to imply
otherwise.

## Regenerating the configuration reference

The reference pages under `reference/nativelink-config/` are generated from
the Rust config crate and must never be hand-edited. The chain is:
`cargo run --bin build-schema --features dev-schema` produces a JSON schema,
`web/apps/docs/scripts/gen-config-reference.mjs` and `scripts/lib/schema-to-mdx.mjs`
turn it into MDX, and the whole thing is invoked as:

## Common questions

It is not guaranteed to. No SemVer policy is stated, and
    `compatibility_level = 0` in `MODULE.bazel` explicitly asserts none. Read
    the release notes, which do call out breaking changes and include
    migration instructions.

# https://docs.nativelink.com/contribute/docs

**Working on documentation**: How these docs are built: the four page archetypes, the components, the anchor and snippet lints, and what a docs PR is reviewed against.

**Who this is for:** anyone writing or editing a page on this site. **What
you'll have at the end:** the local dev loop, the archetype your page has to
satisfy, the components you can use without importing them, and the lints
that will fail your PR if you skip them. **Time:** thirty minutes, once.

## The local loop

Edit any `.mdx` file under `apps/docs/content/docs/` and the dev server picks
it up on save. To preview the marketing site and the docs together (the
marketing app redirects `/docs` to the docs app in dev), run `bun dev` and use
`http://localhost:3000/docs`.

## Pick the archetype first

Every page is exactly one of four kinds, and the kind determines the shape.
Mixing two is the most common structural review comment, because a tutorial
that pauses to explain tradeoffs stops being a guaranteed happy path, and an
explanation that turns into a procedure stops being readable out of order.

## The opening block every page has

Directly under the archetype comment, before anything else:

## Frontmatter and navigation

Two required fields, one optional:

## Components

All of these are provided globally. **Never write an import in an `.mdx`
file**; an import statement is the usual reason a page compiles locally and
fails in the MDX check.

# https://docs.nativelink.com/

**For agents**: How an AI agent should read these docs end to end, where every entry point is, how to cite and verify a claim, and what the docs guarantee to a machine reader.

**Who this is for:** an AI agent (Claude Code, Cursor, Copilot, a homegrown
tool) that has been pointed at NativeLink, and the person pointing it.
**What you'll have at the end:** the entry points, a procedure for reading
the whole corpus in order, the conventions that make pages citable and
verifiable, and a task index into the rest of the docs. **Time:** five
minutes to read, one fetch to load everything.

## Read this first if you are an agent

1. Fetch **[`/llms-full.txt`](/llms-full.txt)**. It is the entire documentation
   corpus, every page body verbatim, in reading order, in one plain-text file
   (under 1 MB). One fetch gives you everything a human would find by
   clicking through the sidebar.
2.

## The entry points

**`AGENTS.md` at the repo root**  is for an
agent working *on the code*. It maps every `nativelink-*` crate to what it
owns, says where config, store, scheduler and worker logic actually live,
gives the build and test commands, and carries a table of "you changed X, so
this doc has to follow" pairs that are often missed.

## Skills for coding agents

The repository's `.claude/skills/` directory carries skills in the `SKILL.md`
format that Claude Code, Cursor, Copilot, Codex and other agents read. Claude
Code loads them automatically inside the repository; elsewhere, copy or
symlink a skill into the agent's skills directory.

## Contributing as an agent

Agents are welcome contributors when a person stands behind the change. The
rules are in
[`CONTRIBUTING.md`](https://github.com/TraceMachina/nativelink/blob/main/CONTRIBUTING.md#ai-assisted-contributions)
and are short: the person submitting the change must understand it well
enough to answer for any line; the pull request template's "AI assistance"
section must say which tools helped and how much ("None" is a comple…

## How the corpus is ordered

The sidebar, `llms.txt` and `llms-full.txt` all use the same order, defined
once in the `meta.json` files under `content/docs/`: