docs: Remove refs for distributed caching (dagger#8867)

* docs: Remove refs to distributed caching

Signed-off-by: Vikram Vaswani <[email protected]>

* Removed image and text

Signed-off-by: Vikram Vaswani <[email protected]>

* Removed more content from FAQ

Signed-off-by: Vikram Vaswani <[email protected]>

* docs: Reword Dagger "distributed cache" to just "persistent cache"

The valuable property is persistence (across runs). Distribution is an
implementation detail (usually platform-specific).

Signed-off-by: Gerhard Lazu <[email protected]>

* docs: Remove S3 Gateway mention from Dagger Cloud

This was only relevant in the context of Dagger Cloud Cache.

Signed-off-by: Gerhard Lazu <[email protected]>

* docs: Remove Shaed Cloud Cache from Kubernetes integration

Mention persistent storage as a platform-specific optimization that is
worth exploring, but don't add more detail since it's contextual.

Signed-off-by: Gerhard Lazu <[email protected]>

---------

Signed-off-by: Vikram Vaswani <[email protected]>
Signed-off-by: Gerhard Lazu <[email protected]>
Co-authored-by: Gerhard Lazu <[email protected]>

docs/current_docs/adopting.mdx

@@ -115,7 +115,7 @@ A few important notes on CI integration:
 
 - Caching makes everything faster. As a general rule, CI pipelines are very inefficient, so it's common that daggerizing them will make them faster. Importantly, it may also *change your scaling requirements*. In extreme cases, a pipeline that needed horizontal scaling across many machines, might fit comfortably on a single machine after daggerizing.
 
-- Caching is harder in CI. Most modern CI systems rely on ephemeral runners. This means that *out of the box, the Dagger cache will always be empty*. The typical symptom is that a Dagger pipeline that was very fast on your local machine, feels slower in CI. The solution is to implement distributed caching. How to do this depends on your CI infrastructure, and is an area of improvement for Dagger. One option is the experimental distributed cache feature in [Dagger Cloud](https://dagger.io/cloud). Another is to deploy CI runners and Dagger engine together on a [shared Kubernetes cluster](https://docs.dagger.io/integrations/kubernetes#base-pattern-persistent-nodes). For more discussion of distributed cache options, join our Discord, contact us directly, or [read this discussion thread](https://github.com/dagger/dagger/issues/6486#issuecomment-1910551524). If you're not sure where to start, you can also start your CI integration without distributed caching, and get back to it later. Everything will still work, just not as fast as it could be. Unlike traditional CI, you don't need to change your pipeline logic to add caching: once you turn it on, everything will just "magically" be faster.
+- Caching is harder in CI. Most modern CI systems rely on ephemeral runners. This means that *out of the box, the Dagger cache will always be empty*. The typical symptom is that a Dagger pipeline that was fast on your local machine, feels slower in CI. The solution is to configure some form of storage for Dagger. How to do this depends on your CI infrastructure. One option is to use persistent volumes for Dagger's state directory. For more discussion of Dagger persistent cache options, join our Discord, contact us directly, or [read this discussion thread](https://github.com/dagger/dagger/issues/6486#issuecomment-1910551524). If you're not sure where to start, you can also start your CI integration without persistent cache, and get back to it later. Everything will still work, just not as fast as it could be. Unlike traditional CI, you don't need to change your pipeline logic to add persistent cache: once you turn it on, everything will just "magically" be faster.
 
 ### The Demo
 

docs/current_docs/configuration/cloud.mdx

@@ -84,15 +84,10 @@ Once you have your token, you can use it to connect Dagger Cloud with your CI en
     You must store the Dagger Cloud token as a secret (not plaintext) with your CI environment and reference it in your CI workflow. Using a secret is recommended to protect your Dagger Cloud account from being used by forks of your project. We provide links in the steps below for configuring secrets with popular CI tools.
     :::
   - Add the secret to your CI environment as a variable named `DAGGER_CLOUD_TOKEN`.
-- For **Team** plan subscribers with ephemeral CI runners only: Add a wait step (example below) for the Docker configuration to allow Dagger to push cache volumes to the experimental Dagger Cloud distributed cache. Subscribers on the **Individual** plan do not have access to the experimental distributed cache and do not need to add this step.
 - If you are using GitHub Actions, install the Dagger Cloud GitHub app for GitHub Checks. This app adds a GitHub check to your pull requests. Each check links to the corresponding pipeline visualization in Dagger Cloud.
 
 You can use Dagger Cloud whether you're hosting your own CI runners and infrastructure or using hosted/SaaS runners.
 
-:::danger
-When using self-hosted CI runners on AWS infrastructure, NAT Gateways are a common source of unexpected network charges. It's advisable to setup an Amazon S3 Gateway for these cases. Refer to the [AWS documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html) for detailed information on how to do this.
-:::
-
 <Tabs groupId="ci">
 <TabItem value="GitHub Actions">
 
@@ -113,15 +108,11 @@ When using self-hosted CI runners on AWS infrastructure, NAT Gateways are a comm
 
     [See an example of a sample GitLab workflow file](../integrations/gitlab.mdx)
 
-1. For **Team** plan subscribers with ephemeral CI runners only: Update your GitLab CI workflow (example below) and adjust the `docker stop` timeout period so that Docker waits longer before killing the Dagger container, to give it more time to push data to the Dagger Cloud cache. Refer to the Docker documentation on the [`docker stop` command](https://docs.docker.com/engine/reference/commandline/stop/).
-
 </TabItem>
 <TabItem value="CircleCI">
 
 1. Create a new environment variable in your CircleCI project named `DAGGER_CLOUD_TOKEN` and set it to the value of the token obtained in [Step 1](#step-1-sign-up-for-dagger-cloud). Refer to the CircleCI documentation on [creating environment variables for a project](https://circleci.com/docs/set-environment-variable/#set-an-environment-variable-in-a-project).
 
-1. For **Team** plan subscribers with ephemeral CI runners only: Update your CircleCI workflow (example below) and adjust the `docker stop` timeout period so that Docker waits longer before killing the Dagger container, to give it more time to push data to the Dagger Cloud cache. Refer to the Docker documentation on the [`docker stop` command](https://docs.docker.com/engine/reference/commandline/stop/).
-
 1. For GitHub, GitLab or Atlassian Bitbucket source code repositories only: Update your CircleCI workflow and add the following pipeline values to the CI environment. Refer to the CircleCI documentation on [using pipeline values](https://circleci.com/docs/variables/#pipeline-values).
 
     GitHub:
@@ -191,10 +182,6 @@ The next step is to test the integration with Dagger Cloud.
 1. In your terminal, call a Dagger Function. You will see a Dagger Cloud URL printed in the terminal.
 1. Click on the URL to view details of the run in Dagger Cloud.
 
-:::info
-Dagger 0.11.0 adds support for visualizing Dagger functions but does not yet print the Dagger Cloud URL for the pipeline run in the CLI output. We're adding this soon.
-:::
-
 ```shell
 dagger -m github.com/shykes/daggerverse/[email protected] call hello
 ```
@@ -235,29 +222,3 @@ In Dagger Cloud, users can have one of two roles: *Admin* or *Member*. The Admin
 | Delete an existing member from an org | ✓ |  |
 
 You cannot change a member's role at this time. Please contact Dagger via the support messenger in Dagger Cloud if you need assistance.
-
-## Cache volumes
-
-:::warning
-Dagger Cloud's distributed caching feature is currently experimental.
-:::
-
-Dagger Cloud automatically detects and creates cache volumes when they are declared in your Dagger pipeline or custom functions.
-
-Your cache volumes will appear in the UI within a few minutes after your Dagger function(s) have completed execution. If you don't see them in the UI, ensure you've referenced the volumes correctly in your code and that you've set up your CI or local development workflows to push the cache volumes to Dagger Cloud.
-
-To see your cache volumes, browse to the **Organization Settings** page of the Dagger Cloud dashboard (accessible by clicking your user profile icon in the Dagger Cloud interface) and navigate to the **Configuration** tab. You should see the newly-created volumes listed and enabled.
-
-![Manage volumes](/img/current_docs/api/manage-volumes.png)
-
-You can create as many volumes as needed and manage them from the **Configuration** tab of your Dagger Cloud organization page.
-
-The volumes cache is shared across pipelines and runs, within the same Dagger Cloud organization. This implies that users in the same organization share the same volumes cache. The cache key for the volumes cache is `bucket / organization-name / volume-name`, where the volume name is set by the user using the Dagger API. Manifests of the layers cache are sent to Dagger Cloud to support merging layers between runs (not supported by Buildkit).
-
-:::important
-No cache data is shared across organizations. For more information on security, visit the [Dagger Trust Center](https://trust.dagger.io/).
-:::
-
-:::tip
-If you've configured cache volumes for the first time in a local development environment, call your Dagger function via the Dagger CLI and then run the command `docker container stop -t 300 "$(docker container list --filter 'name=^dagger-engine-*' -q)"`. This step ensures your new cache volumes populate to Dagger Cloud during the engine shutdown phase. You only need to do this the first time you use Dagger Cloud locally with cache volumes or when you add new cache volumes in your Dagger pipeline.
-:::

docs/current_docs/faq.mdx

@@ -136,15 +136,15 @@ Join us on [Discord](https://discord.com/invite/dagger-io), and ask your questio
 
 ### What is Dagger Cloud?
 
-Dagger Cloud complements the Dagger Engine with a production-grade control plane. Features of Dagger Cloud include pipeline visualization, operational insights, and distributed caching.
+Dagger Cloud complements the Dagger Engine with a production-grade control plane. Features of Dagger Cloud include pipeline visualization and operational insights.
 
 ### Is Dagger Cloud a hosting service for Dagger Engines?
 
 No, Dagger Cloud is a "bring your own compute" service. The Dagger Engine can run on a wide variety of machines, including most development and CI platforms. If the Dagger Engine can run on it, then Dagger Cloud supports it.
 
 ### Which CI providers does Dagger Cloud work with?
 
-Because the Dagger Engine can integrate seamlessly with practically any CI, there is no limit to the type and number of CI providers that Dagger Cloud can work with to provide Dagger pipeline visualization, operational insights, and distributed caching. Users report successfully leveraging Dagger with: GitLab, CircleCI, GitHub Actions, Jenkins,Tekton and many more.
+Because the Dagger Engine can integrate seamlessly with practically any CI, there is no limit to the type and number of CI providers that Dagger Cloud can work with to provide Dagger pipeline visualization and operational insights. Users report successfully leveraging Dagger with: GitLab, CircleCI, GitHub Actions, Jenkins,Tekton and many more.
 
 ### What is pipeline visualization?
 
@@ -154,26 +154,6 @@ Traces, a browser-based interface focused on tracing and debugging Dagger pipeli
 
 Dagger Cloud collects telemetry from all your organization's Dagger Engines, whether they run in development or CI, and presents it all to you in one place. This gives you a unique view on all pipelines, both pre-push and post-push.
 
-### What is distributed caching?
-
-One of Dagger's superpowers is that it caches everything. On a single machine (like a laptop or long-running server), caching "just works", because the same Dagger Engine writing to the cache is also reading from it. But in a multi-machine configuration (like an elastic CI cluster), things get more complicated because all machines are continuously producing and consuming large amounts of cache data. How do we get the right cache data to the right machine at the right time, without wasting compute, networking, or storage resources? This is a complex problem which requires a distributed caching service, to orchestrate the movement of data between all machines in the cluster, and a centralized storage service. Because Dagger Cloud receives telemetry from all Dagger Engines, it can model the state of the cluster and make optimal caching decisions. The more telemetry data it receives, the smarter it becomes.
-
-### Does distributed caching support ephemeral CI runners?
-
-Yes. Ephemeral runners, by definition, lack caching; the runner's local storage is purged when the runner is spun down. However, when your CI is connected to Dagger Cloud, these ephemeral runners gain all the benefits of a persistent shared cache.
-
-### Does Dagger Cloud store my cache data?
-
-Yes. For distributed caching to work, it requires two components: a centralized storage service and an orchestrator. Dagger Cloud provides both, in one integrated package.
-
-### Where does Dagger Cloud store my cache data?
-
-Dagger Cloud features a global data storage service spanning 26 regions across 3 cloud providers: AWS, Google Cloud Platform, and Cloudflare R2. The region closest to your compute is automatically selected.
-
-### Does Dagger Cloud support "bring your own storage" for distributed caching?
-
-The ability to "bring your own storage" is coming soon. Please reach out to us if this capability is needed for your organization.
-
 ### Why am I seeing `dagger functions` in the local trace list?
 
 All commands that require module initialization at an engine level will send telemetry to Dagger Cloud.

docs/current_docs/index.mdx

@@ -46,7 +46,7 @@ Dagger SDKs contain language-native code bindings for the Dagger API, enabling y
 The Dagger API is a unified interface for programming Dagger. It provides a set of powerful core types for creating and managing application delivery pipelines, including types for containers, artifacts, network services, and secrets. It uses GraphQL as its low-level language-agnostic framework, and can be accessed using any standard GraphQL client. In practice, interacting with the API directly is optional; typically, you will use a Dagger Function created with a type-safe Dagger SDK, or the Dagger CLI.
 
 ### Dagger Cloud
-Dagger Cloud complements Dagger with a production-grade control plane. Features of Dagger Cloud include pipeline visualization, operational insights, and distributed caching. Dagger Cloud provides a web interface to visualize each step of your pipeline, drill down to detailed logs, understand how long operations took to run, and whether operations were cached.
+Dagger Cloud complements Dagger with a production-grade control plane. Features of Dagger Cloud include pipeline visualization and operational insights. Dagger Cloud provides a web interface to visualize each step of your pipeline, drill down to detailed logs, understand how long operations took to run, and whether operations were cached.
 
 Dagger Cloud also collects telemetry from all your organization's Dagger Engines, whether they run in development or CI, and presents it all to you in one place. This gives you a unique view on all pipelines, both pre-push and post-push.
 

docs/current_docs/integrations/kubernetes.mdx

@@ -21,7 +21,7 @@ Running Dagger in Kubernetes is generally a good choice for:
 
 This section describe a few common architecture patterns to consider when setting up a Continuous Integration (CI) environment using Dagger on Kubernetes.
 
-#### Base pattern: Persistent nodes
+#### Persistent nodes
 
 The base pattern consists of persistent Kubernetes nodes with ephemeral CI runners.
 
@@ -45,34 +45,19 @@ In this architecture:
 
 ![Kubernetes base architecture](/img/current_docs/integrations/kubernetes-pattern-base.png)
 
-#### Optimization 1: Ephemeral, auto-scaled nodes
+#### Ephemeral, auto-scaled nodes
 
 The base architecture pattern described previously can be optimized by the addition of a *node auto-scaler*. This can automatically adjust the size of node groups based on the current workload. If there are a lot of CI jobs running, the auto-scaler can automatically add more runner nodes to the cluster to handle the increased workload. Conversely, if there are few jobs running, it can remove unnecessary runner nodes.
 
-This optimization reduces the total compute cost since runner nodes are added & removed based on the number of concurrent CI jobs.
+This optimization reduces the total compute cost since runner nodes are added & removed based on the number of concurrent CI jobs. The trade-off is that we lose all Dagger Engine state stored on the runner node that is de-provisioned. Configuring persistent volumes is worth considering if the platform supports it.
 
 In this architecture:
 
 - Kubernetes nodes provisioned on-demand start with a "clean" Dagger Engine containing no cached data.
-- Cached build artifacts from subsequent runs will persist only for the lifetime of the runner node.
+- Cached build artifacts from previous runs will be lost when the runner node gets de-provisioned (can be mitigated via persistent volumes).
 
 ![Kubernetes architecture with ephmeral nodes](/img/current_docs/integrations/kubernetes-pattern-ephemeral.png)
 
-#### Optimization 2: Shared Cloud Cache
-
-The previous pattern makes it possible to scale the Dagger deployment, but comes with the following trade-offs:
-
-1. Runner nodes are automatically de-provisioned when they are not needed. During de-provisioning, the Dagger Engines get deleted too. As a result, data and operations cached in previous runs will be deleted and subsequent runs will not benefit from previous runs. To resolve this, the cached data and operations are stored in a *cloud caching service* and made available to new Dagger Engines when they are provisioned.
-2. The deployment will only scale to a certain point, given that each Dagger Engine can only scale vertically to provide better performance. In order to make the system horizontally scalable, a caching service makes the same data and operations available to as many Dagger Engines as needed.
-
-In this architecture:
-
-- A shared cloud cache stores data from all Dagger Engines running in the cluster. This diagram below references Dagger's experimental Cloud Cache service.
-In the future, other providers will be availabe to store the cache outside Dagger's infrastructure.
-- Auto-provisioned nodes start with access to cached data of previous runs.
-
-![Kubernetes architecture with shared cache](/img/current_docs/integrations/kubernetes-pattern-cache.png)
-
 ### Recommendations
 
 When deploying Dagger on a Kubernetes cluster, it's important to understand the design constraints you're operating under, so you can optimize your configuration to suit your workload requirements. Here are two key recommendations.

docs/current_docs/integrations/snippets/circle-hello.yml

@@ -11,12 +11,6 @@ jobs:
       - run:
           name: Call Dagger Function
           command: dagger -m github.com/shykes/daggerverse/[email protected] call hello --greeting="bonjour" --name="monde"
-      # for ephemeral runners only: override the default docker stop timeout and
-      # give the Dagger Engine more time to push cache data to Dagger Cloud
-      - run:
-          name: Stop Dagger Engine
-          command: docker stop -t 300 $(docker ps --filter name="dagger-engine-*" -q)
-          when: always
 workflows:
   dagger:
     jobs:

docs/current_docs/integrations/snippets/circle-test-build.yml

@@ -12,12 +12,6 @@ jobs:
       - run:
           name: Test
           command: dagger -m github.com/kpenfound/dagger-modules/[email protected] call test --source=.
-      # for ephemeral runners only: override the default docker stop timeout and
-      # give the Dagger Engine more time to push cache data to Dagger Cloud
-      - run:
-          name: Stop Dagger Engine
-          command: docker stop -t 300 $(docker ps --filter name="dagger-engine-*" -q)
-          when: always
   build:
     docker:
       - image: cimg/base:2024.09
@@ -30,12 +24,6 @@ jobs:
       - run:
           name: Build
           command: dagger -m github.com/kpenfound/dagger-modules/[email protected] call build-container --source=https://github.com/golang/example#master:hello --args=. publish --address=ttl.sh/my-app-$RANDOM
-      # for ephemeral runners only: override the default docker stop timeout and
-      # give the Dagger Engine more time to push cache data to Dagger Cloud
-      - run:
-          name: Stop Dagger Engine
-          command: docker stop -t 300 $(docker ps --filter name="dagger-engine-*" -q)
-          when: always
 workflows:
   dagger:
     jobs:

docs/current_docs/integrations/snippets/gitlab-docker-hello.yml

@@ -22,6 +22,3 @@ hello:
   extends: [.dagger]
   script:
     - dagger -m github.com/shykes/daggerverse/[email protected] call hello --greeting="bonjour" --name="monde"
-    # for ephemeral runners only: override the default docker stop timeout and
-    # give the Dagger Engine more time to push cache data to Dagger Cloud
-    - docker stop -t 300 $(docker ps --filter name="dagger-engine-*" -q)