Skip to content

Commit

Permalink
fix: invalidate caches when fetch, docker, or toolchain tasks change (#…
Browse files Browse the repository at this point in the history
…761)

This was an early hack I put in to avoid, eg: retraining entire models because a comment changed in a Dockerfile. We still want to avoid this sort of thing, but these days we do so explicitly with `previous_group_ids` and other techniques in the training config.

Getting rid of this will improve our ability to cache issues in PRs. Eg: #738 had to be tricked into running things on a second iteration that only changed a Dockerfile.
  • Loading branch information
bhearsum authored Jul 25, 2024
1 parent c01034e commit 27f95c9
Showing 1 changed file with 0 additions and 12 deletions.
12 changes: 0 additions & 12 deletions taskcluster/translations_taskgraph/transforms/cached_tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,6 @@
# It exists because there are two features that we need that are missing upstream:
# - The ability to influence the cache digest from parameters.
# (https://github.com/taskcluster/taskgraph/issues/391)
# - The ability to avoid adding some upstream tasks to the cache digest (which
# allows us to avoid rebuilding the world when, eg: we upgrade a Docker base
# image version). No upstream issue is filed for this, because for the vast
# majority of use cases it is better to take the rebuilds. Our use case is
# an exception, because of the truly massive amount of time it takes to
# train a model.


import itertools
Expand All @@ -29,8 +23,6 @@
transforms = TransformSequence()


DONT_INVALIDATE_KINDS = ["docker-image", "fetch", "toolchain"]

SCHEMA = Schema(
{
Required("attributes"): {
Expand Down Expand Up @@ -122,10 +114,6 @@ def cache_task(config, tasks):

dependency_digests = []
for p in task.get("dependencies", {}).values():
# Here in Translations, we explicit do _not_ invalidate cached
# training jobs when non-training jobs are invalidated.
if any([kind in p for kind in DONT_INVALIDATE_KINDS]):
continue
if p in digests:
dependency_digests.append(digests[p])
else:
Expand Down

0 comments on commit 27f95c9

Please sign in to comment.