[artifact] only inspect artifact contents

[chrisroberts]

Description

Artifact inspection would inspect the alloc directory for symlinks
pointing outside of the alloc directory. This can be problematic
for drivers like the raw exec driver which provide the root system
and will likely include symlinks.

Inspecting only the artifact after being fetched can be difficult if
the artifact is fetched to a pre-existing destination as it would
require discerning between new and old paths. Instead, if an artifact
is to be inspected, it is fetched to a temporary location. The isolated
artifact is then inspected before being moved to its final destination.

The behavior of the artifact getter will be inconsistent when artifact
inspection is enabled versus when it is disabled. This is due to the
behavior difference of go-getter with different sources. Because
go-getter does not behave consistently across different source types,
there will be situations where fetching errors may be encountered
when inspection is disabled but fetching is successful when enabled.

Testing & Reproduction steps

Reproduction steps are described in #26865

Links

Fixes #26865

Contributor Checklist

• Changelog Entry If this PR changes user-facing behavior, please generate and add a
  changelog entry using the make cl command.
• Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
  ensure regressions will be caught.
• Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
  and job configuration, please update the Nomad website documentation to reflect this. Refer to
  the website README for docs guidelines. Please also consider whether the
  change requires notes within the upgrade guide.

Reviewer Checklist

• Backport Labels Please add the correct backport labels as described by the internal
  backporting document.
• Commit Type Ensure the correct merge method is selected which should be "squash and merge"
  in the majority of situations. The main exceptions are long-lived feature branches or merges where
  history should be preserved.
• Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
  within the public repository.

• If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

[mismithhisler]

Is the purpose of this merge so that we don't error or override contents that may already exist in the finalDest?

[chrisroberts]

So go-getter has two behaviors based on the source used:

1. unpack/place artifact files within an existing directory structure (overwriting files on collision)
2. delete destination directory and unpack/place artifact files

This is using the first approach (and gets into some of the behavior inconsistencies noted in the description).

[tgross]

If I'm understanding the PR description correctly, this is going to change the behavior of those sources that use option 2, isn't it?

The artifact.destination is placed relative to the task working directory. So couldn't we potentially solve this by having the "root" of the check be the task working directory instead of the allocdir?

[chrisroberts]

Yes, that is correct that it will change the behavior of those sources using option 2, but only when artifact inspection is triggered. On platforms where landlock is available and is not disabled, artifact inspection will not be performed.

Isolating the check to the task directory won't be sufficient when using the exec driver, as an example. In that case the task directory will include things from the host (like bin/) which will contain symlinks.

[tgross]

Yes, that is correct that it will change the behavior of those sources using option 2, but only when artifact inspection is triggered. On platforms where landlock is available and is not disabled, artifact inspection will not be performed.

I think that's an ok tradeoff, but we should document that behavior in the upgrade guide, because that's going to break somebody's jobspecs somewhere.

[aimeeu]

@chrisroberts Because of this behavior change, do we need to update https://developer.hashicorp.com/nomad/docs/configuration/client#disable_filesystem_isolation to something like:

- `disable_filesystem_isolation` `(bool: false)` - Specifies whether to disable
  filesystem isolation for artifact downloads. Applies only to systems where
  filesystem isolation via [landlock] is possible (Linux kernel 5.13+).When
  applied, Nomad merges the unpacked contents of the artifact into the
  destination path if it already exists. Depending on the remote source being
  used, this may be different behavior than what happens  when the artifact is
  unpacked without inspection.

[tgross]

Can we call out the specific remote sources where this is known? We want users to be able to read this and have actionable insights on whether they have to go updating their jobs.