Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow configuration of submodule.fetchJobs and fetch.parallel #1569

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,17 @@ jobs:
- name: Verify submodules recursive
run: __test__/verify-submodules-recursive.sh

# Submodules fetchJobs
- name: Checkout submodules true
uses: ./
with:
ref: test-data/v2/submodule-ssh-url
path: submodules-true
submodules: recursive
submodules-fetch-jobs: "10"
- name: Verify submodules true
run: __test__/verify-submodules-with-jobs.sh

# Basic checkout using REST API
- name: Remove basic
if: runner.os != 'windows'
Expand Down Expand Up @@ -242,7 +253,7 @@ jobs:
path: basic
- name: Verify basic
run: __test__/verify-basic.sh --archive

test-git-container:
runs-on: ubuntu-latest
container: bitnami/git:latest
Expand Down Expand Up @@ -279,4 +290,4 @@ jobs:
- name: Fix Checkout v3
uses: actions/checkout@v3
with:
path: v3
path: v3
12 changes: 12 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,12 @@ Please refer to the [release page](https://github.com/actions/checkout/releases/
# Default: false
fetch-tags: ''

# Specifies the maximal number of fetch operations to be run in parallel at a time
# (submodules, or remotes when the --multiple option of git-fetch is in effect). A
# value of 0 will give some reasonable default. If unset, it defaults to 1.
# Default: 1
fetch-parallel: ''

# Whether to show progress status output when fetching.
# Default: true
show-progress: ''
Expand All @@ -111,6 +117,12 @@ Please refer to the [release page](https://github.com/actions/checkout/releases/
# Default: false
submodules: ''

# Specifies how many submodules are fetched/cloned at the same time. A positive
# integer allows up to that number of submodules fetched in parallel. A value of 0
# will give some reasonable default. If unset, it defaults to 1.
# Default: 1
submodules-fetch-jobs: ''

# Add repository path as safe.directory for Git global config by running `git
# config --global --add safe.directory <path>`
# Default: true
Expand Down
2 changes: 2 additions & 0 deletions __test__/git-auth-helper.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -807,10 +807,12 @@ async function setup(testName: string): Promise<void> {
sparseCheckoutConeMode: true,
fetchDepth: 1,
fetchTags: false,
fetchParallel: 1,
showProgress: true,
lfs: false,
submodules: false,
nestedSubmodules: false,
submodulesFetchJobs: 1,
persistCredentials: true,
ref: 'refs/heads/main',
repositoryName: 'my-repo',
Expand Down
33 changes: 33 additions & 0 deletions __test__/verify-submodules-with-jobs.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#!/bin/bash

if [ ! -f "./submodules-recursive/regular-file.txt" ]; then
echo "Expected regular file does not exist"
exit 1
fi

if [ ! -f "./submodules-recursive/submodule-level-1/submodule-file.txt" ]; then
echo "Expected submodule file does not exist"
exit 1
fi

if [ ! -f "./submodules-recursive/submodule-level-1/submodule-level-2/nested-submodule-file.txt" ]; then
echo "Expected nested submodule file does not exists"
exit 1
fi

echo "Testing fetchJobs exists"
git config --local --get-regexp submodules.fetchJobs | grep 10
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the right way to test this?

if [ "$?" != "0" ]; then
echo "Failed to validate fetchJobs configuration"
exit 1
fi

echo "Testing persisted credential"
pushd ./submodules-recursive/submodule-level-1/submodule-level-2
git config --local --name-only --get-regexp http.+extraheader && git fetch
if [ "$?" != "0" ]; then
echo "Failed to validate persisted credential"
popd
exit 1
fi
popd
209 changes: 109 additions & 100 deletions action.yml
Original file line number Diff line number Diff line change
@@ -1,100 +1,109 @@
name: 'Checkout'
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know why this shows this big diff, I thought it could be formatting but it doesnt seem like it.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you want to see the reason, I would suggest to run git show --word-diff --word-diff-regex=. 2bd5ef5487b953b5db186c7a68dcab4e4a8fe3fc -- action.yml - you will see that it had to do with the line endings (maybe something like e.g. https://docs.github.com/en/get-started/getting-started-with-git/configuring-git-to-handle-line-endings can be of help?)

description: 'Checkout a Git repository at a particular version'
inputs:
repository:
description: 'Repository name with owner. For example, actions/checkout'
default: ${{ github.repository }}
ref:
description: >
The branch, tag or SHA to checkout. When checking out the repository that
triggered a workflow, this defaults to the reference or SHA for that
event. Otherwise, uses the default branch.
token:
description: >
Personal access token (PAT) used to fetch the repository. The PAT is configured
with the local git config, which enables your scripts to run authenticated git
commands. The post-job step removes the PAT.


We recommend using a service account with the least permissions necessary.
Also when generating a new PAT, select the least scopes necessary.


[Learn more about creating and using encrypted secrets](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/creating-and-using-encrypted-secrets)
default: ${{ github.token }}
ssh-key:
description: >
SSH key used to fetch the repository. The SSH key is configured with the local
git config, which enables your scripts to run authenticated git commands.
The post-job step removes the SSH key.


We recommend using a service account with the least permissions necessary.


[Learn more about creating and using
encrypted secrets](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/creating-and-using-encrypted-secrets)
ssh-known-hosts:
description: >
Known hosts in addition to the user and global host key database. The public
SSH keys for a host may be obtained using the utility `ssh-keyscan`. For example,
`ssh-keyscan github.com`. The public key for github.com is always implicitly added.
ssh-strict:
description: >
Whether to perform strict host key checking. When true, adds the options `StrictHostKeyChecking=yes`
and `CheckHostIP=no` to the SSH command line. Use the input `ssh-known-hosts` to
configure additional hosts.
default: true
persist-credentials:
description: 'Whether to configure the token or SSH key with the local git config'
default: true
path:
description: 'Relative path under $GITHUB_WORKSPACE to place the repository'
clean:
description: 'Whether to execute `git clean -ffdx && git reset --hard HEAD` before fetching'
default: true
filter:
description: >
Partially clone against a given filter.
Overrides sparse-checkout if set.
default: null
sparse-checkout:
description: >
Do a sparse checkout on given patterns.
Each pattern should be separated with new lines.
default: null
sparse-checkout-cone-mode:
description: >
Specifies whether to use cone-mode when doing a sparse checkout.
default: true
fetch-depth:
description: 'Number of commits to fetch. 0 indicates all history for all branches and tags.'
default: 1
fetch-tags:
description: 'Whether to fetch tags, even if fetch-depth > 0.'
default: false
show-progress:
description: 'Whether to show progress status output when fetching.'
default: true
lfs:
description: 'Whether to download Git-LFS files'
default: false
submodules:
description: >
Whether to checkout submodules: `true` to checkout submodules or `recursive` to
recursively checkout submodules.


When the `ssh-key` input is not provided, SSH URLs beginning with `[email protected]:` are
converted to HTTPS.
default: false
set-safe-directory:
description: Add repository path as safe.directory for Git global config by running `git config --global --add safe.directory <path>`
default: true
github-server-url:
description: The base URL for the GitHub instance that you are trying to clone from, will use environment defaults to fetch from the same instance that the workflow is running from unless specified. Example URLs are https://github.com or https://my-ghes-server.example.com
required: false
runs:
using: node20
main: dist/index.js
post: dist/index.js
name: 'Checkout'
description: 'Checkout a Git repository at a particular version'
inputs:
repository:
description: 'Repository name with owner. For example, actions/checkout'
default: ${{ github.repository }}
ref:
description: >
The branch, tag or SHA to checkout. When checking out the repository that
triggered a workflow, this defaults to the reference or SHA for that
event. Otherwise, uses the default branch.
token:
description: >
Personal access token (PAT) used to fetch the repository. The PAT is configured
with the local git config, which enables your scripts to run authenticated git
commands. The post-job step removes the PAT.


We recommend using a service account with the least permissions necessary.
Also when generating a new PAT, select the least scopes necessary.


[Learn more about creating and using encrypted secrets](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/creating-and-using-encrypted-secrets)
default: ${{ github.token }}
ssh-key:
description: >
SSH key used to fetch the repository. The SSH key is configured with the local
git config, which enables your scripts to run authenticated git commands.
The post-job step removes the SSH key.


We recommend using a service account with the least permissions necessary.


[Learn more about creating and using
encrypted secrets](https://help.github.com/en/actions/automating-your-workflow-with-github-actions/creating-and-using-encrypted-secrets)
ssh-known-hosts:
description: >
Known hosts in addition to the user and global host key database. The public
SSH keys for a host may be obtained using the utility `ssh-keyscan`. For example,
`ssh-keyscan github.com`. The public key for github.com is always implicitly added.
ssh-strict:
description: >
Whether to perform strict host key checking. When true, adds the options `StrictHostKeyChecking=yes`
and `CheckHostIP=no` to the SSH command line. Use the input `ssh-known-hosts` to
configure additional hosts.
default: true
persist-credentials:
description: 'Whether to configure the token or SSH key with the local git config'
default: true
path:
description: 'Relative path under $GITHUB_WORKSPACE to place the repository'
clean:
description: 'Whether to execute `git clean -ffdx && git reset --hard HEAD` before fetching'
default: true
filter:
description: >
Partially clone against a given filter.
Overrides sparse-checkout if set.
default: null
sparse-checkout:
description: >
Do a sparse checkout on given patterns.
Each pattern should be separated with new lines.
default: null
sparse-checkout-cone-mode:
description: >
Specifies whether to use cone-mode when doing a sparse checkout.
default: true
fetch-depth:
description: 'Number of commits to fetch. 0 indicates all history for all branches and tags.'
default: 1
fetch-tags:
description: 'Whether to fetch tags, even if fetch-depth > 0.'
default: false
fetch-parallel:
description: >
Specifies the maximal number of fetch operations to be run in parallel at a time (submodules, or remotes when the --multiple option of git-fetch is in effect).
A value of 0 will give some reasonable default. If unset, it defaults to 1.
default: 1
show-progress:
description: 'Whether to show progress status output when fetching.'
default: true
lfs:
description: 'Whether to download Git-LFS files'
default: false
submodules:
description: >
Whether to checkout submodules: `true` to checkout submodules or `recursive` to
recursively checkout submodules.


When the `ssh-key` input is not provided, SSH URLs beginning with `[email protected]:` are
converted to HTTPS.
default: false
submodules-fetch-jobs:
description: >
Specifies how many submodules are fetched/cloned at the same time. A positive integer allows up to that number of submodules fetched in parallel. A value of 0 will give some reasonable default. If unset, it defaults to 1.
default: 1
set-safe-directory:
description: Add repository path as safe.directory for Git global config by running `git config --global --add safe.directory <path>`
default: true
github-server-url:
description: The base URL for the GitHub instance that you are trying to clone from, will use environment defaults to fetch from the same instance that the workflow is running from unless specified. Example URLs are https://github.com or https://my-ghes-server.example.com
required: false
runs:
using: node20
main: dist/index.js
post: dist/index.js
17 changes: 16 additions & 1 deletion dist/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -1243,6 +1243,7 @@ function getSource(settings) {
}
// Fetch
core.startGroup('Fetching the repository');
yield git.config('fetch.parallel', settings.fetchParallel.toString(), true);
const fetchOptions = {};
if (settings.filter) {
fetchOptions.filter = settings.filter;
Expand Down Expand Up @@ -1304,6 +1305,7 @@ function getSource(settings) {
core.endGroup();
// Checkout submodules
core.startGroup('Fetching submodules');
yield git.config('submodule.fetchJobs', settings.submodulesFetchJobs.toString(), true);
yield git.submoduleSync(settings.nestedSubmodules);
yield git.submoduleUpdate(settings.fetchDepth, settings.nestedSubmodules);
yield git.submoduleForeach('git config --local gc.auto 0', settings.nestedSubmodules);
Expand Down Expand Up @@ -1752,6 +1754,12 @@ function getInputs() {
result.fetchTags =
(core.getInput('fetch-tags') || 'false').toUpperCase() === 'TRUE';
core.debug(`fetch tags = ${result.fetchTags}`);
// Fetch tags
result.fetchParallel = Math.floor(Number(core.getInput('fetch-parallel') || '1'));
if (isNaN(result.fetchParallel) || result.fetchParallel < 0) {
result.fetchParallel = 0;
}
core.debug(`fetch parallel = ${result.fetchParallel}`);
// Show fetch progress
result.showProgress =
(core.getInput('show-progress') || 'true').toUpperCase() === 'TRUE';
Expand All @@ -1770,10 +1778,17 @@ function getInputs() {
else if (submodulesString == 'TRUE') {
result.submodules = true;
}
result.submodulesFetchJobs = Math.floor(Number(core.getInput('submodules-fetch-jobs') || '1'));
if (isNaN(result.submodulesFetchJobs) || result.submodulesFetchJobs < 0) {
result.submodulesFetchJobs = 0;
}
core.debug(`submodules = ${result.submodules}`);
core.debug(`recursive submodules = ${result.nestedSubmodules}`);
core.debug(`submodules submodules-fetch-jobs = ${result.submodulesFetchJobs}`);
// Auth token
result.authToken = core.getInput('token', { required: true });
result.authToken = core.getInput('token', {
required: true
});
// SSH
result.sshKey = core.getInput('ssh-key');
result.sshKnownHosts = core.getInput('ssh-known-hosts');
Expand Down
6 changes: 6 additions & 0 deletions src/git-source-provider.ts
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,7 @@ export async function getSource(settings: IGitSourceSettings): Promise<void> {

// Fetch
core.startGroup('Fetching the repository')
await git.config('fetch.parallel', settings.fetchParallel.toString(), true)
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the right place to set it?

const fetchOptions: {
filter?: string
fetchDepth?: number
Expand Down Expand Up @@ -232,6 +233,11 @@ export async function getSource(settings: IGitSourceSettings): Promise<void> {

// Checkout submodules
core.startGroup('Fetching submodules')
await git.config(
'submodule.fetchJobs',
settings.submodulesFetchJobs.toString(),
true
)
await git.submoduleSync(settings.nestedSubmodules)
await git.submoduleUpdate(settings.fetchDepth, settings.nestedSubmodules)
await git.submoduleForeach(
Expand Down
Loading