Skip to content
This repository has been archived by the owner on Aug 16, 2024. It is now read-only.

Commit

Permalink
Upload sccache stats (pytorch#65582)
Browse files Browse the repository at this point in the history
Summary:
This adds some tracking to metrics.pytorch.org for sccache build stats per environment

Pull Request resolved: pytorch#65582

Reviewed By: malfet, zhouzhuojie, janeyx99

Differential Revision: D31160761

Pulled By: driazati

fbshipit-source-id: a497918bafbe610a51c92a9139684cd3efe670d3
  • Loading branch information
driazati authored and facebook-github-bot committed Sep 27, 2021
1 parent ea546e2 commit 811601e
Show file tree
Hide file tree
Showing 22 changed files with 310 additions and 127 deletions.
23 changes: 13 additions & 10 deletions .github/templates/linux_ci_workflow.yml.j2
Original file line number Diff line number Diff line change
Expand Up @@ -38,14 +38,17 @@ env:
XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
TORCH_CUDA_ARCH_LIST: 5.2
IN_CI: 1
IS_GHA: 1
# This is used for the phase of adding wheel tests only, will be removed once completed
IN_WHEEL_TEST: 1
# Used for custom_opertor, jit_hooks, custom_backend, see .jenkins/pytorch/build.sh
CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts
ALPINE_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine"
PR_LABELS: ${{ toJson(github.event.pull_request.labels.*.name) }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

AWS_DEFAULT_REGION: us-east-1
CIRCLE_PR_NUMBER: ${{ github.event.pull_request.number }}
CIRCLE_SHA1: ${{ github.event.pull_request.head.sha || github.sha }}
!{{ common.concurrency(build_environment) }}

jobs:
Expand Down Expand Up @@ -133,13 +136,22 @@ jobs:
- name: Pull Docker image
run: |
!{{ common.pull_docker("${DOCKER_IMAGE}") }}
!{{ common.parse_ref() }}
- name: Build
env:
CIRCLE_BRANCH: ${{ steps.parse-ref.outputs.branch }}
run: |
# detached container should get cleaned up by teardown_ec2_linux
container_name=$(docker run \
-e BUILD_ENVIRONMENT \
-e JOB_BASE_NAME \
-e MAX_JOBS="$(nproc --ignore=2)" \
-e AWS_DEFAULT_REGION \
-e IS_GHA \
-e CIRCLE_PR_NUMBER \
-e CIRCLE_SHA1 \
-e CIRCLE_BRANCH \
-e GITHUB_RUN_ID \
-e SCCACHE_BUCKET \
-e XLA_CLANG_CACHE_S3_BUCKET_NAME \
-e CUSTOM_TEST_ARTIFACT_BUILD_DIR \
Expand All @@ -158,17 +170,12 @@ jobs:
"${DOCKER_IMAGE}"
)
docker exec -t "${container_name}" sh -c 'sudo chown -R jenkins . && .jenkins/pytorch/build.sh'
!{{ common.parse_ref() }}
- name: Display and upload binary build size statistics (Click Me)
# temporary hack: set CIRCLE_* vars, until we update
# tools/stats/print_test_stats.py to natively support GitHub Actions
env:
AWS_DEFAULT_REGION: us-east-1
IS_GHA: 1
SCRIBE_GRAPHQL_ACCESS_TOKEN: ${{ secrets.SCRIBE_GRAPHQL_ACCESS_TOKEN }}
CIRCLE_BRANCH: ${{ steps.parse-ref.outputs.branch }}
CIRCLE_PR_NUMBER: ${{ github.event.pull_request.number }}
CIRCLE_SHA1: ${{ github.event.pull_request.head.sha || github.sha }}
CIRCLE_TAG: ${{ steps.parse-ref.outputs.tag }}
CIRCLE_WORKFLOW_ID: '${{ github.run_id }}_${{ github.run_number }}'
run: |
Expand Down Expand Up @@ -292,11 +299,7 @@ jobs:
- name: Test
env:
PR_NUMBER: ${{ github.event.pull_request.number }}
IS_GHA: 1
CIRCLE_BRANCH: ${{ steps.parse-ref.outputs.branch }}
CIRCLE_PR_NUMBER: ${{ github.event.pull_request.number }}
CIRCLE_SHA1: ${{ github.event.pull_request.head.sha || github.sha }}
AWS_DEFAULT_REGION: us-east-1
# Time out the test phase after 3.5 hours
timeout-minutes: 210
run: |
Expand Down
6 changes: 6 additions & 0 deletions .github/templates/windows_ci_workflow.yml.j2
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ env:
BUILD_WHEEL: 1
CUDA_VERSION: "!{{ cuda_version }}"
IN_CI: 1
IS_GHA: 1
INSTALL_WINDOWS_SDK: 1
PYTHON_VERSION: "3.8"
PR_LABELS: ${{ toJson(github.event.pull_request.labels.*.name) }}
Expand All @@ -55,6 +56,9 @@ env:
VC_YEAR: "2019"
ALPINE_IMAGE: "308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine"
no_proxy: !{{ common.squid_no_proxy }}
AWS_DEFAULT_REGION: us-east-1
CIRCLE_PR_NUMBER: ${{ github.event.pull_request.number }}
CIRCLE_SHA1: ${{ github.event.pull_request.head.sha || github.sha }}
{%- if cuda_version != "cpu" %}
TORCH_CUDA_ARCH_LIST: "7.0"
USE_CUDA: 1
Expand Down Expand Up @@ -110,10 +114,12 @@ jobs:
run: |
.circleci/scripts/windows_cudnn_install.sh
{%- endif %}
!{{ common.parse_ref() }}
- name: Build
shell: bash
env:
PYTORCH_FINAL_PACKAGE_DIR: /c/${{ github.run_id }}/build-results/
CIRCLE_BRANCH: ${{ steps.parse-ref.outputs.branch }}
run: |
.jenkins/pytorch/win-build.sh
# Upload to github so that people can click and download artifacts
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

27 changes: 15 additions & 12 deletions .github/workflows/generated-linux-bionic-cuda10.2-py3.9-gcc7.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

27 changes: 15 additions & 12 deletions .github/workflows/generated-linux-bionic-py3.6-clang9.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 811601e

Please sign in to comment.