Skip to content

Commit

Permalink
SNOW-1301084: Migrate jupyter notebook test workflow (VisualizingTaxi…
Browse files Browse the repository at this point in the history
…TripData nb) (#1451)

Please answer these questions before submitting your pull requests.
Thanks!

1. What GitHub issue is this PR addressing? Make sure that there is an
accompanying issue to your PR.

   Fixes SNOW-1301084

2. Fill out the following pre-review checklist:

- [ ] I am adding a new automated test(s) to verify correctness of my
new code
   - [ ] I am adding new logging messages
   - [ ] I am adding a new telemetry message
   - [ ] I am adding new credentials
   - [ ] I am adding a new dependency

3. Please describe how your code solves the related issue.

Adds VisualizingTaxiTripData.ipynb notebook to tests/notebooks/modin and
runs the notebook in daily github workflow using nbmake via
`daily_jupyter_nb_test.yml`

---------

Signed-off-by: Labanya Mukhopadhyay <[email protected]>
  • Loading branch information
sfc-gh-lmukhopadhyay authored May 1, 2024
1 parent fdf7556 commit 476e2c9
Show file tree
Hide file tree
Showing 5 changed files with 15,829 additions and 0 deletions.
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
/tests/unit/modin/ @snowflakedb/snowpandas
/docs/modin_api_coverage/ @snowflakedb/snowpandas
/docs/source/modin/ @snowflakedb/snowpandas
/tests/notebooks/ @snowflakedb/snowpandas
/.github/ @snowflakedb/snowpandas @snowflakedb/snowpark-python-api-reviewers
/scripts/ @snowflakedb/snowpandas @snowflakedb/snowpark-python-api-reviewers
setup.py @snowflakedb/snowpandas @snowflakedb/snowpark-python-api-reviewers
Expand Down
108 changes: 108 additions & 0 deletions .github/workflows/daily_jupyter_nb_test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
name: Snowpark pandas Notebook Tests

on:
schedule:
- cron: "0 16 * * *" # Runs 16:00 UTC on every day
workflow_dispatch:
inputs:
logLevel:
default: warning
description: "Log level"
required: true

jobs:
lint:
name: Check linting
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.9'
- name: Display Python version
run: python -c "import sys; import os; print(\"\n\".join(os.environ[\"PATH\"].split(os.pathsep))); print(sys.version); print(sys.executable);"
- name: Upgrade setuptools and pip
run: python -m pip install -U setuptools pip
- name: Install tox
run: python -m pip install tox
- name: Run fix_lint
run: python -m tox -e fix_lint

build:
needs: lint
name: Build Wheel File
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: 3.9
- name: Upgrade setuptools and pip
run: python -m pip install -U setuptools pip wheel
- name: Generate wheel
run: python -m pip wheel -v -w dist --no-deps .
- name: Show wheels generated
run: ls -lh dist
- uses: actions/upload-artifact@v1
with:
name: wheel
path: dist/
test:
name: Test modin-${{ matrix.os.download_name }}-${{ matrix.python-version }}-${{ matrix.cloud-provider }}
needs: build
runs-on: ${{ matrix.os.image_name }}
strategy:
fail-fast: false
matrix:
os:
- image_name: ubuntu-latest
download_name: linux
- image_name: macos-latest
download_name: macos
- image_name: windows-latest
download_name: windows
python-version: ["3.9", "3.10", "3.11"]
cloud-provider: [aws, azure, gcp]
steps:
- name: Checkout Code
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Display Python version
run: python -c "import sys; print(sys.version)"
- name: Decrypt parameters.py
shell: bash
run: .github/scripts/decrypt_parameters.sh
env:
PARAMETER_PASSWORD: ${{ secrets.PARAMETER_PASSWORD }}
CLOUD_PROVIDER: ${{ matrix.cloud-provider }}
- name: Download wheel(s)
uses: actions/download-artifact@v3
with:
name: wheel
path: dist
- name: Show wheels downloaded
run: ls -lh dist
shell: bash
- name: Upgrade setuptools, pip and wheel
run: python -m pip install -U setuptools pip wheel
- name: Install project
run: python -m pip install -e ".[modin-development]"
- name: Install test requirements
run: python -m pip install -r tests/notebooks/test_requirements.txt
- name: Run notebook tests
run: |
python -c "import sys; sys.path.append('../')"
python -m pytest -ra --nbmake tests/notebooks/modin
env:
PYTHON_VERSION: ${{ matrix.python-version }}
cloud_provider: ${{ matrix.cloud-provider }}
PYTEST_ADDOPTS: --color=yes --tb=short
# Specify SNOWFLAKE_IS_PYTHON_RUNTIME_TEST: 1 when adding >= python3.11 with no server-side support
# For example, see https://github.com/snowflakedb/snowpark-python/pull/681
shell: bash
714 changes: 714 additions & 0 deletions tests/notebooks/modin/VisualizingTaxiTripData.ipynb

Large diffs are not rendered by default.

Loading

0 comments on commit 476e2c9

Please sign in to comment.