Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: update appveyor image #6278

Merged
merged 73 commits into from
Dec 5, 2023
Merged
Show file tree
Hide file tree
Changes from 62 commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
529c697
update newer ubuntu image
mndeveci Nov 16, 2023
0bed432
update newer ubuntu image
mndeveci Nov 16, 2023
7d6e1b5
add py3.12
mndeveci Nov 16, 2023
e393044
add py3.12
mndeveci Nov 16, 2023
abdc298
install py3.12
mndeveci Nov 16, 2023
fde40e9
check what is in 3.12 dir
mndeveci Nov 16, 2023
e11d0ff
check contents
mndeveci Nov 16, 2023
eca3846
check jdk21
mndeveci Nov 16, 2023
ff4a62d
add py3.12 to the path
mndeveci Nov 16, 2023
a1dbe93
install jdk21
mndeveci Nov 16, 2023
86abe21
install jdk21
mndeveci Nov 16, 2023
e4a5e8e
install jdk21
mndeveci Nov 16, 2023
34e1c28
check default java & javac version
mndeveci Nov 16, 2023
8030cf5
versions
mndeveci Nov 16, 2023
0a5fef1
update alternatives
mndeveci Nov 16, 2023
d8297a2
echo path
mndeveci Nov 16, 2023
4f7eb98
get choco logs
mndeveci Nov 16, 2023
c578f33
update java_home and path
mndeveci Nov 16, 2023
92dcae4
run it with powershell
mndeveci Nov 16, 2023
5a27df8
gradle version
mndeveci Nov 16, 2023
408cced
refresh env
mndeveci Nov 16, 2023
539d2ad
update mvn & gradle
mndeveci Nov 16, 2023
9651748
move java installation part
mndeveci Nov 16, 2023
c4f9d48
upgrade choco first
mndeveci Nov 16, 2023
89c78ba
uninstall & re-install gradle
mndeveci Nov 16, 2023
46f28bb
update linux binary
mndeveci Nov 16, 2023
783a2e3
update windows binary configuration
mndeveci Nov 16, 2023
7d554df
Merge branch 'develop' into update_appveyor_image
hawflau Nov 17, 2023
d597afe
Use Ubuntu2004 image
hawflau Nov 17, 2023
c47c7d1
Update appveyor-ubuntu.yml - install python3.12
hawflau Nov 17, 2023
5fee11f
Update appveyor-ubuntu.yml
hawflau Nov 17, 2023
0cd3ab0
Update appveyor-ubuntu.yml
hawflau Nov 17, 2023
0061ce2
Update appveyor-ubuntu.yml
hawflau Nov 17, 2023
6e0cc9b
Apply suggestions from code review
hawflau Nov 17, 2023
9a9fd87
Update appveyor-windows.yml
hawflau Nov 17, 2023
b38bb24
Apply suggestions from code review
hawflau Nov 17, 2023
8eaa029
Update appveyor-ubuntu.yml
hawflau Nov 17, 2023
76426f9
Apply suggestions from code review
hawflau Nov 17, 2023
ad3317d
Update appveyor-windows.yml
hawflau Nov 17, 2023
b70a34b
Update appveyor-windows.yml
hawflau Nov 17, 2023
76e88ff
set python version back to 3.8
mndeveci Nov 21, 2023
6a01132
install pip through bootstrap.pypa.io
mndeveci Nov 21, 2023
cd73918
Merge branch 'develop' into update_appveyor_image
mndeveci Nov 21, 2023
1030237
test with new ubuntu image
mndeveci Nov 21, 2023
0305dec
Merge branch 'develop' into update_appveyor_image
mndeveci Nov 21, 2023
9abe2d0
Remove provided rust integ tests and change python3 to python (#6311)
hnnasit Nov 21, 2023
3e7695f
install jdk8 as well
mndeveci Nov 21, 2023
6326d1e
do not dictate jdk tooling version but set target java version
mndeveci Nov 21, 2023
f949764
don't install jdk8
mndeveci Nov 21, 2023
8e5bec6
Create a separate Makefile for provided Docker integ tests
hnnasit Nov 22, 2023
71089f4
fix ruby issue
mndeveci Nov 22, 2023
76186d1
Remove go1.x debug test as due to different GLIB version, this is not…
hnnasit Nov 22, 2023
e109fcf
remove refreshenv and updating docker engine
mndeveci Nov 23, 2023
1ed2f06
skip gradlew and gradle-kotlin containerized tests for earlier java v…
mndeveci Nov 23, 2023
1c36f70
run other tests sequentially
mndeveci Nov 23, 2023
bc3148f
Merge branch 'develop' into update_appveyor_image
mndeveci Nov 23, 2023
6189e4f
fix test skipping for java arm64
mndeveci Nov 23, 2023
31bfa4e
Merge branch 'develop' into update_appveyor_image
mndeveci Nov 27, 2023
3251f64
aligning changes between regular tests and binary ones
mndeveci Nov 27, 2023
b560229
fix formatting
mndeveci Nov 27, 2023
6c59a06
use provided.al2 for golang terraform tests
mndeveci Nov 27, 2023
83a6ecc
(windows) refresh env after installing correttojdk 21
hawflau Nov 28, 2023
a0654bc
(windows) refreshenv breaking appveyor commands, setting PATH and JAV…
hawflau Nov 28, 2023
19ff783
skip tests involving nodejs20 container in Windows
hawflau Nov 28, 2023
39eebdb
fix golang terraform tests
mndeveci Nov 29, 2023
82d5397
run upgrade gradle instead of doing uninstall & install
mndeveci Nov 29, 2023
e7cdea5
bump timeout of tf golang function
mndeveci Dec 1, 2023
9cf0af3
Merge branch 'develop' into update_appveyor_image
mndeveci Dec 1, 2023
99424c1
update tests to skip certain tests if runtime is not supported by ins…
hawflau Dec 2, 2023
1818c85
fix typing
hawflau Dec 2, 2023
4e5740e
add missing changes and update formatting to match with regular linux…
mndeveci Dec 4, 2023
3815d6f
add missing lines and update formatting with regular windows config
mndeveci Dec 4, 2023
4bf368d
Merge branch 'develop' into update_appveyor_image
mndeveci Dec 4, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 16 additions & 10 deletions appveyor-linux-binary.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
version: 1.0.{build}
image:
- Ubuntu2004
- Ubuntu2204
hawflau marked this conversation as resolved.
Show resolved Hide resolved

configuration:
- BuildIntegTesting
Expand Down Expand Up @@ -33,18 +33,23 @@ install:
# AppVeyor's apt-get cache might be outdated, and the package could potentially be 404.
- sh: "sudo apt-get update --allow-releaseinfo-change"

# install coretto 21
- sh: wget -O - https://apt.corretto.aws/corretto.key | sudo gpg --dearmor -o /usr/share/keyrings/corretto-keyring.gpg
- sh: echo "deb [signed-by=/usr/share/keyrings/corretto-keyring.gpg] https://apt.corretto.aws stable main" | sudo tee /etc/apt/sources.list.d/corretto.list
- sh: sudo apt-get update; sudo apt-get install -y java-21-amazon-corretto-jdk
- sh: JAVA_HOME=/usr/lib/jvm/java-21-amazon-corretto
- sh: PATH=$JAVA_HOME/bin:$PATH
- sh: java --version
- sh: javac --version

- sh: "gvm use go1.19"
- sh: "echo $PATH"
- sh: "ls /usr/"
# install latest maven which is compatible with jdk17
- sh: "sudo apt-get -y remove maven"
- sh: "wget https://dlcdn.apache.org/maven/maven-3/3.8.8/binaries/apache-maven-3.8.8-bin.zip -P /tmp"
- sh: "wget https://dlcdn.apache.org/maven/maven-3/3.9.5/binaries/apache-maven-3.9.5-bin.zip -P /tmp"
- sh: "sudo unzip -d /opt/mvn /tmp/apache-maven-*.zip"
- sh: "PATH=/opt/mvn/apache-maven-3.8.8/bin:$PATH"
- sh: "JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64"
- sh: "PATH=$JAVA_HOME/bin:$PATH"
- sh: "javac -version"
- sh: "java -version"
- sh: "PATH=/opt/mvn/apache-maven-3.9.5/bin:$PATH"
- sh: "mvn --version"

- sh: "source ${HOME}/venv${PYTHON_VERSION}/bin/activate"
Expand All @@ -57,16 +62,17 @@ install:

# Install latest gradle
- sh: "sudo apt-get -y remove gradle"
- sh: "wget https://services.gradle.org/distributions/gradle-7.3.1-bin.zip -P /tmp"
- sh: "wget https://services.gradle.org/distributions/gradle-8.4-bin.zip -P /tmp"
- sh: "sudo unzip -d /opt/gradle /tmp/gradle-*.zip"
- sh: "PATH=/opt/gradle/gradle-7.3.1/bin:$PATH"
- sh: "PATH=/opt/gradle/gradle-8.4/bin:$PATH"
- sh: "gradle --version"

# Install AWS CLI
- sh: "virtualenv aws_cli"
- sh: "./aws_cli/bin/python -m pip install awscli"
- sh: "PATH=$(echo $PWD'/aws_cli/bin'):$PATH"

- sh: "PATH=$PATH:$HOME/venv3.7/bin:$HOME/venv3.8/bin:$HOME/venv3.9/bin:$HOME/venv3.10/bin:$HOME/venv3.11/bin"
- sh: "PATH=$PATH:$HOME/venv3.7/bin:$HOME/venv3.8/bin:$HOME/venv3.9/bin:$HOME/venv3.10/bin:$HOME/venv3.11/bin:$HOME/venv3.12/bin"

# Install pytest
- sh: "python3.9 -m venv $HOME/pytest"
Expand Down
38 changes: 22 additions & 16 deletions appveyor-ubuntu.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
version: 1.0.{build}
image:
- Ubuntu2004
- Ubuntu2204
hawflau marked this conversation as resolved.
Show resolved Hide resolved

configuration:
- BuildIntegTesting
Expand Down Expand Up @@ -33,21 +33,27 @@ install:
# AppVeyor's apt-get cache might be outdated, and the package could potentially be 404.
- sh: "sudo apt-get update --allow-releaseinfo-change"

# install coretto 21
- sh: wget -O - https://apt.corretto.aws/corretto.key | sudo gpg --dearmor -o /usr/share/keyrings/corretto-keyring.gpg
- sh: echo "deb [signed-by=/usr/share/keyrings/corretto-keyring.gpg] https://apt.corretto.aws stable main" | sudo tee /etc/apt/sources.list.d/corretto.list
- sh: sudo apt-get update; sudo apt-get install -y java-21-amazon-corretto-jdk
- sh: JAVA_HOME=/usr/lib/jvm/java-21-amazon-corretto
- sh: PATH=$JAVA_HOME/bin:$PATH
- sh: java --version
- sh: javac --version

- sh: "gvm use go1.19"
- sh: "echo $PATH"
- sh: "ls /usr/"
# install latest maven which is compatible with jdk17
- sh: "sudo apt-get -y remove maven"
- sh: "wget https://dlcdn.apache.org/maven/maven-3/3.8.8/binaries/apache-maven-3.8.8-bin.zip -P /tmp"
- sh: "wget https://dlcdn.apache.org/maven/maven-3/3.9.5/binaries/apache-maven-3.9.5-bin.zip -P /tmp"
- sh: "sudo unzip -d /opt/mvn /tmp/apache-maven-*.zip"
- sh: "PATH=/opt/mvn/apache-maven-3.8.8/bin:$PATH"
- sh: "JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64"
- sh: "PATH=$JAVA_HOME/bin:$PATH"
- sh: "javac -version"
- sh: "java -version"
- sh: "PATH=/opt/mvn/apache-maven-3.9.5/bin:$PATH"
- sh: "mvn --version"

- sh: "source ${HOME}/venv${PYTHON_VERSION}/bin/activate"
- sh: "rvm reinstall ruby-3.2.2 --with-openssl-dir=/usr/lib/x86_64-linux-gnu" # reinstall ruby3.2 to fix OpenSSL issue
mndeveci marked this conversation as resolved.
Show resolved Hide resolved
- sh: "rvm use 3.2.2"
- sh: "docker info"
- sh: "docker version"
Expand All @@ -57,16 +63,17 @@ install:

# Install latest gradle
- sh: "sudo apt-get -y remove gradle"
- sh: "wget https://services.gradle.org/distributions/gradle-7.3.1-bin.zip -P /tmp"
- sh: "wget https://services.gradle.org/distributions/gradle-8.4-bin.zip -P /tmp"
- sh: "sudo unzip -d /opt/gradle /tmp/gradle-*.zip"
- sh: "PATH=/opt/gradle/gradle-7.3.1/bin:$PATH"
- sh: "PATH=/opt/gradle/gradle-8.4/bin:$PATH"
- sh: "gradle --version"

# Install AWS CLI
- sh: "virtualenv aws_cli"
- sh: "./aws_cli/bin/python -m pip install awscli"
- sh: "PATH=$(echo $PWD'/aws_cli/bin'):$PATH"

- sh: "PATH=$PATH:$HOME/venv3.7/bin:$HOME/venv3.8/bin:$HOME/venv3.9/bin:$HOME/venv3.10/bin:$HOME/venv3.11/bin"
- sh: "PATH=$PATH:$HOME/venv3.7/bin:$HOME/venv3.8/bin:$HOME/venv3.9/bin:$HOME/venv3.10/bin:$HOME/venv3.11/bin:$HOME/venv3.12/bin"

# update ca-certificates which causes failures with newest golang library
- sh: "sudo apt-get install --reinstall ca-certificates"
Expand Down Expand Up @@ -113,19 +120,19 @@ install:
# required for RIE with arm64 in linux
- sh: "
if [[ -n $BY_CANARY ]] && [[ -n $DOCKER_USER ]] && [[ -n $DOCKER_PASS ]];
then echo Logging in Docker Hub; echo $DOCKER_PASS | docker login --username $DOCKER_USER --password-stdin registry-1.docker.io;
then echo Logging in Docker Hub; echo $DOCKER_PASS | docker login --username $DOCKER_USER --password-stdin registry-1.docker.io;
fi"
- sh: "
if [[ -n $BY_CANARY ]] && [[ -n $DOCKER_USER ]] && [[ -n $DOCKER_PASS ]];
then echo Logging in Docker Hub; echo $DOCKER_PASS | docker login --username $DOCKER_USER --password-stdin;
then echo Logging in Docker Hub; echo $DOCKER_PASS | docker login --username $DOCKER_USER --password-stdin;
fi"
- sh: "docker run --rm --privileged multiarch/qemu-user-static --reset -p yes"


# Runs only in Linux, logging Public ECR when running canary and cred is available
- sh: "
if [[ -n $BY_CANARY ]];
then echo Logging in Public ECR; aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws;
then echo Logging in Public ECR; aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws;
fi"


Expand All @@ -152,7 +159,7 @@ on_finish:

for:
# Integ testing build
-
-
matrix:
only:
- configuration: BuildIntegTesting
Expand All @@ -168,8 +175,7 @@ for:
- sh: "pytest -vv -n 2 --reruns 3 tests/integration/buildcmd -m 'java or python or provided' --ignore=tests/integration/buildcmd/test_build_cmd_arm64.py --ignore=tests/integration/buildcmd/test_build_terraform_applications.py --ignore=tests/integration/buildcmd/test_build_terraform_applications_other_cases.py --json-report --json-report-file=TEST_REPORT-integration-buildcmd-java-python-provided.json"

# Integ testing build arm64 functions
-
matrix:
- matrix:
only:
- configuration: BuildIntegTestingArm64

Expand Down
15 changes: 9 additions & 6 deletions appveyor-windows-binary.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,20 +54,23 @@ cache:
- C:\ProgramData\chocolatey\lib -> appveyor.yml

install:
# upgrade chocolately
- choco upgrade chocolatey
# setup make
- "choco install make"
# setup Java, Maven and Gradle
- "choco install gradle -y --force"
- 'set JAVA_HOME=C:\Program Files\Java\jdk17'
- 'set PATH=%JAVA_HOME%\bin;%PATH%'
- "javac -version"
- "java -version"
- choco install correttojdk --version=21.0.0
hawflau marked this conversation as resolved.
Show resolved Hide resolved
- refreshenv
hawflau marked this conversation as resolved.
Show resolved Hide resolved
- java --version
- javac --version
- choco uninstall gradle
- choco install gradle
- "gradle -v"
- "mvn --version"

# Make sure the temp directory exists for Python to use.
- ps: "mkdir -Force C:\\tmp"
- 'set PATH=%PYTHON_HOME%;C:\Ruby32-x64\bin;%PATH%;C:\Python37-x64;C:\Python39-x64;C:\Python310-x64;C:\Python38-x64'
- 'set PATH=%PYTHON_HOME%;C:\Ruby32-x64\bin;%PATH%;C:\Python37-x64;C:\Python39-x64;C:\Python310-x64;C:\Python38-x64;C:\Python312-x64'
- "echo %PYTHON_HOME%"
- "echo %PATH%"
- "python --version"
Expand Down
21 changes: 12 additions & 9 deletions appveyor-windows.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,23 +53,29 @@ cache:
- C:\ProgramData\chocolatey\lib -> appveyor.yml

install:
# upgrade chocolately
- choco upgrade chocolatey
# setup make
- "choco install make"
# setup Java, Maven and Gradle
- "choco install gradle -y --force"
- 'set JAVA_HOME=C:\Program Files\Java\jdk17'
- 'set PATH=%JAVA_HOME%\bin;%PATH%'
- "javac -version"
- "java -version"
- choco install correttojdk --version=21.0.0
hawflau marked this conversation as resolved.
Show resolved Hide resolved
- refreshenv
hawflau marked this conversation as resolved.
Show resolved Hide resolved
- java --version
- javac --version
- choco uninstall gradle
- choco install gradle
- "gradle -v"
- "mvn --version"

# Make sure the temp directory exists for Python to use.
- ps: "mkdir -Force C:\\tmp"
- 'set PATH=%PYTHON_HOME%;C:\Ruby32-x64\bin;%PATH%;C:\Python37-x64;C:\Python39-x64;C:\Python310-x64;C:\Python311-x64'
- 'set PATH=%PYTHON_HOME%;C:\Ruby32-x64\bin;%PATH%;C:\Python37-x64;C:\Python39-x64;C:\Python310-x64;C:\Python311-x64;C:\Python312-x64'
- "echo %PYTHON_HOME%"
- "echo %PATH%"
- "python --version"
- ps: "Restart-Service docker"
hawflau marked this conversation as resolved.
Show resolved Hide resolved
# Switch to Docker Linux containers
- ps: Switch-DockerLinux
- "docker info"
- "docker version"

Expand All @@ -83,9 +89,6 @@ install:
# Install AWS CLI Globally via pip3
- "pip install awscli"

# Switch to Docker Linux containers
- ps: Switch-DockerLinux

# Check for git executable
- "git --version"

Expand Down
28 changes: 27 additions & 1 deletion tests/integration/buildcmd/build_integ_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@

from samcli.commands.build.utils import MountMode
from samcli.lib.utils import osutils
from samcli.lib.utils.architecture import X86_64, has_runtime_multi_arch_image
from samcli.lib.utils.architecture import ARM64, X86_64, has_runtime_multi_arch_image
from samcli.local.docker.lambda_build_container import LambdaBuildContainer
from samcli.yamlhelper import yaml_parse
from tests.testing_utils import (
Expand Down Expand Up @@ -587,7 +587,24 @@ def _verify_built_artifact(self, build_dir, function_logical_id, expected_files)

@pytest.mark.java
class BuildIntegJavaBase(BuildIntegBase):
# Run gradlew & gradle-kotlin containerized tests only with latest java version
LATEST_JAVA_VERSION = "java21"
SKIP_ARM64_EARLIER_JAVA_TESTS = "Skipping gradlew & gradle-kotlin test for this Java version"

FUNCTION_LOGICAL_ID = "Function"
EXPECTED_FILES_PROJECT_MANIFEST_GRADLE = {"aws", "lib", "META-INF"}
EXPECTED_FILES_PROJECT_MANIFEST_MAVEN = {"aws", "lib"}
EXPECTED_GRADLE_DEPENDENCIES = {"annotations-2.1.0.jar", "aws-lambda-java-core-1.1.0.jar"}
EXPECTED_MAVEN_DEPENDENCIES = {
"software.amazon.awssdk.annotations-2.1.0.jar",
"com.amazonaws.aws-lambda-java-core-1.1.0.jar",
}

FUNCTION_LOGICAL_ID = "Function"
USING_GRADLE_PATH = os.path.join("Java", "gradle")
USING_GRADLEW_PATH = os.path.join("Java", "gradlew")
USING_GRADLE_KOTLIN_PATH = os.path.join("Java", "gradle-kotlin")
USING_MAVEN_PATH = os.path.join("Java", "maven")

def _test_with_building_java(
self,
Expand All @@ -602,6 +619,15 @@ def _test_with_building_java(
if use_container and (SKIP_DOCKER_TESTS or SKIP_DOCKER_BUILD):
self.skipTest(SKIP_DOCKER_MESSAGE)

# skip arm64 use container tests for gradlew and gradle-kotlin
if (
use_container
and runtime != self.LATEST_JAVA_VERSION
and architecture == ARM64
and any(path in code_path for path in [self.USING_GRADLEW_PATH, self.USING_GRADLE_KOTLIN_PATH])
):
self.skipTest(self.SKIP_ARM64_EARLIER_JAVA_TESTS)

overrides = self.get_override(runtime, code_path, architecture, "aws.example.Hello::myHandler")
cmdlist = self.get_command_list(use_container=use_container, parameter_overrides=overrides)
cmdlist += ["--skip-pull-image"]
Expand Down
Loading