onboarding: docker-compose up errored at pyrrha-simulator

[albertum1]

With a fresh ubuntu on virtualbox, I followed the steps on the docker-compose.md
Got stuck on docker-compose up with looped error as shown below.

[krook]

Is this still an issue @albertum1?

[albertum1]

I believe I changed the package.json file from sensor simulator to get this to work, Pyrrha-Platform/Pyrrha-Sensor-Simulator#19

[upkarlidder]

@albertum1, can you please pull main again and try. I created a dev dockerfile that removed the --only=production line. I think it's okay to install dev dependencies in your local environment. Let me know if this works.

The PR you submitted is another way to fix the problem, however that moves nodemon from dev dependency to production. It's not really a production dependency. I remember that's what we did when we solved it locally for you, but this seems like a better way. Ideally, you would have made this fix on your PR, but I know a few people have been waiting for this and blocked on me. So I went ahead and made the fix in a new PR.

[upkarlidder]

@albertum1 can you please try again when you get a chance? Thank you!

[albertum1]

@upkarlidder confirming receipt

[albertum1]

@upkarlidder, on a fresh ubuntu I am stuck on docker-compose build returning this error

[upkarlidder]

@dacamposol can you please take a look? Looks like another docker dependency issue in Rules-Decision. Docker-compose simply uses the repo Dockerfile. I have not investigated yet. I will take a look later today as well.

Thank you @albertum1 for running through this!

[upkarlidder]

I just tried it and it worked fine for me. Just adding logs here and will dm @albertum1

Sending build context to Docker daemon  1.049MB
Step 1/19 : FROM python:3.8.12-bullseye as build
 ---> f746089c9d02
Step 2/19 : WORKDIR /usr/app
 ---> Using cache
 ---> e0957dee2529
Step 3/19 : RUN python -m venv /usr/app/venv
 ---> Using cache
 ---> dc163f3bdb2b
Step 4/19 : ENV PATH="/usr/app/venv/bin:$PATH"
 ---> Using cache
 ---> d140bd8f8237
Step 5/19 : COPY requirements.txt .
 ---> Using cache
 ---> 7f2c5d54569d
Step 6/19 : RUN pip install --no-cache-dir -r requirements.txt
 ---> Using cache
 ---> 38ed220d2a01
Step 7/19 : FROM python:3.8.12-slim-bullseye@sha256:d31a1beb6ccddbf5b5c72904853f5c2c4d1f49bb8186b623db0b80f8c37b5899
 ---> 214d62795dbb
Step 8/19 : RUN apt-get update &&    apt-get --no-install-recommends -y install curl=7.74.0-1.3+b1 &&    rm -rf /var/lib/apt/lists/* &&    groupadd -g 999 python &&    useradd -u 999 -g python python &&    mkdir /usr/app &&    chown python:python /usr/app
 ---> Using cache
 ---> 1127cf1bdc81
Step 9/19 : WORKDIR /usr/app
 ---> Using cache
 ---> 467cbbaa7e67
Step 10/19 : EXPOSE 8080
 ---> Using cache
 ---> ad19b721fa6d
Step 11/19 : ENV PYTHONUNBUFFERED=1
 ---> Using cache
 ---> a63f3254774a
Step 12/19 : COPY --from=build --chown=python:python /usr/app/venv ./venv
 ---> Using cache
 ---> e2b45561b689
Step 13/19 : ENV PATH="/usr/app/venv/bin:$PATH"
 ---> Using cache
 ---> 9c2b3e4845ed
Step 14/19 : WORKDIR /usr/app/src
 ---> Using cache
 ---> 63b7662b521b
Step 15/19 : COPY --chown=python:python src/* ./
 ---> Using cache
 ---> 6d26a2254181
Step 16/19 : RUN [ -f ".env" ] || cp .env.docker .env
 ---> Using cache
 ---> c4417d21fd74
Step 17/19 : USER python
 ---> Using cache
 ---> a5242d556c4b
Step 18/19 : ENTRYPOINT ["gunicorn"  , "-b", "0.0.0.0:8080", "core_decision_flask_app:app"]
 ---> Using cache
 ---> 7c75adcaa712
Step 19/19 : HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 CMD curl -f http://localhost:8080/health
 ---> Using cache
 ---> 3d1974d980ad
Successfully built 3d1974d980ad
Successfully tagged pyrrha-rulesdecision:latest

[dacamposol]

@upkarlidder you're using your already cached layers on that test, so it doesn't actually show if there is a problem at the Alpine repository level.

I will take a look this evening.

[upkarlidder]

Good catch @dacamposol. I tried from scratch after removing the cache and I do get the same error:

Step 8/19 : RUN apt-get update &&    apt-get --no-install-recommends -y install curl=7.74.0-1.3+b1 &&    rm -rf /var/lib/apt/lists/* &&    groupadd -g 999 python &&    useradd -u 999 -g python python &&    mkdir /usr/app &&    chown python:python /usr/app
 ---> Running in 987b62901bff
Get:1 http://security.debian.org/debian-security bullseye-security InRelease [44.1 kB]
Get:2 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:3 http://deb.debian.org/debian bullseye-updates InRelease [39.4 kB]
Get:4 http://security.debian.org/debian-security bullseye-security/main amd64 Packages [109 kB]
Get:5 http://deb.debian.org/debian bullseye/main amd64 Packages [8183 kB]
Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [2592 B]
Fetched 8494 kB in 2s (5544 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
E: Version '7.74.0-1.3+b1' for 'curl' was not found
The command '/bin/sh -c apt-get update &&    apt-get --no-install-recommends -y install curl=7.74.0-1.3+b1 &&    rm -rf /var/lib/apt/lists/* &&    groupadd -g 999 python &&    useradd -u 999 -g python python &&    mkdir /usr/app &&    chown python:python /usr/app' returned a non-zero code: 100

So in order to learn a little more I did some investigation. I ran python:3.8.12-slim-bullseye@sha256:d31a1beb6ccddbf5b5c72904853f5c2c4d1f49bb8186b623db0b80f8c37b5899 image and used apt to see which curl I can install. I am not exactly sure how apt works as I use Fedora and dnf, but looks like we need to change the version to curl=7.74.0-1.3+deb11u1. Is that right?

root@d676f7657880:/# apt-cache showpkg curl
Package: curl
Versions: 
7.74.0-1.3+deb11u1 (/var/lib/apt/lists/deb.debian.org_debian_dists_bullseye_main_binary-amd64_Packages.lz4)
 Description Language: 
                 File: /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye_main_binary-amd64_Packages.lz4
                  MD5: f83293d10df083ae6f7bb7d01642913c


Reverse Depends: 
  plowshare,curl 7.24
  tang,curl
  youtube-dl,curl
  yorick,curl
  wsl,curl
  wsl,curl
   ...

Does apt only cache certain versions? If so, will we run into this again next time they update the curl version? I guess one workaround would be to just remove the version and always install the latest curl, but I do like pinning the dependency.

[dacamposol]

@upkarlidder I don't think we should unpin the version, we could just pin up to the minor changes and just keep any build or bugfix version to pass through:

RUN apt-get update &&\
    apt-get --no-install-recommends -y install curl=7.74.* &&\
    rm -rf /var/lib/apt/lists/* &&\
    groupadd -g 999 python &&\
    useradd -u 999 -g python python &&\
    mkdir /usr/app &&\
    chown python:python /usr/app

Probably the issue is because the build versions (-1.3+b1) aren't cached in the APT repository.

[upkarlidder]

Thank you @dacamposol. PR merged.

@albertum1, can you please try again when you get a chance? You will have to pull latest main from Pyrrha-Rules-Decision.