Problem uploading reports to docker-based single-instance codecov install #378

andreiz · 2024-05-14T20:21:08Z

Describe the bug
I am running codecov self-hosted on a single EC2 instance using docker-compose.yml. Logged in via Github and set up the test repo per tutorial. However, trying to upload reports fails.

Environment (please complete the following information):)

System: codecov on EC2, CI is Github

CI log

/home/runner/work/_actions/codecov/codecov-action/v4.2.0/dist/codecov --enterprise-url https://codecov.bbbb.ccc/ -v do-upload -Z --git-service github
info - 2024-05-14 18:58:33,195 -- ci service found: github-actions
debug - 2024-05-14 18:58:33,198 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-05-14 18:58:33,201 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
warning - 2024-05-14 18:58:33,203 -- No config file could be found. Ignoring config.
debug - 2024-05-14 18:58:33,203 -- No codecov_yaml found
debug - 2024-05-14 18:58:33,205 -- Starting upload processing --- {"branch": "step2", "build_code": "9084726456", "build_url": "https://github.com/aaaaa/testrepo/actions/runs/9084726456", "commit_sha": "1d711a6f6b0a141f1499341aed98469db16bf83d", "disable_file_fixes": false, "disable_search": false, "enterprise_url": "https://codecov.bbbb.ccc/", "env_vars": {}, "files_search_exclude_folders": [], "files_search_explicitly_listed_files": [], "files_search_root_folder": "/home/runner/work/testrepo/testrepo", "flags": [], "git_service": "github", "handle_no_reports_found": false, "job_code": "API workflow", "name": "9084726456", "network_filter": null, "network_prefix": null, "network_root_folder": "/home/runner/work/testrepo/testrepo", "plugin_names": ["xcode", "gcov", "pycoverage"], "pull_request_number": null, "report_code": "default", "slug": "bbbb/testrepo", "token": "f******************", "upload_file_type": "coverage"}
debug - 2024-05-14 18:58:33,205 -- Selected preparation plugins --- {"selected_plugins": ["<class 'codecov_cli.plugins.xcode.XcodePlugin'>", "<class 'codecov_cli.plugins.gcov.GcovPlugin'>", "<class 'codecov_cli.plugins.pycoverage.Pycoverage'>"]}
debug - 2024-05-14 18:58:33,205 -- Running preparation plugin: <class 'codecov_cli.plugins.xcode.XcodePlugin'>
debug - 2024-05-14 18:58:33,205 -- Running xcode plugin...
warning - 2024-05-14 18:58:33,205 -- xcrun is not installed or can't be found.
debug - 2024-05-14 18:58:33,205 -- Running preparation plugin: <class 'codecov_cli.plugins.gcov.GcovPlugin'>
debug - 2024-05-14 18:58:33,205 -- Running gcov plugin...
warning - 2024-05-14 18:58:33,208 -- No gcov data found.
debug - 2024-05-14 18:58:33,208 -- Running preparation plugin: <class 'codecov_cli.plugins.pycoverage.Pycoverage'>
info - 2024-05-14 18:58:33,209 -- Generating coverage.xml report in /home/runner/work/testrepo/testrepo
info - 2024-05-14 18:58:33,305 -- Wrote XML report to coverage.xml
debug - 2024-05-14 18:58:33,306 -- Collecting relevant files
info - 2024-05-14 18:58:33,315 -- Found 1 coverage files to report
info - 2024-05-14 18:58:33,315 -- > /home/runner/work/testrepo/testrepo/coverage.xml
debug - 2024-05-14 18:58:33,319 -- Selected uploader to use: <class 'codecov_cli.services.upload.upload_sender.UploadSender'>
debug - 2024-05-14 18:58:33,319 -- Sending upload request to Codecov
info - 2024-05-14 18:58:33,457 -- Your upload is now processing. When finished, results will be available at: https://codecov.bbbb.ccc/github/bbbb/testrepo/commit/1d711a6f6b0a141f1499341aed98469db16bf83d
debug - 2024-05-14 18:58:33,457 -- Upload request to Codecov complete. --- {"response": {"external_id": "0f7b6457-cd4f-44ad-bcac-c9f9b171621d", "created_at": "2024-05-14T18:58:34.208832Z", "raw_upload_location": "http://minio:9000/archive/v4/raw/2024-05-14/8B2BBCBD873D6CDADDAC6E1444671A7C/1d711a6f6b0a141f1499341aed98469db16bf83d/9fa1e0a2-cd45-4dfe-ad28-183f3e4e544f/0f7b6457-cd4f-44ad-bcac-c9f9b171621d.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=codecov-default-key%2F20240514%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240514T185834Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=ddd12c90eb7b6c20dda5f33f7256b87d2998d4ffb7424168fe64de4d24852ab7", "state": "", "provider": null, "upload_type": "uploaded", "url": "https://codecov.bbbb.ccc/github/bbbb/testrepo/commit/1d711a6f6b0a141f1499341aed98469db16bf83d", "ci_url": "https://github.com/bbbb/testrepo/actions/runs/9084726456", "flags": [], "env": {}, "name": "9084726456", "job_code": "API workflow"}}
debug - 2024-05-14 18:58:33,458 -- Sending upload to storage
warning - 2024-05-14 18:58:33,465 -- Request failed. Retrying --- {"retry": 0}
warning - 2024-05-14 18:58:33,972 -- Request failed. Retrying --- {"retry": 1}
warning - 2024-05-14 18:58:34,980 -- Request failed. Retrying --- {"retry": 2}
Traceback (most recent call last):
  File "codecov_cli/main.py", line 83, in <module>
  File "codecov_cli/main.py", line 79, in run
  File "click/core.py", line 1157, in __call__
  File "click/core.py", line 1078, in main
  File "click/core.py", line 1688, in invoke
  File "click/core.py", line 1434, in invoke
  File "click/core.py", line 783, in invoke
  File "click/decorators.py", line 33, in new_func
  File "codecov_cli/commands/upload.py", line 257, in do_upload
  File "codecov_cli/services/upload/__init__.py", line 112, in do_upload_logic
  File "codecov_cli/services/upload/upload_sender.py", line 104, in send_upload_data
  File "codecov_cli/helpers/request.py", line 78, in wrapper
Exception: Request failed after too many retries
[1628] Failed to execute script 'main' due to unhandled exception!
Error: Codecov:
                        Failed to properly upload report: The process '/home/runner/work/_actions/codecov/codecov-action/v4.2.0/dist/codecov' failed with exit code 1

Additional context
Add any other context about the problem here.

The text was updated successfully, but these errors were encountered:

drazisil-codecov · 2024-05-15T12:24:57Z

@andreiz

Is http://minio:9000/archive/v4/raw/2024-05-14/8B2BBCBD873D6CDADDAC6E1444671A7C/1d711a6f6b0a141f1499341aed98469db16bf83d/9fa1e0a2-cd45-4dfe-ad28-183f3e4e544f/0f7b6457-cd4f-44ad-bcac-c9f9b171621d.txt reachable from the uploader?

Since it's a local docker name, and you are uploading from GitHub, I'm guessing it is not.

andreiz · 2024-05-15T14:43:44Z

@andreiz

Is http://minio:9000/archive/v4/raw/2024-05-14/8B2BBCBD873D6CDADDAC6E1444671A7C/1d711a6f6b0a141f1499341aed98469db16bf83d/9fa1e0a2-cd45-4dfe-ad28-183f3e4e544f/0f7b6457-cd4f-44ad-bcac-c9f9b171621d.txt reachable from the uploader?

Since it's a local docker name, and you are uploading from GitHub, I'm guessing it is not.

If it is supposed to be reachable from the outside, it's not something I saw mentioned in the docs. I literally just cloned the https://github.com/codecov/self-hosted repo and then followed the Self-Hosted Configuration Guide and nothing there mentioned minio.

I was under the impression that the codecov API uses minio locally as the storage backend API, but I guess I am mistaken?

drazisil-codecov · 2024-05-15T17:02:01Z

Minio uses whatever the host is set for the storage service

https://github.com/codecov/codecov-api/blob/main/services/storage.py#L26

The value needs to be reachable both in the container and externally, as the report is uploaded directed to storage, bypassing Codecov.

andreiz · 2024-05-15T19:54:47Z

I got it running by opening up port 9000 in the firewall and configuring minio host to be the same as the main codecov one. But can codecov-gateway proxy external traffic to minio container instead and do SSL termination? If not, then how do I set up minio to run with SSL enabled?

rohit5roy · 2024-06-19T22:03:25Z

I'm also receiving the same error. Is there any solution or work around?

warning - 2024-06-19 21:46:27,782 -- Request failed. Retrying --- {"retry": 0}
warning - 2024-06-19 21:48:38,854 -- Request failed. Retrying --- {"retry": 1}
warning - 2024-06-19 21:50:49,926 -- Request failed. Retrying --- {"retry": 2}
Traceback (most recent call last):
  File "codecov_cli/main.py", line 83, in <module>
  File "codecov_cli/main.py", line 79, in run
  File "click/core.py", line 1157, in __call__
  File "click/core.py", line 1078, in main
  File "click/core.py", line 1688, in invoke
  File "click/core.py", line 1434, in invoke
  File "click/core.py", line 783, in invoke
  File "click/decorators.py", line 33, in new_func
  File "codecov_cli/commands/upload.py", line 257, in do_upload
  File "codecov_cli/services/upload/__init__.py", line 112, in do_upload_logic
  File "codecov_cli/services/upload/upload_sender.py", line [104](https://github.com/XXXXXX/some-new-service/actions/runs/95884677415/job/264412345062#step:16:105), in send_upload_data
  File "codecov_cli/helpers/request.py", line 78, in wrapper
Exception: Request failed after too many retries
[936] Failed to execute script 'main' due to unhandled exception!
Warning: Codecov:
                        Failed to properly upload report: The process '/runner/_work/_actions/codecov/codecov-action/v4/dist/codecov' failed with exit code 1```

jonas-brauer · 2024-06-20T11:23:39Z

I'm also receiving the same error. Is there any solution or work around?

warning - 2024-06-19 21:46:27,782 -- Request failed. Retrying --- {"retry": 0}
warning - 2024-06-19 21:48:38,854 -- Request failed. Retrying --- {"retry": 1}
warning - 2024-06-19 21:50:49,926 -- Request failed. Retrying --- {"retry": 2}
Traceback (most recent call last):
  File "codecov_cli/main.py", line 83, in <module>
  File "codecov_cli/main.py", line 79, in run
  File "click/core.py", line 1157, in __call__
  File "click/core.py", line 1078, in main
  File "click/core.py", line 1688, in invoke
  File "click/core.py", line 1434, in invoke
  File "click/core.py", line 783, in invoke
  File "click/decorators.py", line 33, in new_func
  File "codecov_cli/commands/upload.py", line 257, in do_upload
  File "codecov_cli/services/upload/__init__.py", line 112, in do_upload_logic
  File "codecov_cli/services/upload/upload_sender.py", line [104](https://github.com/XXXXXX/some-new-service/actions/runs/95884677415/job/264412345062#step:16:105), in send_upload_data
  File "codecov_cli/helpers/request.py", line 78, in wrapper
Exception: Request failed after too many retries
[936] Failed to execute script 'main' due to unhandled exception!
Warning: Codecov:
                        Failed to properly upload report: The process '/runner/_work/_actions/codecov/codecov-action/v4/dist/codecov' failed with exit code 1```

I solved this by placing nginx in front (of docker compose resources) and serving minio on a separate subdomain.

drazisil-codecov · 2024-06-20T13:24:42Z

Thanks for sharing your solution , @jonas-brauer ! Are you also using SSL?

jonas-brauer · 2024-06-20T14:13:59Z

@drazisil-codecov Yes we are.
Depending on your setup you may want to set this in your codecov.yml

services:
    minio:
        host: your.host.name
        port: 443
        verify_ssl: true

drazisil-codecov · 2024-06-20T14:34:19Z

@andreiz Please see if the instructions above help you.

andreiz · 2024-06-20T14:48:47Z

@andreiz Please see if the instructions above help you.

I will give it a shot

rohit5roy · 2024-06-24T08:47:10Z

@drazisil-codecov Yes we are. Depending on your setup you may want to set this in your codecov.yml
services:
    minio:
        host: your.host.name
        port: 443
        verify_ssl: true

This didn't work for me.
Also I'm able to access both Codecov UI and Minio using SSL.

For minio SSL I've added the following:

volumes:
  - archive-volume:/export
  - ./cert/data/live/something.com/fullchain.pem:/root/.minio/certs/public.crt:ro
  - ./cert/data/live/something.com/privkey.pem:/root/.minio/certs/private.key:ro

I'm using certbot for SSL

rohit5roy · 2024-06-24T08:48:56Z

I'm also receiving the same error. Is there any solution or work around?

warning - 2024-06-19 21:46:27,782 -- Request failed. Retrying --- {"retry": 0}
warning - 2024-06-19 21:48:38,854 -- Request failed. Retrying --- {"retry": 1}
warning - 2024-06-19 21:50:49,926 -- Request failed. Retrying --- {"retry": 2}
Traceback (most recent call last):
  File "codecov_cli/main.py", line 83, in <module>
  File "codecov_cli/main.py", line 79, in run
  File "click/core.py", line 1157, in __call__
  File "click/core.py", line 1078, in main
  File "click/core.py", line 1688, in invoke
  File "click/core.py", line 1434, in invoke
  File "click/core.py", line 783, in invoke
  File "click/decorators.py", line 33, in new_func
  File "codecov_cli/commands/upload.py", line 257, in do_upload
  File "codecov_cli/services/upload/__init__.py", line 112, in do_upload_logic
  File "codecov_cli/services/upload/upload_sender.py", line [104](https://github.com/XXXXXX/some-new-service/actions/runs/95884677415/job/264412345062#step:16:105), in send_upload_data
  File "codecov_cli/helpers/request.py", line 78, in wrapper
Exception: Request failed after too many retries
[936] Failed to execute script 'main' due to unhandled exception!
Warning: Codecov:
                        Failed to properly upload report: The process '/runner/_work/_actions/codecov/codecov-action/v4/dist/codecov' failed with exit code 1```

I solved this by placing nginx in front (of docker compose resources) and serving minio on a separate subdomain.

@jonas-brauer thanks for your response.
A simple/small code snippet would have been very helpful.

Thanks.

jonas-brauer · 2024-06-25T08:05:38Z

@rohit5roy I'm not sure I can boil down nginx into a small snippet, but this link should be a good place to start to learn about reverse proxy if it's new to you 😃
https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/

derklaro · 2024-06-27T13:13:24Z

Just ran into the same issue, it seems like the gateway was once set up to proxy requests to <host>/archive (where archive is the default bucket name) to the minio storage. This was broken due to a missing file rename, just opened a PR to resolve this: codecov/codecov-gateway#97

eliatcodecov · 2024-07-10T15:15:13Z

Thank you @derklaro for the contribution, we'll review and move discussion there. In the meantime I'll close this and direct any further discussion to the PR itself: codecov/codecov-gateway#97

covecod bot added the Waiting for: Product Owner label May 14, 2024

covecod bot added this to GitHub Issues with 👀 May 14, 2024

covecod bot moved this to Waiting for: Product Owner in GitHub Issues with 👀 May 14, 2024

drazisil-codecov moved this from Waiting for: Product Owner to Waiting for: Community in GitHub Issues with 👀 May 15, 2024

covecod bot added Waiting for: Community and removed Waiting for: Product Owner labels May 15, 2024

covecod bot added Waiting for: Product Owner and removed Waiting for: Community labels May 15, 2024

covecod bot moved this from Waiting for: Community to Waiting for: Product Owner in GitHub Issues with 👀 May 15, 2024

drazisil-codecov moved this from Waiting for: Product Owner to Waiting for: Community in GitHub Issues with 👀 May 15, 2024

covecod bot added Waiting for: Community and removed Waiting for: Product Owner labels May 15, 2024

covecod bot added Waiting for: Product Owner and removed Waiting for: Community labels May 15, 2024

covecod bot moved this from Waiting for: Community to Waiting for: Product Owner in GitHub Issues with 👀 May 15, 2024

thomasrockhu-codecov added the bug For issues and unexpected behaviors label May 28, 2024

covecod bot removed the Waiting for: Product Owner label May 28, 2024

thomasrockhu-codecov assigned jason-ford-codecov and unassigned jason-ford-codecov May 28, 2024

covecod bot added the Waiting for: Product Owner label Jun 19, 2024

drazisil-codecov moved this from Waiting for: Product Owner to Waiting for: Support in GitHub Issues with 👀 Jun 20, 2024

covecod bot added Waiting for: Support and removed Waiting for: Product Owner labels Jun 20, 2024

covecod bot moved this from Waiting for: Community to Waiting for: Support in GitHub Issues with 👀 Jun 20, 2024

drazisil-codecov moved this from Waiting for: Support to Waiting for: Community in GitHub Issues with 👀 Jun 20, 2024

drazisil-codecov added Waiting for: Product Owner and removed Waiting for: Support bug For issues and unexpected behaviors labels Jun 20, 2024

covecod bot removed the status in GitHub Issues with 👀 Jun 20, 2024

covecod bot moved this to Waiting for: Product Owner in GitHub Issues with 👀 Jun 20, 2024

codecov-hooky bot added the bug For issues and unexpected behaviors label Jun 20, 2024

drazisil-codecov moved this from Waiting for: Product Owner to Waiting for: Community in GitHub Issues with 👀 Jun 20, 2024

covecod bot added Waiting for: Community and removed Waiting for: Product Owner labels Jun 20, 2024

drazisil-codecov removed the bug For issues and unexpected behaviors label Jun 20, 2024

codecov-hooky bot added the bug For issues and unexpected behaviors label Jun 20, 2024

covecod bot added Waiting for: Product Owner and removed Waiting for: Community labels Jun 24, 2024

covecod bot moved this from Waiting for: Community to Waiting for: Product Owner in GitHub Issues with 👀 Jun 24, 2024

eliatcodecov closed this as completed Jul 10, 2024

covecod bot removed the Waiting for: Product Owner label Jul 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problem uploading reports to docker-based single-instance codecov install #378

Problem uploading reports to docker-based single-instance codecov install #378

andreiz commented May 14, 2024

drazisil-codecov commented May 15, 2024

andreiz commented May 15, 2024

drazisil-codecov commented May 15, 2024

andreiz commented May 15, 2024

rohit5roy commented Jun 19, 2024

jonas-brauer commented Jun 20, 2024 •

edited

Loading

drazisil-codecov commented Jun 20, 2024 •

edited

Loading

jonas-brauer commented Jun 20, 2024

drazisil-codecov commented Jun 20, 2024

andreiz commented Jun 20, 2024

rohit5roy commented Jun 24, 2024 •

edited

Loading

rohit5roy commented Jun 24, 2024 •

edited

Loading

jonas-brauer commented Jun 25, 2024

derklaro commented Jun 27, 2024

eliatcodecov commented Jul 10, 2024

Problem uploading reports to docker-based single-instance codecov install #378

Problem uploading reports to docker-based single-instance codecov install #378

Comments

andreiz commented May 14, 2024

drazisil-codecov commented May 15, 2024

andreiz commented May 15, 2024

drazisil-codecov commented May 15, 2024

andreiz commented May 15, 2024

rohit5roy commented Jun 19, 2024

jonas-brauer commented Jun 20, 2024 • edited Loading

drazisil-codecov commented Jun 20, 2024 • edited Loading

jonas-brauer commented Jun 20, 2024

drazisil-codecov commented Jun 20, 2024

andreiz commented Jun 20, 2024

rohit5roy commented Jun 24, 2024 • edited Loading

rohit5roy commented Jun 24, 2024 • edited Loading

jonas-brauer commented Jun 25, 2024

derklaro commented Jun 27, 2024

eliatcodecov commented Jul 10, 2024

jonas-brauer commented Jun 20, 2024 •

edited

Loading

drazisil-codecov commented Jun 20, 2024 •

edited

Loading

rohit5roy commented Jun 24, 2024 •

edited

Loading

rohit5roy commented Jun 24, 2024 •

edited

Loading