Buildpack suddenly failing our builds #13
Comments
|
same here |
1 similar comment
|
same here |
|
same ;( |
|
Exactly same problem here ☝️ |
|
Interesting. I changed my heroku env var JEMALLOC_VERSION to 5.2.1, that didn't fix the problem so I rolled it back using the heroku interface under "activity" which reset to 5.1.0 and now it deployed successfully. I was on 5.1.0 originally and not sure why it was failing trying to vendor 5.2.1 and ignoring my JEMALLOC_VERSION setting in the heroku ENV. Hope this helps someone. To clarify - it now vendored 5.1.0 as my env var states. |
|
First thought is that there was a hiccup on GitHub’s end with their releases endpoint. We don’t do a good job of validating that the request was successful so a 404 ends up erroring in bzip. I’ll look into this more once I’m able to get to a computer. |
|
Specifying |
|
GitHub is returning |
|
There is an open issue for errors downloading releases |
|
I'm getting 302 -> 200s in any authenticated request and signature errors for anonymous requests: <?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<AWSAccessKeyId>AKIAIWNJYAX4CSVEH53A</AWSAccessKeyId>
<StringToSign>AWS4-HMAC-SHA256
20200507T232450Z
20200507/us-east-1/s3/aws4_request
2ed18a106fbed2bb740aaac554aab63be163e9e9f68a1aa6c0271b84aac98891</StringToSign>
<SignatureProvided>33271da11e2ae3f394724a061294e257401af1c4a7e1dd309d82cfef6e77d23e</SignatureProvided>
<StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 30 30 35 30 37 54 32 33 32 34 35 30 5a 0a 32 30 32 30 30 35 30 37 2f 75 73 2d 65 61 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 32 65 64 31 38 61 31 30 36 66 62 65 64 32 62 62 37 34 30 61 61 61 63 35 35 34 61 61 62 36 33 62 65 31 36 33 65 39 65 39 66 36 38 61 31 61 61 36 63 30 32 37 31 62 38 34 61 61 63 39 38 38 39 31</StringToSignBytes>
<CanonicalRequest>GET
/113065003/685c2e00-b7a1-11e9-87f8-a8905d4da36a
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200507%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200507T232450Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&actor_id=0&repo_id=113065003&response-content-disposition=attachment%3B%20filename%3Djemalloc-5.2.1.tar.bz2&response-content-type=application%2Foctet-stream
host:github-production-release-asset-2e65be.s3.amazonaws.com
host
UNSIGNED-PAYLOAD</CanonicalRequest>
<CanonicalRequestBytes>47 45 54 0a 2f 31 31 33 30 36 35 30 30 33 2f 36 38 35 63 32 65 30 30 2d 62 37 61 31 2d 31 31 65 39 2d 38 37 66 38 2d 61 38 39 30 35 64 34 64 61 33 36 61 0a 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 4b 49 41 49 57 4e 4a 59 41 58 34 43 53 56 45 48 35 33 41 25 32 46 32 30 32 30 30 35 30 37 25 32 46 75 73 2d 65 61 73 74 2d 31 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 32 30 30 35 30 37 54 32 33 32 34 35 30 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 33 30 30 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 68 6f 73 74 26 61 63 74 6f 72 5f 69 64 3d 30 26 72 65 70 6f 5f 69 64 3d 31 31 33 30 36 35 30 30 33 26 72 65 73 70 6f 6e 73 65 2d 63 6f 6e 74 65 6e 74 2d 64 69 73 70 6f 73 69 74 69 6f 6e 3d 61 74 74 61 63 68 6d 65 6e 74 25 33 42 25 32 30 66 69 6c 65 6e 61 6d 65 25 33 44 6a 65 6d 61 6c 6c 6f 63 2d 35 2e 32 2e 31 2e 74 61 72 2e 62 7a 32 26 72 65 73 70 6f 6e 73 65 2d 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3d 61 70 70 6c 69 63 61 74 69 6f 6e 25 32 46 6f 63 74 65 74 2d 73 74 72 65 61 6d 0a 68 6f 73 74 3a 67 69 74 68 75 62 2d 70 72 6f 64 75 63 74 69 6f 6e 2d 72 65 6c 65 61 73 65 2d 61 73 73 65 74 2d 32 65 36 35 62 65 2e 73 33 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 0a 68 6f 73 74 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44</CanonicalRequestBytes>
<RequestId>766A2F7938E03094</RequestId>
<HostId>yfkv3G5Ojad9Q2h/UsGfU2AltkuORvpdbjyklZZsqdSKhfhSv6+rsgueVzynoyO001hmKpcnUbE=</HostId>
</Error>Interesting that it is only for some releases. Luck 5.1.0 works! |
|
Thank you @gaffneyc for your quick responses, appreciate it! |
|
GitHub has marked the incident as resolved, so everything should be working again. This does highlight a couple areas that could be improved with the buildpack.
I'm looking at 1 right now. For 2, I've done something similar for the spiped buildpack where it compiles a cached version into $CACHE_DIR. This might be a good chance to look at solving #12 as well. We've had a number of folks fork the repo so they can guarantee the binaries they're using are safe and being able to provide an alternative location would help work around this kind of thing in the future. |
GitHub's releases service recently had an issue where downloads were failing. This attempts to fix two main issues: resilience for deploys and improved error messages. Using Heroku's cache we should only need to download each version once. This will allow deploys should GitHub have issues in the future. Before this change, if there was an error downloading the bundle then the buildpack would fail with an error about bzip file formats. This now checks the http status code and prints a more useful error should it fail. Fixes #13
|
I can confirm that everything is working again. Thanks @gaffneyc for your prompt attention and for all your work with this project! Closing this since it sounds like there's a long-term fix in the works as well. |
We've been using this buildback with no problems for some time now.
Today, our builds started failing with the below message.
There have been no buildpack/configuration changes and we did not have this problem as recently as yesterday.
After removing the buildpack and retrying, the build was successful.
Any ideas on what could be the issue?
Thanks
The text was updated successfully, but these errors were encountered: