access-api: invoking store/list in production errors with `HandlerExecutionError` + `HTTPError` #363

gobengo · 2023-01-18T21:18:47Z

Motivation:

testing Unify UCAN API endpoints #325

Context:

after fix: upload-api-proxy default fetch binds to globalThis #361 , my w3protocol-test can successfully send can=store/list aud=did:web:staging.web3.storage invocations through access-api staging
but it can't do the same against prod

Problem

Error message seen in github actions run

# testing aud=did:web:web3.storage url=https://up.web3.storage/
# testing aud=did:web:web3.storage url=https://access.web3.storage/
# unexpected result from store/list invocation aud=did:web:web3.storage url=https://access.web3.storage/ {
#   name: 'HandlerExecutionError',
#   cause: {
#     url: 'https://up.web3.storage/',
#     name: 'HTTPError',
#     stack: 'HTTPError: HTTP Request failed\\n' +
#       '    at lN.throw (d1-beta-facade.entry.js:14585:11)\\n' +
#       '    at dN.request (d1-beta-facade.entry.js:14579:135)\\n' +
#       '    at async WN (d1-beta-facade.entry.js:15030:45)\\n' +
#       '    at async Object. (d1-beta-facade.entry.js:15037:132)\\n' +
#       '    at async IB (d1-beta-facade.entry.js:14077:12)\\n' +
#       '    at async AB (d1-beta-facade.entry.js:14065:12)\\n' +
#       '    at async TB (d1-beta-facade.entry.js:14059:42)\\n' +
#       '    at async PO (d1-beta-facade.entry.js:15097:11)\\n' +
#       '    at async ed.fetch (d1-beta-facade.entry.js:9434:17)\\n' +
#       '    at async Object.fetch (d1-beta-facade.entry.js:20183:13)',
#     status: 530,
#     message: 'HTTP Request failed',
#     statusText: ''
#   },
#   error: true,
#   stack: 'HandlerExecutionError: service handler {can: "store/list"} error: HTTP Request failed\\n' +
#     '    at IB (d1-beta-facade.entry.js:14079:13)\\n' +
#     '    at async AB (d1-beta-facade.entry.js:14065:12)\\n' +
#     '    at async TB (d1-beta-facade.entry.js:14059:42)\\n' +
#     '    at async PO (d1-beta-facade.entry.js:15097:11)\\n' +
#     '    at async ed.fetch (d1-beta-facade.entry.js:9434:17)\\n' +
#     '    at async Object.fetch (d1-beta-facade.entry.js:20183:13)',
#   message: 'service handler {can: "store/list"} error: HTTP Request failed',
#   capability: {
#     can: 'store/list',
#     with: 'did:key:z6MknC4PQP4MSBxgnDdotXtgTCAXtrGker1BM4R5LfG5u3Pa'
#   }
# }

The same doesn't happen when I run access-api locally configured with env.{DID,PRIVATE_KEY} same as production. This leads me to think it's something specific about our production deployment running in cloudflare workers

TAP version 13
# Subtest: /Users/bengo/bengo.is/monorepo/experiments/w3protocol-test/dist/src/w3protocol.test.js
    # testing aud=did:web:staging.web3.storage url=https://staging.up.web3.storage/
    # testing aud=did:web:staging.web3.storage url=https://w3access-staging.protocol-labs.workers.dev/
    # testing aud=did:web:web3.storage url=https://up.web3.storage/
    # testing aud=did:web:web3.storage url=http://localhost:8787/
    # Subtest: can list items in a space
    ok 1 - can list items in a space
      ---
      duration_ms: 5206.01275
      ...
    # Subtest: w3protocol-test can upload file
    ok 2 - w3protocol-test can upload file # SKIP
      ---
      duration_ms: 0.061333
      ...
    1..2
ok 1 - /Users/bengo/bengo.is/monorepo/experiments/w3protocol-test/dist/src/w3protocol.test.js
  ---
  duration_ms: 5373.008917
  ...
1..1
# tests 1
# pass 1
# fail 0

Possible Solutions

access-api proxy should catch HTTPError and respond with a helpful non-error result
- feat: access-api proxy.js has configurable options.catchInvocationError, by default catches HTTPError -> error result w/ status=502 #366
access-api responses should have sourcemap-aware stack traces

The text was updated successfully, but these errors were encountered:

gobengo · 2023-01-18T21:32:09Z

The status: 530 in the HTTPError is pretty unusual. It could be cloudflare https://community.cloudflare.com/t/community-tip-fixing-error-530-error-1016-origin-dns-error/44264

gobengo · 2023-01-19T01:11:30Z

this PR more gracefully handles a situation where the ucanto proxy's upstream URL fetch errors with an HTTPError (so we don't get a HandlerExecutionError feat: access-api proxy.js has configurable options.catchInvocationError, by default catches HTTPError -> error result w/ status=502 #366
the first PR may add enough debug info to tell what's going on, but e.g. it can't add response.text or response.headers, and this PR adds temporary logging to show those, but is meant to be reverted after debug feat: access-api createStoreProxy uses patched fetch that will log any non-ok responses (to debug) #364

…y non-ok responses (to debug) (#364) …that logs unexpected responses with status=530 Motivation: * gather more info for [this error happening in production](#363 (comment)) Plan * gather info, then revert this PR

…or, by default catches HTTPError -> error result w/ status=502 (#366) , which defaults to something that catches HTTPErrors and rewrites to a status=502 error result Motivation: * relates to: #363 * instead of that behavior of getting a `HandlerExecutionError` (due to internally having an uncaught `HTTPError`), after this PR we should * not have the uncaught `HTTPError`, so no uncaught `HandlerExecutionError` * the result will have `status=502` and `x-proxy-error` with information about the proxy invocation request that errored (which will help debug #363) Limitations * This inlcudes in `result['x-proxy-error']` any info from the underlying `@ucanto/transport/http` `HTTPError`, but unless/until we put more info on that error, we still won't have the raw response object and e.g. won't have response headers. * but if those properties are ever added to `HTTPError`, they should show up in `x-proxy-error`

gobengo · 2023-01-19T19:58:56Z

phew I recovered response text from the mysterious 530 response:

<!DOCTYPE html>
<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->
<!--[if IE 7]>    <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endif]-->
<!--[if IE 8]>    <html class=\"no-js ie8 oldie\" lang=\"en-US\"> <![endif]-->
<!--[if gt IE 8]><!--> <html class=\"no-js\" lang=\"en-US\"> <!--<![endif]-->
<head>
<title>Origin DNS error | up.web3.storage | Cloudflare</title>
<meta charset=\"UTF-8\" />
<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />
<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\" />
<meta name=\"robots\" content=\"noindex, nofollow\" />
<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\" />
<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/main.css\" />


<script>
(function(){if(document.addEventListener&&window.XMLHttpRequest&&JSON&&JSON.stringify){var e=function(a){var c=document.getElementById(\"error-feedback-survey\"),d=document.getElementById(\"error-feedback-success\"),b=new XMLHttpRequest;a={event:\"feedback clicked\",properties:{errorCode:1016,helpful:a,version:1}};b.open(\"POST\",\"https://sparrow.cloudflare.com/api/v1/event\");b.setRequestHeader(\"Content-Type\",\"application/json\");b.setRequestHeader(\"Sparrow-Source-Key\",\"c771f0e4b54944bebf4261d44bd79a1e\");
b.send(JSON.stringify(a));c.classList.add(\"feedback-hidden\");d.classList.remove(\"feedback-hidden\")};document.addEventListener(\"DOMContentLoaded\",function(){var a=document.getElementById(\"error-feedback\"),c=document.getElementById(\"feedback-button-yes\"),d=document.getElementById(\"feedback-button-no\");\"classList\"in a&&(a.classList.remove(\"feedback-hidden\"),c.addEventListener(\"click\",function(){e(!0)}),d.addEventListener(\"click\",function(){e(!1)}))})}})();
</script>

<script defer src=\"https://performance.radar.cloudflare.com/beacon.js\"></script>
</head>
<body>
  <div id=\"cf-wrapper\">
    <div class=\"cf-alert cf-alert-error cf-cookie-error hidden\" id=\"cookie-alert\" data-translate=\"enable_cookies\">Please enable cookies.</div>
    <div id=\"cf-error-details\" class=\"p-0\">
      <header class=\"mx-auto pt-10 lg:pt-6 lg:px-8 w-240 lg:w-full mb-15 antialiased\">
         <h1 class=\"inline-block md:block mr-2 md:mb-2 font-light text-60 md:text-3xl text-black-dark leading-tight\">
           <span data-translate=\"error\">Error</span>
           <span>1016</span>
         </h1>
         <span class=\"inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed\">Ray ID: 78c2234cf35dfab6 &bull;</span>
         <span class=\"inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed\">2023-01-19 19:57:10 UTC</span>
        <h2 class=\"text-gray-600 leading-1.3 text-3xl lg:text-2xl font-light\">Origin DNS error</h2>
      </header>

      <section class=\"w-240 lg:w-full mx-auto mb-8 lg:px-8\">
          <div id=\"what-happened-section\" class=\"w-1/2 md:w-full\">
            <h2 class=\"text-3xl leading-tight font-normal mb-4 text-black-dark antialiased\" data-translate=\"what_happened\">What happened?</h2>
            <p>You've requested a page on a website (up.web3.storage) that is on the <a href=\"https://www.cloudflare.com/5xx-error-landing/\" target=\"_blank\">Cloudflare</a> network. Cloudflare is currently unable to resolve your requested domain (up.web3.storage).
            
          </div>

          
          <div id=\"resolution-copy-section\" class=\"w-1/2 mt-6 text-15 leading-normal\">
            <h2 class=\"text-3xl leading-tight font-normal mb-4 text-black-dark antialiased\" data-translate=\"what_can_i_do\">What can I do?</h2>
            <p><strong>If you are a visitor of this website:</strong><br />Please try again in a few minutes.</p><p><strong>If you are the owner of this website:</strong><br />Check your DNS settings. If you are using a CNAME origin record, make sure it is valid and resolvable. <a rel=\"noopener noreferrer\" href=\"https://support.cloudflare.com/hc/en-us/articles/234979888-Error-1016-Origin-DNS-error\">Additional troubleshooting information here.</a></p>
          </div>
          
      </section>

      <div class=\"feedback-hidden py-8 text-center\" id=\"error-feedback\">
    <div id=\"error-feedback-survey\" class=\"footer-line-wrapper\">
        Was this page helpful?
        <button class=\"border border-solid bg-white cf-button cursor-pointer ml-4 px-4 py-2 rounded\" id=\"feedback-button-yes\" type=\"button\">Yes</button>
        <button class=\"border border-solid bg-white cf-button cursor-pointer ml-4 px-4 py-2 rounded\" id=\"feedback-button-no\" type=\"button\">No</button>
    </div>
    <div class=\"feedback-success feedback-hidden\" id=\"error-feedback-success\">
        Thank you for your feedback!
    </div>
</div>


      <div class=\"cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300\">
  <p class=\"text-13\">
    <span class=\"cf-footer-item sm:block sm:mb-1\">Cloudflare Ray ID: <strong class=\"font-semibold\">78c2234cf35dfab6</strong></span>
    <span class=\"cf-footer-separator sm:hidden\">&bull;</span>
    <span id=\"cf-footer-item-ip\" class=\"cf-footer-item hidden sm:block sm:mb-1\">
      Your IP:
      <button type=\"button\" id=\"cf-footer-ip-reveal\" class=\"cf-footer-ip-reveal-btn\">Click to reveal</button>
      <span class=\"hidden\" id=\"cf-footer-ip\">2a06:98c0:3600::103</span>
      <span class=\"cf-footer-separator sm:hidden\">&bull;</span>
    </span>
    <span class=\"cf-footer-item sm:block sm:mb-1\"><span>Performance &amp; security by</span> <a rel=\"noopener noreferrer\" href=\"https://www.cloudflare.com/5xx-error-landing\" id=\"brand_link\" target=\"_blank\">Cloudflare</a></span>
    
  </p>
  <script>(function(){function d(){var b=a.getElementById(\"cf-footer-item-ip\"),c=a.getElementById(\"cf-footer-ip-reveal\");b&&\"classList\"in b&&(b.classList.remove(\"hidden\"),c.addEventListener(\"click\",function(){c.classList.add(\"hidden\");a.getElementById(\"cf-footer-ip\").classList.remove(\"hidden\")}))}var a=document;document.addEventListener&&a.addEventListener(\"DOMContentLoaded\",d)})();</script>
</div><!-- /.error-footer -->


    </div><!-- /#cf-error-details -->
  </div><!-- /#cf-wrapper -->

  <script>
  window._cf_translation = {};
  
  
</script>

</body>
</html>

…of up.web3.storage (#376) Motivation * work around #363 by not using the domain name that cloudflare workers can't resolve. Instead bypass that dns rule and just request the aws http gateway domain

gobengo · 2023-01-19T21:41:20Z

#status - we worked around this by using a different prod URL that goes directly to aws, but will leave this open until there is more update from my support ticket to cloudflare

Motivation: * cloudflare support asked for a URL to reproduce the undelrying cause of #363 * this adds a route just for that. once they use it to reproduce and diagnose, we remove it

gobengo · 2023-01-31T18:40:52Z

cloudflare support got back to me basically saying this is because it's resolving up.web3.storage using their internal DNS, which doesn't resolve the same as 'external dns', and that he put in a feature request to support it.... I think we will keep using the workaround indefinitely (avoid cloudflare dns alltogether by using https://3bd9h7xn3j.execute-api.us-west-2.amazonaws.com/ instead of https://up.web3.storage (only when requesting from inside cloudflare workers access-api)

…y non-ok responses (to debug) (#364) …that logs unexpected responses with status=530 Motivation: * gather more info for [this error happening in production](#363 (comment)) Plan * gather info, then revert this PR

…or, by default catches HTTPError -> error result w/ status=502 (#366) , which defaults to something that catches HTTPErrors and rewrites to a status=502 error result Motivation: * relates to: #363 * instead of that behavior of getting a `HandlerExecutionError` (due to internally having an uncaught `HTTPError`), after this PR we should * not have the uncaught `HTTPError`, so no uncaught `HandlerExecutionError` * the result will have `status=502` and `x-proxy-error` with information about the proxy invocation request that errored (which will help debug #363) Limitations * This inlcudes in `result['x-proxy-error']` any info from the underlying `@ucanto/transport/http` `HTTPError`, but unless/until we put more info on that error, we still won't have the raw response object and e.g. won't have response headers. * but if those properties are ever added to `HTTPError`, they should show up in `x-proxy-error`

…of up.web3.storage (#376) Motivation * work around #363 by not using the domain name that cloudflare workers can't resolve. Instead bypass that dns rule and just request the aws http gateway domain

Motivation: * cloudflare support asked for a URL to reproduce the undelrying cause of #363 * this adds a route just for that. once they use it to reproduce and diagnose, we remove it

This was referenced Jan 18, 2023

feat: access-api createStoreProxy uses patched fetch that will log any non-ok responses (to debug) #364

Merged

access-api: when store/upload proxy encounters an HTTPError, the proxied invocation should get an error result, not a HandlerExecutionError #365

Open

gobengo changed the title ~~access-api: invoking store/list in production errors with HTTPError~~ access-api: invoking store/list in production errors with HandlerExecutionError + HTTPError Jan 19, 2023

gobengo mentioned this issue Jan 19, 2023

feat: access-api proxy.js has configurable options.catchInvocationError, by default catches HTTPError -> error result w/ status=502 #366

Merged

gobengo mentioned this issue Jan 19, 2023

access-api sentry logs should show stack traces that are sourcemap-aware #367

Closed

gobengo self-assigned this Jan 19, 2023

gobengo mentioned this issue Jan 19, 2023

feat: upload-api-proxy production url is to aws http gateway instead of up.web3.storage #376

Merged

This was referenced Jan 20, 2023

feat: add /reproduce-cloudflare-error route to access-api #379

Closed

feat: add /reproduce-cloudflare-error route to access-api #380

Merged

gobengo closed this as completed Jan 31, 2023

gobengo mentioned this issue Jan 31, 2023

remove /reproduce-cloudflare-error route #413

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

access-api: invoking store/list in production errors with `HandlerExecutionError` + `HTTPError` #363

access-api: invoking store/list in production errors with `HandlerExecutionError` + `HTTPError` #363

gobengo commented Jan 18, 2023 •

edited

Loading

gobengo commented Jan 18, 2023 •

edited

Loading

gobengo commented Jan 19, 2023 •

edited

Loading

gobengo commented Jan 19, 2023

gobengo commented Jan 19, 2023 •

edited

Loading

gobengo commented Jan 31, 2023

access-api: invoking store/list in production errors with HandlerExecutionError + HTTPError #363

access-api: invoking store/list in production errors with HandlerExecutionError + HTTPError #363

Comments

gobengo commented Jan 18, 2023 • edited Loading

gobengo commented Jan 18, 2023 • edited Loading

gobengo commented Jan 19, 2023 • edited Loading

gobengo commented Jan 19, 2023

gobengo commented Jan 19, 2023 • edited Loading

gobengo commented Jan 31, 2023

access-api: invoking store/list in production errors with `HandlerExecutionError` + `HTTPError` #363

access-api: invoking store/list in production errors with `HandlerExecutionError` + `HTTPError` #363

gobengo commented Jan 18, 2023 •

edited

Loading

gobengo commented Jan 18, 2023 •

edited

Loading

gobengo commented Jan 19, 2023 •

edited

Loading

gobengo commented Jan 19, 2023 •

edited

Loading