CORS issue causing event stream error on gesis

[minrk]

Builds on gesis are failing with "Failed to connect to event stream" due to a CORS issue. The request to open the event stream is failing with:

[Error] Failed to load resource: Cross-origin redirection to https://notebooks.gesis.org/binder/build/gh/TomAugspurger/idp-results/master?binder_launch_host=https%3A%2F%2Fmybinder.org%2F denied by Cross-Origin Resource Sharing policy: Cannot use wildcard in Access-Control-Allow-Origin when credentials flag is true. (master, line 0)
[Error] EventSource cannot load https://gesis.mybinder.org/build/gh/TomAugspurger/idp-results/master?binder_launch_host=https%3A%2F%2Fmybinder.org%2F due to access control checks.
[Error] Failed to construct event stream – Event {isTrusted: true, type: "error", target: EventSource, …}
Event {isTrusted: true, type: "error", target: EventSource, currentTarget: EventSource, eventPhase: 2, …}Event
(anonymous function) (bundle.js:21152)

I'm not sure what setting is different on gesis that causes this. We should probably disable gesis until this works out, since all builds will appear to fail (the builds and launches do not appear to actually be affected, only the event stream)

[bitnik]

hi @minrk , can you help me here? I am not sure if I really understand the error. How can I produce it? So I can investigate what is wrong in our deployment.

[minrk]

I'm trying to understand the root cause myself. Binder deployments have Access-Control-Allow-Origin: * to allow anyone to connect. However, there is a conflict between this option and Access-Control-Allow-Credentials: true and/or sending requests withCredentials.

It is
There should be tw

[minrk]

What I don't understand is what is different about gesis that is triggering this, why the credentials flag would be different.

I think the best way to test this would be to add gesis to the staging federation so that you can try builds there. It's hard to reproduce when gesis is temporarily out of rotation.

[manics]

@bitnik Did anything change with your authenticated binderhub that might have inadvertently affected the public one?

[minrk]

You can reproduce this in the js console in a browser with:

evt = new EventSource("https://mybinder.org/build/gh/TomAugspurger/idp-results/master", {withCredentials: true})
evt.close()

what's weird is that we don't pass withCredentials: true, and without specifying that, I can't reproduce the issue from the console, even if I specify the exact URL that doesn't work from the same page that doesn't work

[betatim]

One thing that is different for Gesis compared to our other clusters is that there is a nginx which performs a redirect (30x) on requests we make

[minrk]

Ah, that has a very high probability of being it

[minrk]

since CORS queries tend to be made with OPTIONS requests, and may not be redirected the same

[minrk]

what URLs get those redirects?

(edit: apologies for using GitHub comments like a chat app)

[betatim]

https://github.com/gesiscss/orc/blob/master/load_balancer/sites-available/gesis_mybinder is the nginx config running "at" gesis.mybinder.org

[betatim]

Maybe we need to add OPTIONS to https://github.com/gesiscss/orc/blob/f657f2213e5f4d90762ed5311cfb304f280208f0/load_balancer/sites-available/gesis_mybinder#L29?

[minrk]

Yeah, worth a try. I'm not sure if OPTIONS can follow redirects, we may need to write the OPTIONS reply ourselves (it should just set the CORS headers and that's it)

[bitnik]

hey, I am sorry that I couldn't really follow you in this conversation. I even still cant reproduce it myself. According to what you wrote I understand that builds are failing at GESIS when it is done through federation, but I don't experience this neither.

I am adding now OPTIONS as Tim suggested.

[arnim]

Hi, we have just compared @minrk s example (TomAugspurger/idp-results => GKE):

evt = new EventSource("https://staging.mybinder.org/build/gh/TomAugspurger/idp-results/master", {withCredentials: true})
EventSource {url: "https://staging.mybinder.org/build/gh/TomAugspurger/idp-results/master", withCredentials: true, readyState: 0, onopen: null, onmessage: null, …}url: "https://staging.mybinder.org/build/gh/TomAugspurger/idp-results/master"withCredentials: truereadyState: 2onopen: nullonmessage: nullonerror: null__proto__: EventSource
(index):1 Access to resource at 'https://gke.staging.mybinder.org/build/gh/TomAugspurger/idp-results/master?binder_launch_host=https%3A%2F%2Fmybinder.org%2F' (redirected from 'https://staging.mybinder.org/build/gh/TomAugspurger/idp-results/master') from origin 'https://staging.mybinder.org' has been blocked by CORS policy: The value of the 'Access-Control-Allow-Origin' header in the response must not be the wildcard '*' when the request's credentials mode is 'include'.
gke.staging.mybinder.org/build/gh/TomAugspurger/idp-results/master?binder_launch_host=https%3A%2F%2Fmybinder.org%2F:1 GET https://gke.staging.mybinder.org/build/gh/TomAugspurger/idp-results/master?binder_launch_host=https%3A%2F%2Fmybinder.org%2F net::ERR_FAILED

and bitnik/simple-binder-repo-2 => GESIS

evt = new EventSource("https://staging.mybinder.org/build/gh/bitnik/simple-binder-repo-2/master", {withCredentials: true})
EventSource {url: "https://staging.mybinder.org/build/gh/bitnik/simple-binder-repo-2/master", withCredentials: true, readyState: 0, onopen: null, onmessage: null, …}
(index):1 Access to resource at 'https://gesis.mybinder.org/build/gh/bitnik/simple-binder-repo-2/master?binder_launch_host=https%3A%2F%2Fmybinder.org%2F' (redirected from 'https://staging.mybinder.org/build/gh/bitnik/simple-binder-repo-2/master') from origin 'https://staging.mybinder.org' has been blocked by CORS policy: The value of the 'Access-Control-Allow-Origin' header in the response must not be the wildcard '*' when the request's credentials mode is 'include'.
gesis.mybinder.org/build/gh/bitnik/simple-binder-repo-2/master?binder_launch_host=https%3A%2F%2Fmybinder.org%2F:1 GET https://gesis.mybinder.org/build/gh/bitnik/simple-binder-repo-2/master?binder_launch_host=https%3A%2F%2Fmybinder.org%2F net::ERR_FAILED
evt
EventSource {url: "https://staging.mybinder.org/build/gh/bitnik/simple-binder-repo-2/master", withCredentials: true, readyState: 2, onopen: null, onmessage: null, …}

on the first look the seem to behave the same.

[betatim]

Not sure I understand what is going wrong yet. As I was poking around things I noticed that we have https://gke.staging.mybinder.org/build/gh/binder-examples/requirements/master?binder_launch_host=https%3A%2F%2Fmybinder.org%2F. Does someone know why the binder_launch_host query parameter gets added (and where)?

[manics]

Sounds like it's related to this feature added to support Gesis:

• [MRG]: badge_base_url can be a function that gets the parameter from the HTTP RequestHandler binderhub#1031
• remove appendix and use new env vars from runtime #1284 (comment)

[bitnik]

#1307:

mybinder.org-deploy/images/federation-redirect/app.py

Line 195 in 9cba7bf

redirect = url_concat(host_name + uri, {'binder_launch_host': 'https://mybinder.org/'})

It is used to set BINDER_LAUNCH_HOST env variable correctly. If user is redirected from federation BINDER_LAUNCH_HOST is "https://mybinder.org/" or if user is launching a repo directly through GESIS Binder, it is "https://notebooks.gesis.org/binder/".

This env variable is used for the value of "Copy Binder Link" button. And also in https://github.com/manics/jupyter-offlinenotebook extention. That's all I remember.

[bitnik]

Not sure I understand what is going wrong yet.

@betatim does it mean that you can reproduce the error?

[betatim]

What I meant is that I didn't even get close to trying to reproduce it because I ended up with other questions first. I do get the error Min reported if I set withCredentials to true.

I will try again today with a locally running federation redirector which only contains Gesis. Otherwise trying to find a repo that gets redirected to the place you want is a bit tedious :)

Another thing I am unsure about: does the result you get from typing things into the console depend on which website you opened the console for? For example "open console on mybinder.org" vs "open console on gesis.mybinder.org" vs "..."?

[minrk]

FWIW, I'm able to reproduce this currently at https://staging.mybinder.org/v2/gh/binderhub-ci-repos/requirements/master which is currently binned to gesis. I can see it with Chrome and Safari, but not Firefox. Safari gives informative error messages about withCredentials, while Chrome only says "Failed to construct event stream".

When I run the federation-redirector on its own locally (on localhost), there do not appear to be issues. The main difference I can think of is that it's on http not https.

[minrk]

It doesn't make sense to set the binder_launch_host param on all redirected URLs (mostly build URLs), so I'm going to try reverting that part of #1307 to see if it helps.

[betatim]

On Firefox (74.0b4) and Chrome (79.0.3945.130 (Official Build) (64-bit)) I don't see any errors for https://staging.mybinder.org/v2/gh/betatim/first-binder/master.

[minrk]

@betatim I didn't see errors on my very first test with Chrome 79 this morning, but after restarting to update to 80 I did!

[bitnik]

@consideRatio thanks a lot for your inputs! I will try your suggestions out, when I am really able to debug this.

[betatim]

The 'Keep waiting' thing is (I think) an unrelated problem with the JupyterLab workspace.

https://staging.mybinder.org/v2/gh/binder-examples/requirements/master and https://staging.mybinder.org/v2/gh/ogrisel/notebooks/master?filepath=sklearn_demos/ames_housing.ipynb both give me a instant "error to connect to event source"

[betatim]

You are right! Privacy badger (a browser extension I use) is blocking the request to notebooks.gesis.org because it looks like it is a tracker. Privacy badger has a bunch of heuristics it uses to decide if requests look like "trackers" or not. Need to do some digging why/what the rules are. The last time we had a problem with appearing like a tracker it was related to setting cookies and loading an image. This isn't the case here but it will be something similar (I bet).

Nice find of the reason for the error message!

Just to confirm: if I disable all "anti tracking" extensions and settings in Firefox I "just works again" :)

[bitnik]

Does anyone have any recommendation what to do here as next step?

[bitnik]

I realized that in chrome I disabled privacy badger for mybinder.org some time ago for some reason I don't remember now. I enabled it again and had the same problem (notebooks.gesis.org is blocked on staging.mybinder.org, ex URL: https://staging.mybinder.org/v2/gh/ogrisel/notebooks/mastersss) as @betatim has. Then I installed privacy badger in Firefox, just to see if it behaves the same, and it didn't behave the same, it didn't block notebooks.gesis.org on staging.mybinder.org with the same URL.

Then today I was reading this post (https://github.com/EFForg/privacybadger/wiki/Find-out-why-Privacy-Badger-is-blocking-a-domain) to understand why privacy badger is blocking notebooks.gesis.org in chrome and here is the output for notebooks.gesis.org:

**** ACTION_MAP for gesis.org
gesis.org {
  "dnt": false,
  "heuristicAction": "block",
  "nextUpdateTime": 0,
  "userAction": ""
}
notebooks-test.gesis.org {
  "dnt": false,
  "heuristicAction": "",
  "nextUpdateTime": 1565035835885,
  "userAction": "user_cookieblock"
}
notebooks.gesis.org {
  "dnt": false,
  "heuristicAction": "block",
  "nextUpdateTime": 1582943208262,
  "userAction": "user_block"
}
**** SNITCH_MAP for gesis.org
gesis.org [
  "127.0.0.1",
  "0.0.0.0",
  "github.com"
]

To be honest this didn't helped me really. Later I found this issue and because of

To follow up on the last comment: if you've had PB installed since before we added the MDFP list, your badger might have learned to block wikimedia domains before that time, and never "un-learned" to block them if you didn't reset Privacy Badger's local storage. We can fix that problem by adding a migration to Privacy Badger's startup procedure, as we have with the yellowlist.

I decided to uninstall privacy badger in chrome and install it again and see what happens. And now it doesn't block notebooks.gesis.org anymore on staging.mybinder.org.

Now I am thinking about creating an issue on https://github.com/EFForg/privacybadger/.

[bitnik]

Here is the link to the issue: EFForg/privacybadger#2558

[arnim]

We've got a response on the Privacy badger issue, however, I'm not certain if / how fare we should go down debugging PB. Doing so slows us (in the best case) down to make the persistent BH code available in a usable fashion. In the worst case, it hinders us to do this all together since we have only so much time for this. I agree that we should put in 20x the effort so [we] can reach 1000x fewer people. Yet I wonder:

• how widespread this PB issue is,
• if there are other CORS or related issues blocking us and
• if sketching API v2 binderhub#1048 could (potentially) help without debugging PB.

Maybe this could help us to stay efficient.

[bitnik]

Btw @betatim if you still have the problem, could you also post the output of https://github.com/EFForg/privacybadger/wiki/Find-out-why-Privacy-Badger-is-blocking-a-domain into the issue? It might be helpful.

[bitnik]

I realized that in chrome I disabled privacy badger for mybinder.org some time ago for some reason I don't remember now.

Now I remember. One time I had problem with adblocker on mybinder.org, I didn't see any build/launch logs. Then I disabled blocker extensions (including Privacy Badger) for mybinder.org and turned adblocker off completely on the browser that I do tests.

And now just to test, I turned adblocker on and I don't see any launch/build logs on mybinder.org (e.g. https://mybinder.org/v2/gh/binder-examples/requirements/master). The weird thing is that if I disable adblocker only for mybinder.org, it doesn't help, I don't see any logs.

There is one more thing :) Because now I have Privacy Badger enabled for mybinder.org, I see that it is also blocking nbviewer.jupyter.org on loading page and I don't see any preview. Again I just seen it with https://mybinder.org/v2/gh/binder-examples/requirements/master

[betatim]

I think trying to understand this is worth it. Not so much because of the PB use-case (install base is probably small). However I think more and more browsers are shipping with (aggressive) ad/tracker/privacy settings on by default. At the last few updates Firefox informed me about doing more to block stuff automatically and I think Safari is also pretty aggressive. I saw a headline somewhere that Google was going to make Chrome so that it blocks most ads (except for Google whitelisted ads).

Over the last few months it has felt like the number of people complaining about "can't connect to eventstream" errors has increased. This was a mystery to me until we discovered this. Now I am wondering if the increased reports are related to the more aggressive ad blocker settings that browsers ship with by default.

I am wondering if it makes a difference if the "tracker domain" (in our case nbviewer or gesis) sets a cookie or not. The way the Facebook like button was used to track people is by embedding it in many many pages and having a unique cookie set on the domain that the like button was served from. This way evil-button.com could track you visiting fun.com and happyplace.de (if they both embedded the button).

I think if we fall into this pattern we need to reconsider how we do things because it is a losing battle. My prediction is that browsers will get ever more aggressive at blocking requests that match this pattern because they can't tell the difference to trackers.

Right now we serve the images and HTML from mybinder.org (via a proxy) but open an event stream from another domain (notebooks.gesis.org, gke.mybinder.org, etc). When the pod has launched we redirect the user to a different domain.

My hunch is that the fact that a website served from mybinder.org is making requests to notebooks.gesis.org (potentially with cookies?) is what triggers the tracker detection. There is no way to know that notebooks.gesis.org isn't track-all-notebooks.evil.com.

Three ideas:

1. figure out if there is a way to open the event stream without sending "any" information (no cookies, referrer, etc) and see if that (doesn't) trigger the tracker detection
2. proxy the event stream through the federation proxy (traffic goes from browser <-> fed proxy <-> actual site). More traffic on the proxy but everything appears to be on mybinder.org
3. not redirect gesis.mybinder.org to notebooks.gesis.org for the event stream endpoint but only for the other paths. It seems like that using a subdomain doesn't trigger the tracker detection (yet?)

[psychemedia]

In terms of browser extensions causing issues with MyBinder, using Ghostery Chrome extension (on a Mac), and mybinder.org as an untrusted site, builds often(?; not sure if it's always; will keep a close watch...) don't actually launch for me and build log remains blank.

Ghostery only reports one tracker blocked, and that's Google Analytics. (I'm on a free plan; not sure if I can get more detailed reports on what Ghostery does / sees if I sign up, subscribe etc.)

Works fine if I trust the site. Also note that this means that try.jupyter.org can fail as it tries to launch using MyBinder. (Another issue is I regularly see Too many users running https://github.com/ipython/ipython-in-depth! Try again soon. when testing from the try.jupyter.org link.)

[psychemedia]

With Ghostery enabled and site not trusted, if i try:

• https://mybinder.org/v2/gh/ipython/ipython-in-depth/master?filepath=binder/Index.ipynb: blank log, doesn't work;
• https://ovh.mybinder.org/v2/gh/ipython/ipython-in-depth/master?filepath=binder/Index.ipynb: works;
• https://gesis.mybinder.org/v2/gh/ipython/ipython-in-depth/master?filepath=binder/Index.ipynb: works;
• https://gke.mybinder.org/v2/gh/ipython/ipython-in-depth/master?filepath=binder/Index.ipynb: works;

[betatim]

Right now https://mybinder.org/v2/gh/ipython/ipython-in-depth/master?filepath=binder/Index.ipynb connects to gke.mybinder.org (for me).

[manics]

It's going to hub.mybinder.turing.ac.uk for me.

Edit: Tried in a second browser window and this time it went to hub.gke.mybinder.org

[manics]

To make testing cross-domain request easier is there any chance of adding an external domain as an alias for staging.mybinder.org, e.g. 104.197.11.66.xip.io ? Then every visit to the BinderHub at 104.197.11.66.xip.io would go to a different domain.

[psychemedia]

So even if I trust MyBinder site with Ghostery now, the top level mybinder.org site is broken. Launching from X.mybinder.org is fine.

[betatim]

What do you mean with "adding an alias"? We have notebooks.gesis.org in the staging federation so every second request should go to a "not subdomain" host.

[manics]

Gesis has a weight of 1, compared to 4 for gke.staging.

mybinder.org-deploy/config/staging.yaml

Line 128 in 29f4b21

weight: 1

.
Doesn't that imply only 20% of requests will go to Gesis?

[bitnik]

@manics I was also confused (#1363)

The size of the weight doesn't do anything (as long as it is above zero) when we are using rendezvous routing.

mybinder.org-deploy/images/federation-redirect/app.py

Lines 189 to 193 in 29f4b21

	if self.load_balancer == "rendezvous":
	host_name = rendezvous_rank(self.host_names, cache_key(path))[0]
	# "random" is our default or fall-back
	else:
	host_name = random.choices(self.host_names, self.host_weights)[0]

[bitnik]

#1344 (comment):

not redirect gesis.mybinder.org to notebooks.gesis.org for the event stream endpoint but only for the other paths. It seems like that using a subdomain doesn't trigger the tracker detection (yet?)

I have just made changes for this, so /build/ endpoint is not redirect anymore (e.g. https://gesis.mybinder.org/build/gh/binder-examples/requirements/master), so redirection happens on the same domain. And now I don't get "Failed to connect to event stream" error anymore, when Privacy Badger is enabled (my Privacy Badger recently learned to block notebooks.gesis.org on staging.mybinder.org). @betatim could you confirm this?

I know this doesn't resolve problems regarding privacy extentions in general, but at least brings GESIS Binder at the same level with other members in federation. So maybe we can bring GESIS Binder back to mybinder.org?

Edit: in order to test this change don't forget to use https://staging.mybinder.org/

[betatim]

I think we can close this now. The immediate problem is solved and the issue has gotten very long. I propose we open a new issue when we want to work on other privacy extension issues.

Thanks a lot for all the brains and hours spent on this everyone!