Docker update broke everything #1
Comments
|
Fixed in 0bb65ae. Complain below. TL;DR FxA is under heavy development. It is quite unstable.Config files are not constent and startup methods of docker and pm2 quite differs . And they do not care about self hosting at all (I think). |
|
Thanks for the quick fix, it's fine again now. It's sad that they don't really care to maintain it in a compatible fashion. What do you recommend, how frequently should the stack be updated? |
|
I'm not very sure. If there're feature or functional or security updates , it might be fine. However if the updates are about configs or startup scripts, I have no better idea than opening an issue here when failed to start up. :-( |
|
Okay, thank you so much for the maintenance, I hope the repo will take off as the only self-hostable FxA stack! :) Maybe it's a good idea to post it on https://www.reddit.com/r/selfhosted/ then later on HackerNews (Show HN) when you think it is the time to get more attention or contributors. Also, the numerous topics we've talked about in the original issue thread, maybe could be here as separate issues along with the todo items in the README. I'd gladly create them, I just don't want you to feel I'm giving you orders :D I'm so honored you took all this together! |
|
I think it's not the time to spread it widely. I'm current working on |
|
Wow, so many updates on the master branch, it seems you made quite a progress! My instance was not updated since we've talked here, I'll look into what has changed and preferably do the update this afternoon. Anything to watch out specifically besides the obvious migration to the YAML config? |
|
I got some unexpected output after setting up the config yaml and running the init:
Here is the diff of the fix with some missing whitespaces added as well: diff --git a/init.sh b/init.sh
index 0981c20..b15b801 100755
--- a/init.sh
+++ b/init.sh
@@ -140,7 +140,7 @@ HERE
# TODO: yq r only once
if test $(yq r config.yml option.webext_storagesync.enable) == "true" ; then
- echo '"webextensions.storage.sync.serverURL": "https://$KINTO_SUB.$DOMAIN_NAME/v1"'
+ echo " \"webextensions.storage.sync.serverURL\": \"https://$KINTO_SUB.$DOMAIN_NAME/v1\""
fi
echo -e "\e[0m" #reset
@@ -155,13 +155,13 @@ cat <<HERE
"identity.fxaccounts.remote.webchannel.uri":"https://$CONTENT_SUB.$DOMAIN_NAME/",
"identity.sync.tokenserver.uri": "https://$SYNC_SUB.$DOMAIN_NAME/token/1.0/sync/1.5",
-APPEND/PREPEND https://$CONTENT_SUB.$DOMAIN_NAME to "webchannel.allowObject.urlWhitelist"
+ APPEND/PREPEND https://$CONTENT_SUB.$DOMAIN_NAME to "webchannel.allowObject.urlWhitelist"
HERE
echo -e "\e[0m" #reset
-if test $(yq r config.yml mail.type.enable) == "localhelper" ; then
+if test "$(yq r config.yml mail.type.enable)" == "localhelper" ; then
echo -e "\e[32m Check sigincode \e[0m"
echo -e "\e[33m"
cat <<HERE |
|
After starting the stack with docker-compose with all services enabled, 3 containers are in restart loop: The auth server log is all about this output: Content server has no output, and nginx doesn't honor the debug logs option in the config, need to manually edit the compose file. Anyways, it says it misses the content server, probably because it's in a restart loop: |
|
I've found some keys and comments in this old PR, maybe it helps: https://github.com/mozilla/fxa-oauth-server/pull/368/files |
|
Sorry for the inconvince. |
|
They use a docker image DO NOT contains pre-generated publickey.json key.json and other keys in fxa-auth-server. AND also the docker image does not contain So i need to make a breaking change for fxa v1.173+ , i made a branch
Again |
|
Thanks for your review on and i decide not to set fxa_version to latest finally. I can't always keep trace with mozilla's development spped.
|
what a coincidence. I'm struggling with the latest fxa too at the quite same time.
Basically. I implement all features discuesed in mozilla/fxa#3652. Reverse proxies, subdomain name ,certs , firefox send, webextension storage.sync , firefox notes. Thanks for giving a try. :) and also questions are welcomed. |
That's so sad. However, until it is open-source and can be debugged how it works and reproducible locally, we can keep up with their changes! :) And maybe through this repo, self-hosting can get intentional support from the core team. I can imagine they could even learn from these efforts to create a better hosting alternative for at least dev envs or maybe for production, too.
That's okay and completely normal, they are more than one dev :) This is also why I didn't update my FXA for ~1.5 months, I didn't want to complain about something going south with some rouge updates :D Today was the day that I finally found suitable to look into it what's new around here.
And it's wonderful! I could immediately update the config file with the custom subdomains, etc. really good job! By replacing the custom stuff with Now, it's none! :) (With one temporary exception, see below) I switched to the master branch, and as I use SSL, I've found a bug that keeps nginx from starting: The fix is simple: diff --git a/_init/nginx/kinto.conf.tmpl b/_init/nginx/kinto.conf.tmpl
index 1507383..0ab1301 100644
--- a/_init/nginx/kinto.conf.tmpl
+++ b/_init/nginx/kinto.conf.tmpl
@@ -3,7 +3,7 @@ server {
listen 443 ssl;
listen [::]:443 ssl;
ssl_certificate /certs/kinto.cer;
- ssl_certificate_key /certs/kinto.cer;
+ ssl_certificate_key /certs/kinto.key;
location / { |
|
Question: the Notes app is although enabled in the config, it is not yet compiled into the docker-compose? Suggestion: maybe when Send is enabled, the |
|
Hmm, the Send login doesn't work with a final redirect to The JS console says:
I did not tamper with any Firefox CSP settings in about:config, all default. Maybe an Nginx custom CSP header could solve it. |
|
Send loads with this CSP header, and as it doesn't contain the custom self-hosted URL, it blocks the login:
I hid it as a quick fix to prove it is the source, and now the login works. A bit drastic solution, and effectively defeats the purpose of CSP headers in a preferably secure E2EE app :( diff --git a/_init/nginx/send.conf.tmpl b/_init/nginx/send.conf.tmpl
index fe8ecdf..b590a8e 100644
--- a/_init/nginx/send.conf.tmpl
+++ b/_init/nginx/send.conf.tmpl
@@ -8,6 +8,8 @@ server {
location / {
proxy_http_version 1.1;
proxy_pass http://send:1443;
+
+ proxy_hide_header Content-Security-Policy;
}
location /api/ws {
proxy_http_version 1.1; |
do you mean note webextension/android client ?
blame on these lines fxa-selfhosting/docker-compose.tmpl.yml Lines 619 to 620 in 5f55b74 however if you ** do ** use prod env , csp are always wrong. so your solution seem the best. or we can set correct CSP via nginx too.
good idea! |
I don't know, I meant no running containers for https://github.com/mozilla/notes/ I thought this is why there is a note block in the config. It might be completely unrelated though.
After some tinkering, I found a reasonably good solution. Hiding the original header and providing a new one. I could not calculate the script nonce value, so I needed to use diff --git a/_init/nginx/send.conf.tmpl b/_init/nginx/send.conf.tmpl
index fe8ecdf..d5bbfb2 100644
--- a/_init/nginx/send.conf.tmpl
+++ b/_init/nginx/send.conf.tmpl
@@ -8,6 +8,9 @@ server {
location / {
proxy_http_version 1.1;
proxy_pass http://send:1443;
+
+ proxy_hide_header Content-Security-Policy;
+ add_header Content-Security-Policy "default-src 'self'; connect-src 'self' https://${SEND}.${NGINX_DOMAIN_NAME} https://${OAUTH}.${NGINX_DOMAIN_NAME} https://${PROFILE}.${NGINX_DOMAIN_NAME}; img-src 'self' https://firefoxusercontent.com https://secure.gravatar.com; script-src 'self' 'unsafe-inline'; form-action 'none'; frame-ancestors 'none'; object-src 'none'; report-uri /__cspreport__";
}
location /api/ws {
proxy_http_version 1.1;
diff --git a/docker-compose.tmpl.yml b/docker-compose.tmpl.yml
index d9de2b9..ae63bb1 100644
--- a/docker-compose.tmpl.yml
+++ b/docker-compose.tmpl.yml
@@ -590,7 +590,7 @@ services:
#@ command = []
#@ command.append('/bin/sh -c ' + '"' + "envsubst '$$NGINX_DOMAIN_NAME $$CONTENT $$AUTH $$OAUTH $$PROFILE $$SYNC' < /etc/nginx/conf.d/fxa.conf.tmpl > /etc/nginx/conf.d/fxa.conf ")
#@ if data.values.option.send.enable == True:
- #@ command.append("envsubst '$$NGINX_DOMAIN_NAME $$SEND' < /etc/nginx/conf.d/send.conf.tmpl > /etc/nginx/conf.d/send.conf ")
+ #@ command.append("envsubst '$$NGINX_DOMAIN_NAME $$SEND $$OAUTH $$PROFILE' < /etc/nginx/conf.d/send.conf.tmpl > /etc/nginx/conf.d/send.conf ")
#@ end
#@ if data.values.option.notes.enable == True or data.values.option.webext_storagesync.enable == True:
#@ command.append("envsubst '$$NGINX_DOMAIN_NAME $$KINTO' < /etc/nginx/conf.d/kinto.conf.tmpl > /etc/nginx/conf.d/kinto.conf ")Maybe with some Lua code, the new domains could be injected into the original header instead of forging a new one from scratch. This way the nonce could be calculated by the Send backend as usual, and no need for the unsafe-inline directive. It would be just as safe as the original. A couple of links if you want to look into it, I somehow feel it would worth the effort to create ~5-6 lines of Lua code inside the Send nginx config to solve this: |
|
Oh, and one thing, the postgres_data folder is not on gitignore :D |
notes app use kinto to store data. so there's no notes container.
can we prepend or append our to the origin one ? i'm not quite familar with nginx , maybe possbile? |
|
Yes, through the Lua module, we can modify headers on the fly. However, the Lua module is not included in the original nginx image, so I needed to add it in the command section of the compose. Since it installs a new base config, the gzip on directive is redundant, and it needs to be removed otherwise the container fails to start. I also commented out the logging driver none from the compose template as it made me go crazy to not see the logs while debugging nginx and regenerating the config many times. Here is the final solution: diff --git a/_init/nginx/fxa.conf.tmpl b/_init/nginx/fxa.conf.tmpl
index 85cf2cb..945dcd7 100644
--- a/_init/nginx/fxa.conf.tmpl
+++ b/_init/nginx/fxa.conf.tmpl
@@ -1,4 +1,3 @@
-gzip on;
gzip_types
text/css
text/javascript
diff --git a/_init/nginx/send.conf.tmpl b/_init/nginx/send.conf.tmpl
index fe8ecdf..f338275 100644
--- a/_init/nginx/send.conf.tmpl
+++ b/_init/nginx/send.conf.tmpl
@@ -8,6 +8,30 @@ server {
location / {
proxy_http_version 1.1;
proxy_pass http://send:1443;
+
+ # replace the original Mozilla domains to our custom domains in the original CSP header directives to make the login work
+ # @see: https://github.com/openresty/lua-nginx-module/blob/master/README.markdown#ngxheaderheader
+ # @see: https://github.com/openresty/lua-nginx-module#ngxresub
+ header_filter_by_lua_block {
+ local newstr, n, err = ngx.re.sub(ngx.header["Content-Security-Policy"], "(connect-src 'self').*?;", "$1 https://${SEND}.${NGINX_DOMAIN_NAME} https://${OAUTH}.${NGINX_DOMAIN_NAME} https://${PROFILE}.${NGINX_DOMAIN_NAME} https://${CONTENT}.${NGINX_DOMAIN_NAME} wss://${SEND}.${NGINX_DOMAIN_NAME};")
+ if newstr then
+ ngx.header["Content-Security-Policy"] = newstr
+ ngx.header["X-Content-Security-Policy"] = newstr
+ ngx.header["X-WebKit-CSP"] = newstr
+ else
+ ngx.log(ngx.ERR, "error: ", err)
+ return
+ end
+ local newstr, n, err = ngx.re.sub(ngx.header["Content-Security-Policy"], "(img-src 'self').*?;", "$1 https://${SEND}.${NGINX_DOMAIN_NAME} https://${PROFILE}.${NGINX_DOMAIN_NAME} https://secure.gravatar.com;")
+ if newstr then
+ ngx.header["Content-Security-Policy"] = newstr
+ ngx.header["X-Content-Security-Policy"] = newstr
+ ngx.header["X-WebKit-CSP"] = newstr
+ else
+ ngx.log(ngx.ERR, "error: ", err)
+ return
+ end
+ }
}
location /api/ws {
proxy_http_version 1.1;
diff --git a/docker-compose.tmpl.yml b/docker-compose.tmpl.yml
index d9de2b9..7e13fa1 100644
--- a/docker-compose.tmpl.yml
+++ b/docker-compose.tmpl.yml
@@ -583,18 +583,21 @@ services:
- logging:
- driver: "none"
+ #! logging:
+ #! driver: "none"
#! use envsubst only replace domain name https://github.com/docker-library/docs/issues/496#! issuecomment-186149231
#! build command!
#@ command = []
#@ command.append('/bin/sh -c ' + '"' + "envsubst '$$NGINX_DOMAIN_NAME $$CONTENT $$AUTH $$OAUTH $$PROFILE $$SYNC' < /etc/nginx/conf.d/fxa.conf.tmpl > /etc/nginx/conf.d/fxa.conf ")
#@ if data.values.option.send.enable == True:
- #@ command.append("envsubst '$$NGINX_DOMAIN_NAME $$SEND' < /etc/nginx/conf.d/send.conf.tmpl > /etc/nginx/conf.d/send.conf ")
+ #@ command.append("envsubst '$$NGINX_DOMAIN_NAME $$SEND $$OAUTH $$PROFILE $$CONTENT' < /etc/nginx/conf.d/send.conf.tmpl > /etc/nginx/conf.d/send.conf ")
#@ end
#@ if data.values.option.notes.enable == True or data.values.option.webext_storagesync.enable == True:
#@ command.append("envsubst '$$NGINX_DOMAIN_NAME $$KINTO' < /etc/nginx/conf.d/kinto.conf.tmpl > /etc/nginx/conf.d/kinto.conf ")
#@ end
+ #@ command.append("apt update")
+ #@ command.append("apt autoremove -y")
+ #@ command.append("apt install -y nginx-common nginx-full libnginx-mod-http-lua libnginx-mod-http-headers-more-filter")
#@ command.append("nginx -g 'daemon off;'" + '"')
command: #@ "&& ".join(command)
restart: unless-stoppedYou can test the old and the new headers with the following command, Lua rocks! :) curl -I https://send.local |
|
Updated the previous diff with another Lua block to replace the img-src as well, otherwise, the profile picture would not show up after login. |
|
Updated the previous diff with adding the CONTENT url to the connect-src, otherwise, some metrics-flow requests to the www subdomain would be blocked. I think I'm done now :D |
|
Hello @immanuelfodor Since they merged my PR, now Note:
|
|
Hi, I've checked all the changes since 78e054a, all looks good, and impressive solution with the Send upstream changes! :) Updated the whole stack, config and compose alike, it works fine with v1.179.0 out of the box. It feels like the project is somewhere around the beta stage 🚀 |
|
Hi @jackyzy823 , it seems FF 83 has broke the sync :(
When I sign in or sign out + sign in, the account seems fine but as soon as I turn on sync, the above message appears. $ firefox -v
Mozilla Firefox 83.0I tried updating from v1.179.0 to v1.186.0 with no luck, it's the same. I also tried using |
|
Have you try to force update synserver's docker image? From FF80 useOAuthForSyncToken is set to true by default. see https://cat-in-136.github.io/2020/08/diff-between-firefox-800-beta-7-default.html if your about:config If previous method do not help. Could you provide your about:sync-log 's error log. And i'll try to follow up with FxA and other related project (syncserver etc...) |
|
Hmm, my $ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx latest bc9a0695f571 6 days ago 133MB
redis latest 74d107221092 13 days ago 104MB
mysql/mysql-server 5.6 853cb833f211 6 weeks ago 238MB
mozilla/syncserver latest cb31d5dbb897 6 weeks ago 447MB
mozilla/send latest 17fad6a03bfe 2 months ago 327MB
mozilla/fxa-profile-server v1.186.0 b271dc1f2e58 3 months ago 1.04GB
mozilla/fxa-content-server v1.186.0 3d5149bb15f6 3 months ago 1.04GB
mozilla/fxa-auth-server v1.186.0 483619d8c3fb 3 months ago 1.04GB
mozilla/fxa-auth-db-mysql v1.186.0 5f775075ab74 3 months ago 1.04GB
mozilla/browserid-verifier v1.186.0 2ac4b70be132 3 months ago 1.04GB
mozilla/pushbox latest d2ff07b96d59 5 months ago 143MBAnd here is my sync log after a logout-login pair. There was no error before turning sync on, the file appeared right after enabling sync and then instantly asking for signing in again. Actual domain names are replaced with local, and based on the bad requests, it should be something oauth related, just as you thought. |
|
Well . that's weird. with the exact same config as yours , i could sync normally. Could you please use a clean firefox ( all configs set to default) and only change these configs and try again . |
|
I tried starting a new FF instance with an empty profile, it starts up fine as a brand new FF: mkdir -p /tmp/mozilla-temp-profile/
firefox --profile /tmp/mozilla-temp-profile/Set up the above options in about:config but it wouldn't even login, an infinite loader is displayed:
Inspecting the network requests of the login page, there is none that would contain my email address, just the ones to load the page and then the anonymized metrics:
Restarting with the same temp profile path helps, the account is recognized. Maybe this behavior is normal after entering custom FXA URLs:
However, the account menu is instantly yellow:
Clicking that menu takes back to the sync page with no option to turn on sync but to reconnect:
Reconnect won't help, it is asking for reconnect instantaneously. |
|
then i have no idea. 😞 maybe you need to set fxa-auth-server 's log level to debug and see what it complains about. |
|
I've set it to debug, and anonymized the output with search and replace to be consistent, maybe you can make something out of it: fxa-auth-server-debug.log The only thing I see is that I get a flow id but it is somehow getting lost and starts to be missing in later calls. |
|
I'll try to dig into it. And have you try a totally clean test ?
if this works. then maybe problem in database. else maybe in my scripts. I would be thankful to you if you could do this test for me. some problem in re-init: |
|
Oh my, you're a genius, why I haven't thought of this before I don't know :D Starting over with a clean project clone helped logging in even with the existing desktop profile, now it syncs fine. This means something was indeed bad in the database, I think. The yq:3 addition was also much appreciated to avoid broken init.sh output. One thing is left, logging in with Firefox Android. I have the Beta app, so when I tap the Firefox logo 5 times on the about page, a few hidden settings are revealed:
Bottom of the settings, two new menus but these are either empty or containing nothing useful:
On the other hand, over the top, there are two new URL fields:
No help is displayed, just an input box in a popup with the same title as the button. When I enter the https://www.domain.local (for account) and https://token.domain.local (for sync) URLs, tapping the sign in with email option takes to a page which is infinitely loading. The page is the https://www.domain.local/authorize?.... page with lots of parameters in the query. In desktop mode, it also displays the Mozilla logo at the bottom of the loading page. It looks like as if some resources are not downloaded or JS is not executed on the login page. However, there are no |
|
selfhosting fxa for firefox beta (aka fenix ) is not supported . because they hardcoded urls that could run content_scripts for login. i try to make this work , but failed. ========== And about database. if you are interested and want to merge data back if needed . you could try to find out which database caused the issue. (which is time consuming). |
|
This is so sad, the whole point of sync was desktop<->mobile for me :( Since the Fenix update, I started to use self-chat messages to transfer links between the platforms, and only synced the desktop in hope mobile would be solved once. Although desktop sync broke a month ago as well, we could fix it now, but not having Fenix is still heartbreaking. Yesterday, after the auth debug, I was on the brink of shutting the whole stack down, but now that desktop works, it's saved but still a bit useless. I really don't get why Mozilla is making this hard to self-host the open-source stack and make everything work together. They might be really just after the profits, and don't care for the community wholeheartedly. I really hope the linked issue gets some attention, and they make the Beta app able to whitelist the necessary webchannel URL or don't validate the source. |
|
if you insist on desktop <-> mobile sync, you could downgrade fenix to fennec (version 68 lts and more extension can be used , that's also the reason why i don't upgrade to fenix). |
|
What kind of data loss? A one-time loss upon starting to sync, or losing data constantly? And what data is being lost? Like browser history or extension data? Maybe I just need to setup all extensions on mobile manually? It's an interesting option to downgrade, I'd like to know more about this data loss. |
|
In android, application downgrade will remove all data under this app. (just like remove the app and install it again) |
|
I see, the app data. Thanks! |
Hi, I've been using the
masterbranch with delight since we talked at mozilla/fxa#3652 but I did a docker-compose pull to update the images to get any security etc fixes, and it broke everything, now the stack is in a restart loop. I think the main problem is that the mysql container wouldn't come up, and then all dependent containers fail, too.I'd gladly debug the script but there is no script in the first place :D Any idea how to get the stack working again?
The text was updated successfully, but these errors were encountered: