Headless Version

[tekicat]

Due to the fact PowderWeb is dependent on xdg/gui for intial login. Containerising such and amazing tool had become a nightmare. Is there any plan on releasing a docker container ?

[tekicat]

This may help with head start

FROM ubuntu:18.04

RUN apt update && \
    apt install -y git curl && \ 
    curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.34.0/install.sh | bash && \
    . ~/.bashrc && \
    nvm install 8.10.0 && \
    . ~/.profile && \
    git clone https://github.com/jaruba/PowderWeb.git && \
    cd PowderWeb && \
    npm install && \
    cd public && \
    npm install -g bower && \
    bower install --allow-root

WORKDIR /PowderWeb/public


ENTRYPOINT ["/bin/bash", "-c"]

CMD ["npm", "start"]

[jaruba]

A docker container would be cool. The UI could work as Web only, so a version of Powder Web without Electron should be possible.

But there is a big drawback with this, Electron isn't used just for the Windowed App, it's also used for select folder / file boxes (which Node.js has absolutely no native method for). But selecting folder / file can only be done locally anyway, so it could be changed to a string input that should point to folders / files..

My biggest limitation right now is finding the required time to work on this..

[jaruba]

I've renamed this to "Headless Version" and tagging it as a priority task. There might be many other good reasons to have a headless version. It will stay as a priority task unless I can find good enough reasons why Powder Web would dependent on Electron.

[jaruba]

@tekicat I've made a separate branch for the headless version. And did get a version of Powder Web to work completely without Electron.

I never tested it with Docker though, it would sure help if you could test it..

If the lines you previously posted are correct, to use the headless branch you should use:

FROM ubuntu:18.04

RUN apt update && \
    apt install -y git curl && \ 
    curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.34.0/install.sh | bash && \
    . ~/.bashrc && \
    nvm install 8.10.0 && \
    . ~/.profile && \
    git clone -b headless [email protected]:jaruba/PowderWeb.git && \
    cd PowderWeb && \
    npm install && \
    cd public && \
    npm install -g bower && \
    bower install --allow-root

WORKDIR /PowderWeb/public


ENTRYPOINT ["/bin/bash", "-c"]

CMD ["npm", "run", "start-headless"]

[tekicat]

@jaruba With the below docker file

FROM node:8-alpine3.10

RUN apk add git && \
    npm install -g bower && \
    addgroup -S powderweb && adduser -S powderweb -G powderweb  && \
    mkdir /app && \
    chown -R powderweb:powderweb /app/ 

USER powderweb

WORKDIR /app/

RUN git clone -b headless https://www.github.com/jaruba/PowderWeb.git && \
    cd PowderWeb && \
    npm install && \
    cd public && \
    bower install

WORKDIR /app/PowderWeb/public

ENTRYPOINT ["/bin/sh", "-c"]

CMD ["npm run start-headless"]

I get a 404 on opening http://localhost:3000/auth?token=<master key>

workspace/PowderWeb/docker# docker run -p 3000:3000 powderweb:18.04

> [email protected] start-headless /app/PowderWeb
> node main.js

[ '/usr/local/bin/node', '/app/PowderWeb/main.js' ]
master key: b1b506746eb647bdee953b35fe1abe13
connect ECONNREFUSED 127.0.0.1:80

Am I missing something here ?

[jaruba]

connect ECONNREFUSED 127.0.0.1:80

This seems to be the only error, and it says that something is trying to connect to localhost:80, everything should work on port 3000 so maybe this docker command is wrong somehow?

docker run -p 3000:3000 powderweb:18.04

You could also try using the LAN IP instead of localhost..

[Tekka90]

Same here, even before trying to open any browser. Just launching "npm run start-headless" get those 2 lines of output (master key + connect ECONNREFUSED 127.0.0.1:80)

Then if I try to open a browser (through internet in my case), I get "404 File Not Found" in the browser and on the server I get something like:
(node:249) TimeoutOverflowWarning: 9007199254740991 does not fit into a 32-bit signed integer.
Timer duration was truncated to 2147483647.
(Use node --trace-warnings ... to show where the warning was created)
(node:249) TimeoutOverflowWarning: 9007199254740991 does not fit into a 32-bit signed integer.
Timer duration was truncated to 2147483647.

I used a totally different DockerFile (to test) I doubt this is linked to Docker

[Tekka90]

FYI, I fiigured out this issue...
After the npm install, I had to run a "npm run build" to build the dist folder....

[Tekka90]

Coming back again for updates on the topic...
So after the build + removing this warning which was annoying (but not an issue)
There was an issue with video-js.
The version present in 0.9 (which is basically commenting some code) is working fine with this headless version.

So now I confirm, that the headless is working in docker. Need to merge all the new changes, and try to keep this version up to date in the future :)

[jaruba]

@Tekka90 Wow, I'm glad to hear u got it working, but the headless branch still needs to be updated to the latest version, and I should release it separately too.

Sorry for the late response, I was on vacation. I'll look into it a bit more too when I have some time, and I always wanted to make an official release of the headless version available.

[jaruba]

@Tekka90 I updated the headless branch to the latest version, but did not test it much, please test it if possible, and also paste the Dockerfile you used that worked for you so I can look into making an official Docker image for Powder Web.

[jaruba]

@Tekka90 if all that was needed was the dist folder, then we should also be able to use npm run build-front instead of npm run build, which is much more lightweight.

[Tekka90]

Hi @jaruba

1. Here is my DockerFile
   FROM node:alpine3.12

RUN apk add git &&
npm install -g bower &&
addgroup -S powderweb && adduser -S powderweb -G powderweb &&
mkdir /app &&
chown -R powderweb:powderweb /app/

USER powderweb

WORKDIR /app/

RUN git clone -b headless https://www.github.com/jaruba/PowderWeb.git &&
cd PowderWeb &&
npm install &&
cd public &&
bower install &&
sed -i 's/Number.MAX_SAFE_INTEGER/2147483647/g' /app/PowderWeb/server/index.js &&
npm run build

WORKDIR /app/PowderWeb

EXPOSE 3000
CMD npm run start-headless

I did the sed to get readable logs otherwise am flooded with warnings...

2. Thanks a lot for the merge to 0.9.0 !!

3. It is working for the navigation.However, the bin folder is not present, so for example when trying to add a Torrent, I get «/bin/sh: /app/PowderWeb/bin/torrent-thread/torrent-thread: not found »
   I m really not familiar with node.js, but I assume something is wrong in the build process ? 

[Tekka90]

I can add that the bin folder with its content is there .. In the container if I try to launch torrent-thread I get the same « not found » even though it is there...
I try to launch ffmpeg, no problem there, so something is not working with torrent-thread

[jaruba]

@Tekka90

well, i made torrent-thread too, so i'm sure it can be solved

what OS and what architecture are you running your Docker installation on?

[Tekka90]

@jaruba The host machine should not matter... the OS torrent-thread is running on in minimalist (alpine, node version as you can see in the Dockerfile.
In case it can matter, the host is a Debian Stretch. Nothing much running there outside of docker...
Would be useful to get more logs :) can we enable more verbosity in torrent thread ??
BTW if you want to debug, this is one the huge advantage of docker, you should be able to reproduce the issue just by running the container on your machine.

[jaruba]

@Tekka90 i'll try to answer as best i can:

• the warning you were getting: (node:249) TimeoutOverflowWarning: 9007199254740991 does not fit into a 32-bit signed integer. Timer duration was truncated to 2147483647. appears only if you use a newer version of node instead of node v8.12.0 (which is recommended for this project, as the readme states: Node v8.12.0 and NPM v6.4.1 were used for this project, it should work with newer versions too, but i can't vouch for stability on them)
• torrent-thread (along with all other binaries needed for this project, other binaries are: subtitles-thread, ffmpeg and youtube-dl) are all downloaded for the currently running OS and architecture during npm install, my guess is that the download failed for one of the next reasons: unsupported OS / architecture (doubtful), newer node / npm version used (possible), random download failure (possible: download server unreachable, download interrupted, etc.) or some other unexpected reason

There's a chance that torrent-thread might be there if you just rebuild the Docker image, but I strongly recommend that we use the expected Node and NPM versions for this in order to guarantee stability of the installation and app.

[Tekka90]

@jaruba Sorry I missed the part about needing a specific version of Nodejs sorry :(
So now I did use 2 different images with Node 8.12.0 + rpm 6.4.1. One based on Alpine, one based on Debian
In both cases, now I have an error at launch:

/app/PowderWeb/node_modules/youtube-dl/lib/youtube-dl.js:33
if (!fs.existsSync(ytdlBinary)) {
^

ReferenceError: ytdlBinary is not defined
at Object. (/app/PowderWeb/node_modules/youtube-dl/lib/youtube-dl.js:33:20)
at Module._compile (module.js:653:30)
at Object.Module._extensions..js (module.js:664:10)
at Module.load (module.js:566:32)
at tryModuleLoad (module.js:506:12)
at Function.Module._load (module.js:498:3)
at Module.require (module.js:597:17)
at require (internal/module.js:11:18)
at Object. (/app/PowderWeb/server/ytdl.js:2:14)
at Module._compile (module.js:653:30)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start-headless: node main.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start-headless script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

I assume something is missing in my Image :(

[Tekka90]

This is a certificate issue while npm install

Error: certificate has expired
at TLSSocket. (_tls_wrap.js:1116:38)
at emitNone (events.js:106:13)
at TLSSocket.emit (events.js:208:7)
at TLSSocket._finishInit (_tls_wrap.js:643:8)
at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:473:38)
npm WARN [email protected] requires a peer of eslint-plugin-jsx-a11y@^6.1.1 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of eslint@^2.10.2 || ^3 || ^4 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] No license field.
npm WARN The package npm-run-all is included as both a dev and production dependency.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/watchpack/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

• [email protected]

[jaruba]

I know: przemyslawpluta/node-youtube-dl#310

And the error you got from youtube-dl when running the app is because of the same reason, that error shows when the binary was not downloaded.

I'm more curious why it worked for you to install youtube-dl 7 days ago, as this certificate issue is pretty old, and should of happened then too.

[jaruba]

I think this is why torrent-thread didn't download either, because youtube-dl failed during npm install and all the install scripts after it were never started.

I think i may be able to fix this by disabling certificate validation in my fork of node-youtube-dl by setting strictSSL to false here:
https://github.com/jaruba/node-youtube-dl/blob/master/lib/downloader.js#L18

I'll try it a bit later tonight.

[jaruba]

@Tekka90 I made the change, but did not test it. I think this has a good chance of fixing everything, please try to build the docker image again at your earliest convenience.

[Tekka90]

@jaruba
Good news: Your last fix is working, now I am able to get the YouTube-dl installed correctly in alpine os
Bad news: back to error : torrent child process exited with code 127
/bin/sh: /app/PowderWeb/bin/torrent-thread/torrent-thread: not found

Again, the file /app/PowderWeb/bin/torrent-thread/torrent-thread does exist in the container, and if I try to launch it using command line, I get the same result !

For reference, this is the new Dockerfile:
`
FROM node:8.12.0-alpine

RUN apk add git &&
npm install -g bower &&
addgroup -S powderweb && adduser -S powderweb -G powderweb &&
mkdir /app &&
chown -R powderweb:powderweb /app/

USER powderweb

WORKDIR /app/

RUN git clone -b headless https://www.github.com/jaruba/PowderWeb.git &&
cd PowderWeb &&
npm install &&
cd public &&
bower install &&
cd .. &&
npm run build

WORKDIR /app/PowderWeb

EXPOSE 3000
CMD npm run start-headless
`

Could be linked to another issue I see at build time :

[email protected] postinstall /app/PowderWeb/node_modules/window-prebuilt
node install.js

(node:17) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added. Use emitter.setMaxListeners() to increase limit

[Tekka90]

FYI, linked to my previous message, the issue with torrent-thread is not linked to the warning...
I fixed the warning (npm install -g npm@next)
And still get the same issue :(
FYI:
ls -la /app/PowderWeb/bin/torrent-thread
total 44120
drwxr-xr-x 2 powderwe powderwe 4096 Jul 15 08:01 .
drwxr-xr-x 6 powderwe powderwe 4096 Jul 15 08:01 ..
-rwxr-xr-x 1 powderwe powderwe 45170269 Jul 15 08:01 torrent-thread

/app/PowderWeb $ /app/PowderWeb/bin/torrent-thread/torrent-thread
/bin/sh: /app/PowderWeb/bin/torrent-thread/torrent-thread: not found

[jaruba]

@Tekka90 the "not found" error can also happen if a html page / http error leaked into the torrent-thread file when it downloaded, is it possible to open /app/PowderWeb/bin/torrent-thread/torrent-thread with a text editor? and if it includes plain text instead of a binary, that could lead to a clue of what's going on

[Tekka90]

Yes I can open it, and the content is binary, I also tried to change rights (777), same issue...

I will try to build it from source and see if I have the same issue or not

[jaruba]

@Tekka90 building from source is more trouble then it's worth, i'll add some logs to the install scripts of torrent-thread and subtitles-thread so we can understand what's going on better.

[jaruba]

aw snap, yeah, that's definitely my fault

[jaruba]

@Tekka90 try now, it should be fixed

[tekicat]

@jaruba npm run build-front its working fine now.
On the side note, download playlist throws Unknown Error and the below in logs

basic torrent data
<Buffer 6d 61 67 6e 65 74 3a 3f 78 74 3d 75 72 6e 3a 62 74 69 68 3a 45 30 36 34 32 39 37 33 34 36 33 30 38 45 42 30 36 39 42 46 45 42 38 39 42 39 41 32 43 35 ... >

[Tekka90]

I confirm it is ok for me too !
And I also found an issue :)
Same video file from a torrent, downloading, start play:
-> Open on PC (through Chrome) -> all ok
-> From Safari on iOS -> Crash of the server with error:
events.js:183
throw er; // Unhandled 'error' event
^

Error: spawn /app/PowderWeb/bin/ffmpeg/linux/ia32/ffprobe ENOENT
at Process.ChildProcess._handle.onexit (internal/child_process.js:190:19)
at onErrorNT (internal/child_process.js:362:16)
at _combinedTickCallback (internal/process/next_tick.js:139:11)
at process._tickCallback (internal/process/next_tick.js:181:9)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start-headless: node main.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start-headless script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /home/powderweb/.npm/_logs/2020-07-16T12_51_14_719Z-debug.log

[jaruba]

@tekicat i'll check

@Tekka90 does /app/PowderWeb/bin/ffmpeg/linux/ia32/ffprobe exist?

[Tekka90]

Yes it does exist and I can launch it by hand this time, I guess again a missing lib, I m looking into it

[Tekka90]

@jaruba I tried in a Debian instead but get the same error...
Logs are not helping me much more:
13 verbose stack Error: [email protected] start-headless: node main.js
13 verbose stack Exit status 1
13 verbose stack at EventEmitter. (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:332:16)
13 verbose stack at emitTwo (events.js:126:13)
13 verbose stack at EventEmitter.emit (events.js:214:7)
13 verbose stack at ChildProcess. (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack at emitTwo (events.js:126:13)
13 verbose stack at ChildProcess.emit (events.js:214:7)
13 verbose stack at maybeClose (internal/child_process.js:915:16)
13 verbose stack at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)

What difference could there be between opening the same video in Safari (same issue in iOS, iPadOS and MacOS, all same issue) and Chrome ?

[jaruba]

@Tekka90 Some videos might already be compatible with playback in Chrome, but not on Safari, as they support different video types. If the file is already compatible, then transcoding should not be needed at all. The strange thing is that regardless of if the video is compatible or not, ffprobe is always used on all videos before trying to play them in a browser. (as ffprobe is used to get the video / audio encoding of the file)

But generally speaking, video playback wise, Safari only supports HLS, while Chrome (and other browsers) support MP4 containers.

[Tekka90]

Do you know of an open torrent containing a file not compatible with Chrome to force transcoding ?
But I doubt it is that, I did few test and I see in the log that it starts ffmpeg from chrome and all is ok in this context...
Really weird...
Do you have a way to test the same from your end to figure out if it is an issue in dockerisation or anything else ?

[jaruba]

@Tekka90 i don't think this is a transcoding issue either, it's a strange issue tbh, i'll try to find a way to reproduce it

[jaruba]

@tekicat @Tekka90 i can reproduce both your issues, "download playlist" only seems to work locally for some reason, and safari crashes at ffprobe (Safari on both OSX and iOS does this apparently)

[jaruba]

@tekicat I fixed "Download Playlist" on remote devices, you can try it out when you have time.

[jaruba]

@Tekka90 your issue is a complete mystery to me though, if i play in the web browser from Chrome, the command:
/app/PowderWeb/bin/ffmpeg/linux/ia32/ffprobe -show_streams -show_format http://127.0.0.1:6884/5

goes through without any issues

then if i play in the web browser from Safari, the exact same command is made and it gives the ENOENT error 🤦

[jaruba]

TBH, i don't see how the same command can have 2 different results depending on the client browser, I'm wondering if something else isn't happening here.. maybe docker is the one that's crashing, and the ENOENT error is the result of docker crashing for some unknown reason.

[jaruba]

Nope, this is not a Docker specific issue, I can 100% crash it the exact same way even while running the headless version locally without Docker. 😳

I've totally tested in Safari in the past, so this comes as a complete surprise for me now.

[jaruba]

@Tekka90 I think I fixed it, try it now.

There seems to be another issue with torrent db persistence, but that's less of a problem, i hope.

@tekicat @Tekka90 Please both confirm that your issues are resolved now.

[Tekka90]

Hi @jaruba
I confirm it is working fine now :)
Thanks for your hard work, I hope you will be rewarded with more success around this service which looks very promising!
I will do more testing and create new issues in case I find some, I don't have a lot of time at the moment to deep dive :( but I will help as I can !

[Tekka90]

Sorry I found a bug, I wanted to use the "donate" button, it is only sending you to /# with nothing else :)

[jaruba]

@Tekka90 That's a nice find actually, as Powder Web was initially built with a GUI version to be used locally, pressing external links makes a server call to open the link (as the GUI version is not a common browser)

I obviously see now how this logic for opening links was essentially flawed. I'll fix it so external links work remotely too.

Btw, I'm not sure if you know this, but when you run the Docker image, it shows a master key in the logs. This master key changes every time you run it, and the master account can be accessed by going to:
http://localhost:3000/auth?token=${masterKey} where ${masterKey} needs to be replaced with the master key token

By accessing the master account, you get many more administrative specific settings.

I'll look into adding Chromecast support to the web player too.

[Tekka90]

@jaruba Yes I know about the masterkey :)
I also set up my Jackett service and it works great with the app.
On my end, the service is now running on a dedicated server I own (in a data center) and am using it 100% remotely through internet.
The server is quite powerful so transcoding is not an issue, and bandwidth is off course crazy good so it makes powder web a quite interesting tool...

[jaruba]

@Tekka90 Cool setup you got, that's one of the many ways this project was meant to be used, personally I use it locally to just start torrents directly in my preferred video player, and it's faster then any other project I ever tried before for this purpose. I also use it to stream to different devices around the house, even used it to stream lower res transcoded videos (to save on data usage) to the tablet in my car when i got stuck in traffic jams. It's pretty much the swiss army knife of streaming.

[jaruba]

I've made quite a few important changes:

• Fixed external links in browsers (donate link, jackett installation guide link, etc)
• Fixed DB persistency issues
• Secured external links on the server side (for local GUI usage though, should not affect the headless version)
• Show update message in Web UI for the primary user: this is headless version specific, it now shows a message inside the Web UI dashboard only to the primary user (very first user account that is created) announcing if a new update is available
• Re-styled the authentication page so it is now dark

Please test it when possible, some of these (db persistency and the update message) are important to have in the headless version imo.

[jaruba]

Although I believe that the DB persistency issue is solved now in case of restarting or writing failures, I'm still uncertain how to handle DB and downloaded content persistency through docker image updates, because afaik docker doesn't really update, it just creates new images.

Is updating internally (through the app, which has not yet been implemented) the only way to preserve the current user data and downloads with docker? Any tips in regards to this topic would be appreciated, as you have probably used docker more then I have.

[Tekka90]

Hi @jaruba
I built a new Image with last code and it so far so good (donate link is now working ;) )

Regarding persistence of the data, here is how it works in Docker.
-> Once you start a container, you create some "volumes" (won't get into Docker details) which will leave the same life as the container. Meaning on restart of a same container, data remains there...
-> Now this is not recommended, because drop of container -> lost of data + you don't really manage easily where your data is saved (especially when you are keeping big files ...)
-> Solution is to "mount" volumes. If you look at my DockerFile + Docker run, this is the way I do it:

1. In Dockerfile, I create 1 symlink to /home/powderweb/.local/share/PowderWeb to /app/Data
2. in Docker run I specify a folder on my host I mount in /app/Data. Then the data is stored in my host directly and mounted at boot time of the container (I did that also for the .npm folder to get the logs)

Then will come the question "what do we put in the image ?" Compiled code ready to use but not easy to update without a new image (usually my preferred solution, and what is done now). Or image ready to build the code, downloading a prepackaged version of the code (prebuild somewhere). Or worse case an image ready to build which would git clone the code and build at start...

[jaruba]

In Dockerfile, I create 1 symlink to /home/powderweb/.local/share/PowderWeb to /app/Data

But then you are only persisting your user data, not the downloaded videos too, as you did not create a symlink for the downloaded videos.

The items would still show in the dashboard this way (as they are part of the user data), and I believe they would even be playable as it would just restart the download every time you play something.

[Tekka90]

You are right, data is not kept in my case
But if you want to, you can mount the /tmp/PowderWeb folder too.... (although /tmp is not really the right place then ...)

[jaruba]

/tmp is definitely not the right place, but it makes sense for the temp folder to be the default. I still need to fix setting the download folder to something else in the headless version: #51

I guess as long as the user data is persisted properly, and users can change the download folder, that should be enough for users to create a symlink for the download folder if they want to.

I'll might change this in the future if users start complaining about it though, but from my experience, the temp folder being the default is better for user experience.