Formerly live videos fail to download: "failed to download [download] ... does not pass filter (live_status=?not_live); skipping" #188
Comments
holta
added
bug
The error suggests the video is still a live video. On YouTube, the video is advertised as a finished live stream. I bet the live_status column in media table (xklb-metadata) says it's still a live. |
|
Progress! @nzola please also take note if recently "live" YouTube videos download OR do not download — in case these's a clear pattern as to which succeed and which fail? 🙏 |
@holta @deldesir I understand that these videos were live when I tried to download them??? I will try downloading them again tomorrow, Sunday and give the report. |
|
@deldesir I tried to download this playlist now: https://www.youtube.com/@TOPCONGOFM/playlists It failed. Just remembered. I downloaded the same playlist yesterday, it failed PUBLISHING TO URL... https://dpaste.com/H4SYGYT8U |
|
@nzola This is a playlist of playlists. I am trying to download it right now on my side (LRN2). Although only a subset of 100 videos will be attempted, the metadata fetch is going to take very long to gather metadata for all videos in every playlists found. I will report back on errors found. This may help me understanding the issues you have encountered. |
|
ASIDE: Of course YouTube videos that are truly "live" (happening at that moment) are not possible to download — until after the live event is complete. 🎤🎙️ |
|
Downloading https://www.youtube.com/@TOPCONGOFM/playlists failed due to an unavailable video ( |
|
NON-URGENT: "Tasks" view needs to use more clear language than bureaucratic language like "unavailable video" eventually. So regular teachers in all countries know what this really means 😇 Ideally with actionable suggestions (in those cases where that's realistic!) |
|
I tried again to download these 7 videos now , but still failed. I did not do any upgrade or new install.
|
Curiously the screenshot shows no error message / explanation at all, during this 2nd attempt. (Definitely room for improvement, thanks @nzola.) |
|
The error message will be displayed now, but the videos will not download. Live videos (finished or not) are not downloadable with xklb by design. |
By whose design? FYI this design assumption seems extremely weak, given how videos labeled as "live" are actually used:
|
@deldesir clarify which PR and code improved error reporting here? Thanks, please if possible! |
|
@deldesir please try to find a legit way to tell if a YouTube video is actually live or not. (Instead of the bogus information that we're currently using — that's as good as useless — given the fact that so many podcasters permanently leave all their episodes marked as "live" ...rather intentionally... as permanently labeling videos as "live" serves as de facto marketing it would appear!) |
If something like this can be upstreamed to become a part of xklb, even better! ✊ |
|
@deldesir please test & use these URLs to make sure forward progress is steady in coming days — thank you to everyone working on this very common and very serious problem:
[ ADDITIONAL TEST CASES BELOW E.G. FOR "NOT YET LIVE" YOUTUBE URL'S! ] |
Ok. I understands. Thank you. |
|
Until we solve this serious problem properly... @deldesir believes that an initial hack/workaround should confirm the path forward, using yt-dlp options like...
|
|
iiab-diagnostics: https://dpaste.com/7XBX3XZE2 I can replicate the error, and see, "failed to download [download] ... does not pass filter (live_status=?not_live); skipping" when downloading:
It's not clear to me what the workaround is or what I need to do to test? Is the ask to use the command line and testing the download with yt-dlp?
|
|
@deldesir should outline a suggested path forward within 24h:
|
Here's the output from my terminal as confirmation. I don't know how to view the webm files, but presuming they succeeded based on the output.
|
|
@deldesir let me reframe / re-ask my question from chapmanjacobd/library#41 (comment) more directly:
|
|
CAVEAT: I concede that some people who click on "Download to IIAB" don't mind extremely slow downloads of "actually live" videos.
|
Yes that issue of overweight videos is very important. And very well known since 2023[*]. But please let's declare that separate question (of overly long duration, bandwidth-heavy, disk-heavy, and RAM/memory-heavy videos) to be off-topic for now — at least in this particular context here: 😉 [*] ASIDE: Hopefully to be solved in a few short months, or possibly much earlier! |
|
ALSO: Pre-announced / Pre-scheduled / Upcoming videos ("not yet live" !) are a whole other category of YouTube URL's... ...that I didn't realize also matter! (Presumably these too are commonly categorized as "live" videos, even though no video whatsoever exists yet!?) |
<!--- Provide a general summary of your changes in the Title above --> ## Description <!--- Describe your changes in detail --> This pull request introduces a new --live flag to enable the download of live videos. This flag adjusts the match filters to include live streams, which were previously excluded by default. ## Motivation and Context <!--- Why is this change required? What problem does it solve? --> The primary motivation for this change is to allow users to download live videos directly, a feature that was not supported due to default match filters excluding live content. <!--- If it fixes an open issue, please link to the issue here. --> Some context: iiab/calibre-web#188 ## How has this been tested? <!--- Please describe in detail how you tested your changes. --> The changes were tested by: - Implementing the --live flag and ensuring it correctly modifies the match filters. - Running the updated script to download live videos from various sources to confirm the flag's effectiveness. - Verifying that non-live videos are still downloadable without the --live flag. - Ensuring no regression in existing functionalities. <!--- Include details of your testing environment, tests ran to see how --> Tested on Ubuntu 24.04 <!--- your change affects other areas of the code, etc. --> ## Screenshots (if appropriate):  ## Types of changes <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: --> - [x] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) ## Checklist: <!--- Go over all the following points, and put an `x` in all the boxes that apply. --> - [x] My code follows the code style of this project. - [ ] My change requires a change to the documentation. - [ ] I have updated the documentation accordingly. - [x] I have added tests to cover my changes. - [x] All new and existing tests passed.
@nzola the above is not a playlist. It is a list of playlists. Can you try again with one of its individual playlists? For example, maybe start with their 8-video playlist https://youtube.com/playlist?list=PLi6Z1Wvj99SKPHMiqLkREMQrMyZy1YTPE ? @deldesir can you please clean up the error message so that @nzola and others have a crystal clear error message — instead of, or alongside Conversely: If a list of playlists can easily in future be downloaded into a single bookshelf, much like we do with a YouTube channel (a YouTube channel is basically also a list of playlists!) then we might consider that functionality, if it's genuinely needed? |
|
@nzola and AFTER you've completed testing of individual playlist(s): Try the much more ambitious experiment of downloading the entire channel — e.g. try the "Download to IIAB" button with URLs like: |
|
Downloaded this playlist:https://www.youtube.com/playlist?list=PLi6Z1Wvj99SLJRvpu2S6CPRHvECynHS0y This error happened during the download, but the process continued until all 99 videos were completed download.
|
|
@deldesir @holta pi4 Calibre-Web also downloaded this playlist: https://www.youtube.com/playlist?list=PLi6Z1Wvj99SKvAGNRiDmjhgQEZ8pU-4Y2 very smoothly without any problems |
@deldesir @holta I tried downloading the 2 above video links on both pi4 and multipass Calibre-Web, but they are still stuck on STARTED.
multipass Calibre-Web results: PUBLISHING TO URL... https://dpaste.com/GFTD2QH9D
|
|
|
@deldesir were channel downloads like this working in the past?
|
@EMG70 wonders if part of the reason is...
(@deldesir can you help clarify?!) |
Very ugly. Likely a network error. |
We did tests with channels in the past but not big as https://youtube.com/@TOPCONGOFM. This channel has 4.1k videos. Even though only 100 videos will be downloaded, the first task will take a long time processing metadata for all those 4.1k videos and sorting them out based on their views per year. |
|
The long wait is justified as I could see all playlists being indexed running |
|
@EMG70 further explains #188 (comment)
|
|
@EMG70 explained it well. Tasks are run sequentially. This is the normal behavior. I was investigating the channel having a "started" message with a progress stuck at 0%. |
|
I tried to download this video link: https://youtu.be/Zm-ROp6EMZo as requested by @holta on multipass iiab, pi4 iiab and pi400 iiab. multipass iiab: PUBLISHING TO URL... https://dpaste.com/3R8RXUUZ2 pi4: PUBLISHING TO URL... https://dpaste.com/2VXWWLBZB pi400: PUBLISHING TO URL... https://dpaste.com/8Z4TA84FH |
|
Thanks @nzola! Let's ask @deldesir to try to fix the situation in coming days if he can!
Tangentially related: |
@nzola experienced many such errors.
Videos that were once live appear to be erroneously blocked by IIAB Calibre-Web.
Example of a video that should be downloading, but fails to download:
@nzola mentioned:
VM's iiab-diagnostics: https://dpaste.com/6H8F53GPQ
@deldesir: Any idea what's happening?
The text was updated successfully, but these errors were encountered: