3 kinds of errors on a fresh test: (1) 'NoneType' object cannot be interpreted as an integer (2) 'NoneType' object is not subscriptable (3) Download appears to be stuck

[holta]

Thanks @nzola for having confirmed!

I could only copy 2 videos. Most of them failed. Single videos and playlist. I re-installed multipass and Calibre-Web on my new computer.

[holta]

Possibly related:

• When no format is available (downloadable) for a video [e.g. member-only videos, ETC] notify user & explain why ['NoneType' object is not subscriptable] #114
• 2 videos "failed to download: 'NoneType' object is not subscriptable" — yt-dlp --format best fails to download complete video (same 1503232-byte .mp4.part fragment every time!) #135
• Playlists did not fully download on newly built IIAB or upgraded server — 'NoneType' object cannot be interpreted as an integer — failed to download: Download appears to be stuck. #158

[holta]

@nzola clarififies that these were some of the videos he tried above:

• https://www.youtube.com/watch?v=zOHi37jWcro (25 Advanced English Vocabulary Phrases)
• https://www.youtube.com/watch?v=xQpImggreLE (March 19, 2024, ASCODERU)
• https://www.youtube.com/@iogacademy/playlists (IOG Academy)
• https://www.youtube.com/watch?v=5WenPx1krGE&t=90s (Essential Cardano360 May 2024 Edition)
• https://www.youtube.com/@IohkIo/playlists (Input Output)

[holta]

Thank you @EMG70 for clarifying your own experience testing Nzola's 5 links above!

https://www.youtube.com/watch?v=zOHi37jWcro ✅
https://www.youtube.com/watch?v=xQpImggreLE ✅
https://www.youtube.com/@iogacademy/playlists 297 videos in this channel,download failed ❌
https://www.youtube.com/watch?v=5WenPx1krGE&t=90s✅
https://www.youtube.com/@IohkIo/playlists very long channel of 972 videos.I had to cancel it as it took long to initialize ❌

[deldesir]

Explanations of errors:

• (1) 'NoneType' object cannot be interpreted as an integer : likely due to a missing metadata field like view_count or duration during the metadata fetching task. Restricted videos will throw this error.

• (2) 'NoneType' object is not subscriptable: this means the video file was not found. It might a premature check for the file or a failure on xklb side to process the file.

• (3) Download appears to be stuck: a timeout issue experienced during the download process. Will be strengthened to avoid false positives.

[nzola]

Last night, I successfully upgraded Calibre-Web on a PI4 and run the video playlist "https://www.youtube.com/@IohkIo/playlists", this is the "PUBLISHING TO URL...https://dpaste.com/ELHD529SS", and this is the results I got on the task view and status view:

[holta]

@nzola

You've pasted in the wrong iiab-diagnostics:

• https://dpaste.com/BGKGBZK7G shows a PC CPU, rather than a Raspberry Pi CPU.

• Please run iiab-diagnostics on the Raspberry Pi 4 and post the resulting URL (a dpaste.com pastebin) to this ticket when you can, Thanks!

[nzola]

@nzola

You've pasted in the wrong iiab-diagnostics:

• https://dpaste.com/BGKGBZK7G shows a PC CPU, rather than a Raspberry Pi CPU.
• Please run iiab-diagnostics on the Raspberry Pi 4 and post the resulting URL (a dpaste.com pastebin) to this ticket when you can, Thanks!
  PUBLISHING TO URL...
  https://dpaste.com/ELHD529SS

[holta]

• (1) 'NoneType' object cannot be interpreted as an integer : likely due to a missing metadata field like view_count or duration during the metadata fetching task. Restricted videos will throw this error.

Thanks @deldesir and @EMG70 and @nzola:

Browsing to https://youtu.be/ROgfEuRI6Sw shows:

If the owner of this video has granted you access, please sign in.

While seemingly obvious:

• Videos are hidden from and disappear from YouTube regularly / eternally for a variety of different reasons, consequently...
• IIAB Teachers expect and deserve IIAB Calibre-Web "Tasks" view to show a "Googlable" (or very friendly!!) message explaining exactly what they should try next~

🎨

[deldesir]

You're right. The "NoneType" object can stay in the log file, but the user should see a message explaining what's going on. I'll work on this.

[holta]

Also, a message like "'NoneType' object cannot be interpreted as an integer" strongly indicates that a Python variable type needs validation? 🙃

• Let's gather all such/similar reproducible cases. As much as we can in coming weeks. ✊

• And stress test these during our move to Ruff-based CI (GitHub Actions), while gradually increasing static type-checking of Python variables!

[holta]

@deldesir

@nzola says that playlist https://www.youtube.com/@iogacademy/playlists still continues to fail to download — does this mean that older playlists with videos that are "unavailable" or "owner of this video has [NOT] granted you access" need more sophisticated handling?

1. On PC/VM it failed with:

Metadata fetch: https://www.youtube.com/watch?v=V0zV_xO3Yal failed: 'NoneType' object cannot be interpreted as an integer

https://dpaste.com/CDSUG68CD

2. On RPi 4 it failed with:

Metadata fetch: https://www.youtube.com/watch?v=ROgfEuRI6Sw failed: 'NoneType' object cannot be interpreted as an integer

https://dpaste.com/6V66CQ7NQ

[deldesir]

I see, I'm addressing this with #175 and another PR which will make calculations of views per day failsafe.

[holta]

I see, I'm addressing this with #175

PR #175 is now merged:

• PR Read error from xklb-metadata.db [leveraging yt-dlp STDERR] #175

and another PR which will make calculations of views per day failsafe.

Great!

[holta]

@nzola says that playlist https://www.youtube.com/@iogacademy/playlists still continues to fail to download — does this mean that older playlists with videos that are "unavailable" or "owner of this video has [NOT] granted you access" need more sophisticated handling?

@deldesir it appears playlists/channels are blocked from downloading if even a single YouTube video is "unavailable" or "owner of this video has [NOT] granted you access".

• Can this strict approach be made more pragmatic — for educators wanting the bulk of the playlist/channel downloaded — even despite a few videos no longer being available?

• (Instead of blocking the full download!)

[holta]

@nzola says that playlist https://www.youtube.com/@iogacademy/playlists still continues to fail to download — does this mean that older playlists with videos that are "unavailable" or "owner of this video has [NOT] granted you access" need more sophisticated handling?

@deldesir it appears playlists/channels are blocked from downloading if even a single YouTube video is "unavailable" or "owner of this video has [NOT] granted you access".

• Can this strict approach be made more pragmatic — for educators wanting the bulk of the playlist/channel downloaded — even despite a few videos no longer being available?
• (Instead of blocking the full download!)

@deldesir's PR #179 proposes a solution to this problem, please see:

• Handle restricted/unavailable videos [and list them in "Tasks" view, when downloading (stale!) playlists] #179 (comment)