Formerly live https://youtu.be/4BL65HElOPg fails to download [same "no such column: error" issue as #186 ?] [unavailable fragments?]

[holta]

Thanks @nzola for reporting the inability to download this formerly live video:

PARLONS HISTOIRE : L’ARMÉE CONGOLAISE ET LES DEUX GUERRES MONDIALES
https://youtu.be/4BL65HElOPg

https://dpaste.com/4D4SEN6F6 Ubuntu 24.04 here:

https://dpaste.com/5CJPTTJRM Raspberry Pi 4 here:

ASIDE: Does the repetition in this "Tasks" view UX need to be clarified, or tightened up a bit?

Related:

• Formerly live videos fail to download: "failed to download [download] ... does not pass filter (live_status=?not_live); skipping" #188
• PR Add --live flag to support downloading of "live" videos #199
• "Upcoming live" and "Actually live" videos are caught in subsequent downloads [need to clearly explain errors, e.g. in "Tasks" view] #202
• PR Adjust message for formerly "live" videos #211

[holta]

@deldesir

Is it possible the video's long length/duration (1:56:49) might be part of the problem?

[holta]

https://dpaste.com/4D4SEN6F6 Ubuntu 24.04 here:

Lines 1110 and 1111 pasted in below — this issue appears to be the same as:

• (1) download.py: "sqlite3.OperationalError: no such column: error" after "TypeError: 'NoneType' object is not subscriptable" (2) editbooks.py: SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed #186

[2024-07-05 23:59:45,842] ERROR {cps.tasks.download:132} An error occurred during the subprocess execution: 'NoneType' object is not subscriptable
[2024-07-05 23:59:45,843] ERROR {cps.services.worker:202} no such column: error
Traceback (most recent call last):
  File "/usr/local/calibre-web-py3/cps/tasks/download.py", line 93, in run
    requested_file = conn.execute("SELECT path FROM media WHERE webpath = ? AND path NOT LIKE 'http%'", (self.media_url,)).fetchone()[0]
                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
TypeError: 'NoneType' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/calibre-web-py3/cps/services/worker.py", line 199, in start
    self.run(*args)
  File "/usr/local/calibre-web-py3/cps/tasks/download.py", line 133, in run
    self.message = f"{self.media_url_link} failed to download: {self.read_error_from_database()}"
                                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/calibre-web-py3/cps/tasks/download.py", line 149, in read_error_from_database
    error = conn.execute("SELECT error FROM media WHERE webpath = ?", (self.media_url,)).fetchone()[0]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: no such column: error

[holta]

@deldesir do you understand this bug @nzola reported about a week ago?

[deldesir]

Yes, that means the video got stuck due to unavailable fragments.

[holta]

means the video got stuck due to unavailable fragments

Dozens of "Some fragments are taking longer than expected to download" messages need to be explained better in "Tasks" view for everyday users:

1. What does this really mean?
2. Being told dozens of times to "please wait" is not believable — is this truly the best that can be done?
3. Conversely, worst case if there's absolutely nothing that can be done in this case and such cases, the root cause yt-dlp issue(s) need to be mentioned!
4. Finally, what can be done about (1) download.py: "sqlite3.OperationalError: no such column: error" after "TypeError: 'NoneType' object is not subscriptable" (2) editbooks.py: SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed #186 ?

[deldesir]

means the video got stuck due to unavailable fragments

Dozens of "Some fragments are taking longer than expected to download" messages need to be explained better in "Tasks" view for everyday users:

1. What does this really mean?

This is an unrecoverable error happened mysteriously with some videos. During the download some fragments just gone missing and the download is stuck. Nothing to see with Calibre-Web, xklb and yt-dlp. The expected fragments just don't leave YouTube servers.

2. Being told dozens of times to "please wait" is not believable — is this truly the best that can be done?

Fixed in #223

3. Conversely, worst case if there's absolutely nothing that can be done in this case and such cases, the root cause yt-dlp issue(s) need to be mentioned!

There is nothing we can do actually. This is an external issue yt-dlp cannot do anything about

4. Finally, what can be done about (1) download.py: "sqlite3.OperationalError: no such column: error" after "TypeError: 'NoneType' object is not subscriptable" (2) editbooks.py: SAWarning: Object of type not in session, add operation along 'Authors.books' won't proceed #186 ?

I have done my best to get rid of the warning editbooks.py: SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed. There is still work to do to ship a successful PR.

[holta]

There is nothing we can do actually. This is an external issue yt-dlp cannot do anything about

1. Ok. So please now provide a link in "Tasks" view to a yt-dlp (or similar!) ticket explaining the context. Thanks.

2. Explain here why the error message is being repeated, so everyone can understand the root cause, and what it means.

   Whereas suppressing error messages is almost always neither useful nor healthy.

   (Ref: Adjust error message for stuck/slow videos [if elapsed_time >= timeout, e.g. due to unavailable fragments &/or xklb media_check] #223 (comment))

3. Where is the work below? Thanks!

I have done my best to get rid of the warning editbooks.py: SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed. There is still work to do to ship a successful PR.

[deldesir]

I have new understanding of the issue. The video was not stuck at yt-dlp level. That means it actually reached 100% progress. Another video that I tested (which was previously stuck) recovered successfully. I am looking into xklb's media check for hints. I have a suspicion memory might be an issue because the video that was recovered did not use over 4G which is set as the maximum in the media check code here at L48

https://github.com/chapmanjacobd/library/blob/96273e689c5f113003e9b7c169198c72946adb44/xklb/mediafiles/media_check.py#L43-L59

[holta]

Great progress! 💯

Explaining to everyone a bit better what's happening is indeed the goal, as intuitively/clearly as we can — on PR #223 and/or similar, so these can be merged quickly — as soon as any of these appear to be reasonable and safe (no matter how small!) steps forward! 🏗️

[holta]

@nzola & @deldesir

Can you please help retest &/or recap? Now that these PR's are merged...

• PR Ensure error and live_status columns are created from the start #191
• PR Increase fragment stuck timeout to 60 seconds [was 30s] #227
• PR Increase timeout to 120s [fragment_stuck_timeout = 120 # seconds] #229

...thanks!! To help everyone deepen our understanding of next steps, e.g.:

• PR Adjust error message for stuck/slow videos [if elapsed_time >= timeout, e.g. due to unavailable fragments &/or xklb media_check] #223

[deldesir]

@nzola, this issue should have been fixed by #191.

[holta]

@nzola can you restest & confirm? (On July 24th I wrote...)

@nzola & @deldesir

Can you please help retest &/or recap? Now that these PR's are merged...

• PR Ensure error and live_status columns are created from the start #191
• PR Increase fragment stuck timeout to 60 seconds [was 30s] #227
• PR Increase timeout to 120s [fragment_stuck_timeout = 120 # seconds] #229

[holta]

@nzola & @deldesir

I just tested "Download to IIAB" with https://youtu.be/4BL65HElOPg using the latest yt-dlp and the latest xklb, and it works:

1. Great that it was solved more than 5 weeks ago!
2. How do we work more closely together, to keep tickets like this from being ignored for 5+ weeks in future?