2024.11.21.0

API.Instagram: code refactoring (settings); add setting to skip errors; add 'ForceUpdateUserName' and 'ForceUpdateUserInfo' properties; add 'IgnoreStoriesDownloadingErrors' to the settings; improve username update algorithm API.OnlyFans: add 'UpdateRules401' property; update the code to handle error 401 API.YouTube: 404 error handling (community) UserDataBase: raise event to update user in exceptions; add extra buttons for special download (limited and dated) UserDataBind: extra buttons (UserDataBase) UserCreatorForm: fix network paths GlobalSettingsForm, MainFrame, SettingsCLS: add ability to change the feed opening shortcut MainFrame: update button captions, update 'DownloadSelectedUser' function
AAndyProgram · Nov 21, 2024 · b428327 · b428327
1 parent aedcebc
commit b428327
Show file tree

Hide file tree

Showing 18 changed files with 678 additions and 489 deletions.
diff --git a/Changelog.md b/Changelog.md
@@ -1,14 +1,37 @@
+# 2024.11.21.0
+
+*2024-11-21*
+
+- Added
+  - Sites:
+    - Instagram:
+      - setting to skip errors without disabling download *(site settings)*
+      - settings to force update of username and/or user information *(user settings)*
+      - setting to continue downloading profile if error 560 occurs while downloading user stories *(site settings)*
+      - improve username update algorithm
+    - YouTube: 404 error handling (community)
+  - Main window: add extra buttons for special download (limited and dated) in collection
+  - Global settings: ability to change the feed opening shortcut (`Ctrl+F`/`Alt+F` *(Settings - Behavior)*)
+  - Minor improvements
+- Updated
+  - yt-dlp up to version **2024.11.18**
+  - gallery-dl up to version **1.27.7**
+- Fixed
+  - Users: network paths aren't working
+  - Main window: in some cases users are not updated in the list
+  - Minor bugs
+
 # 2024.10.24.0
 
 *2024-10-24*
 
 - Added
-  - YouTube (standalone app)
+  - YouTube (standalone app):
     - settings `Embed thumbnail (cover)` and `Allow webm formats`
     - changed cover selection for music downloads
-    - allow `webm` formats is there are no `mp4` formats via http protocol (issue #211)
+    - allow `webm` formats if there are no `mp4` formats via http protocol (issue #211)
   - Sites:
-    - Instagram
+    - Instagram:
       - **ability to manually change username**
       - **mark user as non-existent if user `ID` cannot be obtained**
     - Twitter: **ability to manually change username**

diff --git a/FAQ.md b/FAQ.md
@@ -33,15 +33,15 @@ I strongly recommend you to **regularly** create backup copies of the settings f
 ## General questions
 - **PROFILES**
     - I added a profile but **nothing downloaded** :arrow_forward: check your cookies and [site requirements](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements). If there are any optional fields that you don't fill in, do so. Still nothing works - [report it](#how-to-report-a-problem)!
-    - User downloading failed :arrow_forward: check your credentials and **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**. If all settings are set and nothing works, [report it](#how-to-report-a-problem). <u>Don't forget to attach the LOG.</u>
+    - User downloading failed :arrow_forward: check your credentials and **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**. If all settings are set and nothing works, [report it](#how-to-report-a-problem). **Don't forget to attach the LOG.**
     - [How to redownload user](https://github.com/AAndyProgram/SCrawler/wiki#redownload-user)
     - How to **add profile** to download :arrow_forward: copy the **[profile URL](https://github.com/AAndyProgram/SCrawler/wiki#add-user)** and press `Insert` or `Ctrl+Insert`. **ALWAYS PASTE THE USER PROFILE URL**. After that select this user and press `F5` or click the `Download selected` button.
 - How to download **[saved posts](https://github.com/AAndyProgram/SCrawler/wiki#saved-posts)**
 - **[HOW TO ADD COOKIES](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies)**
 - [How to report a problem](#how-to-report-a-problem)
 - I want you to **add the site** to SCrawler :arrow_forward: **I'm not currently accepting requests to add new sites**, but you can [create a plugin](https://github.com/AAndyProgram/SCrawler/wiki/Plugins) (for your site) for SCrawler.
-- What language is SCrawler written in :arrow_forward: vb.net
-- I don't know vb.net and I can't write a plugin :arrow_forward: you can write a plugin in `C#`
+- What language is SCrawler written in :arrow_forward: `vb.net`
+- I don't know `vb.net` and I can't write a plugin :arrow_forward: you can write a plugin in `C#`
 - I have a suggestion, will it be added :arrow_forward: maybe if it interested me.
 - How to name files using a pattern (e.g. `Site_PostID_Name.jpg`) :arrow_forward: **there is no such functionality and there are no such plans**.
 - **DON'T CHANGE THE DEFAULT SITE SETTINGS UNLESS YOU KNOW EXACTLY WHAT YOU'RE DOING!** SCrawler already has all the default settings to work. You only need to add credentials (where [required](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)).
@@ -61,7 +61,7 @@ I strongly recommend you to **regularly** create backup copies of the settings f
 
 *How to use: find the site you need in the list and read the answer.*
 
-- Reddit: don't use credentials at all or configure [OAuth](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-get-reddit-credentials). **Reddit profiles can be downloaded without any credentials at all. Subreddits require OAuth! If nothing downloads, use OAuth!**
+- Reddit: don't use credentials at all or configure [OAuth](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-get-reddit-credentials). **Reddit profiles can be downloaded without any credentials at all. Subreddits require OAuth! If nothing downloads, use OAuth!** Don't use OAuth token to download saved posts (use cookies only).
 - **META** (**Instagram**, Threads, Facebook): you need **cookies** and fill in **all fields**
 - **Instagram saved posts**: I don't consider questions like "I have 10k saved posts and only 1000 were downloaded". Download posts, remove them from saved posts, delete the `Saved posts` **settings folder**, repeat.
 - TikTok: works via yt-dlp. If something doesn't download, we need to wait until yt-dlp fixes it. TikTok doesn't require cookies to download.

diff --git a/SCrawler/API/Base/UserDataBase.vb b/SCrawler/API/Base/UserDataBase.vb
@@ -80,6 +80,8 @@ Namespace API.Base
         Private _CollectionButtonsExists As Boolean = False
         Private _CollectionButtonsColorsSet As Boolean = False
         Friend WithEvents BTT_CONTEXT_DOWN As ToolStripKeyMenuItem
+        Friend WithEvents BTT_CONTEXT_DOWN_LIMIT As ToolStripKeyMenuItem
+        Friend WithEvents BTT_CONTEXT_DOWN_DATE As ToolStripKeyMenuItem
         Friend WithEvents BTT_CONTEXT_EDIT As ToolStripMenuItem
         Friend WithEvents BTT_CONTEXT_DELETE As ToolStripMenuItem
         Friend WithEvents BTT_CONTEXT_ERASE As ToolStripMenuItem
@@ -98,6 +100,8 @@ Namespace API.Base
                 End If
             End With
             BTT_CONTEXT_DOWN = New ToolStripKeyMenuItem(tn, i) With {.Name = tnn("DOWN"), .Tag = Me}
+            BTT_CONTEXT_DOWN_LIMIT = New ToolStripKeyMenuItem(tn, i) With {.Name = tnn("DOWN_LIMIT"), .Tag = Me}
+            BTT_CONTEXT_DOWN_DATE = New ToolStripKeyMenuItem(tn, i) With {.Name = tnn("DOWN_DATE"), .Tag = Me}
             BTT_CONTEXT_EDIT = New ToolStripMenuItem(tn, i) With {.Name = tnn("EDIT"), .Tag = Me}
             BTT_CONTEXT_DELETE = New ToolStripMenuItem(tn, i) With {.Name = tnn("DELETE"), .Tag = Me}
             BTT_CONTEXT_ERASE = New ToolStripMenuItem(tn, i) With {.Name = tnn("ERASE"), .Tag = Me}
@@ -117,7 +121,8 @@ Namespace API.Base
                 cb = MyColor.EditBack
                 cf = MyColor.EditFore
             End If
-            For Each b As ToolStripMenuItem In {BTT_CONTEXT_DOWN, BTT_CONTEXT_EDIT, BTT_CONTEXT_DELETE, BTT_CONTEXT_ERASE,
+            For Each b As ToolStripMenuItem In {BTT_CONTEXT_DOWN, BTT_CONTEXT_DOWN_LIMIT, BTT_CONTEXT_DOWN_DATE,
+                                                BTT_CONTEXT_EDIT, BTT_CONTEXT_DELETE, BTT_CONTEXT_ERASE,
                                                 BTT_CONTEXT_OPEN_PATH, BTT_CONTEXT_OPEN_SITE}
                 If Not b Is Nothing Then b.BackColor = cb : b.ForeColor = cf
             Next
@@ -1282,12 +1287,14 @@ BlockNullPicture:
                         ErrorsDescriber.Execute(EDP.SendToLog, exit_ex, $"{ToStringForLog()}: downloading interrupted (exit)")
                     End If
                 End If
+                If _EnvirInvokeUserUpdated Then OnUserUpdated()
                 Canceled = True
             Catch dex As ObjectDisposedException When Disposed
                 Canceled = True
             Catch ex As Exception
                 UpdateUserInformation_Ex()
                 LogError(ex, "downloading data error")
+                If _EnvirInvokeUserUpdated Then OnUserUpdated()
                 HasError = True
             Finally
                 If Not UserExists Then AddNonExistingUserToLog($"User '{ToStringForLog()}' not found on the site")
@@ -2191,6 +2198,12 @@ BlockNullPicture:
         Private Sub BTT_CONTEXT_DOWN_KeyClick(sender As Object, e As MyKeyEventArgs) Handles BTT_CONTEXT_DOWN.KeyClick
             Downloader.Add(Me, e.IncludeInTheFeed)
         End Sub
+        Private Sub BTT_CONTEXT_DOWN_LIMIT_KeyClick(sender As Object, e As MyKeyEventArgs) Handles BTT_CONTEXT_DOWN_LIMIT.KeyClick
+            ControlInvokeFast(MainFrameObj.MF, Sub() MainFrameObj.MF.DownloadSelectedUser(MainFrame.DownUserLimits.Number, e.IncludeInTheFeed, Me), EDP.SendToLog)
+        End Sub
+        Private Sub BTT_CONTEXT_DOWN_DATE_KeyClick(sender As Object, e As MyKeyEventArgs) Handles BTT_CONTEXT_DOWN_DATE.KeyClick
+            ControlInvokeFast(MainFrameObj.MF, Sub() MainFrameObj.MF.DownloadSelectedUser(MainFrame.DownUserLimits.Date, e.IncludeInTheFeed, Me), EDP.SendToLog)
+        End Sub
         Private Sub BTT_CONTEXT_EDIT_Click(sender As Object, e As EventArgs) Handles BTT_CONTEXT_EDIT.Click
             Using f As New Editors.UserCreatorForm(Me)
                 f.ShowDialog()
@@ -2271,6 +2284,8 @@ BlockNullPicture:
                     If Not ProgressPre Is Nothing Then ProgressPre.Reset() : ProgressPre.Dispose()
                     If Not Responser Is Nothing Then Responser.Dispose()
                     If Not BTT_CONTEXT_DOWN Is Nothing Then BTT_CONTEXT_DOWN.Dispose()
+                    If Not BTT_CONTEXT_DOWN_LIMIT Is Nothing Then BTT_CONTEXT_DOWN_LIMIT.Dispose()
+                    If Not BTT_CONTEXT_DOWN_DATE Is Nothing Then BTT_CONTEXT_DOWN_DATE.Dispose()
                     If Not BTT_CONTEXT_EDIT Is Nothing Then BTT_CONTEXT_EDIT.Dispose()
                     If Not BTT_CONTEXT_DELETE Is Nothing Then BTT_CONTEXT_DELETE.Dispose()
                     If Not BTT_CONTEXT_ERASE Is Nothing Then BTT_CONTEXT_ERASE.Dispose()

diff --git a/SCrawler/API/Instagram/EditorExchangeOptions.vb b/SCrawler/API/Instagram/EditorExchangeOptions.vb
@@ -9,7 +9,7 @@
 Imports SCrawler.Plugin.Attributes
 Imports DN = SCrawler.API.Base.DeclaredNames
 Namespace API.Instagram
-    Friend Class EditorExchangeOptions
+    Friend NotInheritable Class EditorExchangeOptions
 #Region "Download"
         <PSetting(Caption:="Get timeline", ToolTip:="Download user timeline")>
         Friend Property GetTimeline As Boolean
@@ -37,7 +37,11 @@ Namespace API.Instagram
         <PSetting(Caption:="Place the extracted image into the video folder")>
         Friend Property PutImageVideoFolder As Boolean
         <PSetting(Address:=SettingAddress.User, Caption:=DN.UserNameChangeCaption, ToolTip:=DN.UserNameChangeToolTip)>
-        Friend Overridable Property UserName As String = String.Empty
+        Friend Property UserName As String = String.Empty
+        <PSetting(Address:=SettingAddress.User, Caption:="Force update UserName", ToolTip:="Try to force update UserName if it is not found on the site")>
+        Friend Property ForceUpdateUserName As Boolean = False
+        <PSetting(Address:=SettingAddress.User, Caption:="Force update user information")>
+        Friend Property ForceUpdateUserInfo As Boolean = False
         Friend Sub New(ByVal u As UserData)
             With u
                 GetTimeline = .GetTimeline
@@ -55,6 +59,8 @@ Namespace API.Instagram
                 PutImageVideoFolder = .PutImageVideoFolder
 
                 UserName = .NameTrue(True)
+                ForceUpdateUserName = .ForceUpdateUserName
+                ForceUpdateUserInfo = .ForceUpdateUserInfo
             End With
         End Sub
         Friend Sub New(ByVal s As SiteSettings)