v1.26.0

• Extractors

  Additions

  • [behance] add date metadata field (#4417)
  • [danbooru] support booru.borvar.art (#4096)
  • [danbooru] support donmai.moe
  • [deviantart] add is_original metadata field (#4559)
  • [e621] support e6ai.net (#4320)
  • [exhentai] add fav option (#4409)
  • [gelbooru_v02] support xbooru.com (#4493)
  • [instagram] add following extractor (#1848)
  • [pillowfort] support /tagged/ URLs (#4570)
  • [pornhub] add gif support (#4463)
  • [reddit] add previews option (#4322)
  • [redgifs] add niches extractor (#4311, #4312)
  • [redgifs] support order parameter for user URLs (#4583)
  • [twitter] add user extractor and include option (#4275)
  • [twitter] add tweet-endpoint option (#4307)
  • [twitter] add date_original metadata for retweets (#4337, #4443)
  • [twitter] extract source metadata (#4459)
  • [twitter] support x.com URLs (#4452)

  Improvements

  • include Referer header in all HTTP requests (#4490, #4518)
    (can be disabled with referer option)
  • [behance] show errors for mature content (#4417)
  • [deviantart] re-add quality option and /intermediary/ transform
  • [fantia] improve metadata extraction (#4126)
  • [instagram] better error messages for invalid users (#4606)
  • [mangadex] support multiple values for lang (#4093)
  • [mastodon] support /@USER/following URLs (#4608)
  • [moebooru] match search URLs with empty tags (#4354)
  • [pillowfort] extract b2_lg_url media (#4570)
  • [reddit] improve comment metadata (#4482)
  • [reddit] ignore /message/compose URLs (#4482, #4581)
  • [redgifs] provide collection metadata as separate field (#4508)
  • [redgifs] match gfycat image URLs (#4558)
  • [twitter] improve error messages for single Tweets (#4369)

  Fixes

  • [acidimg] fix extraction
  • [architizer] fix extraction (#4537)
  • [behance] fix and update user extractor (#4417)
  • [behance] fix cookie usage (#4417)
  • [behance] handle videos without renditions (#4523)
  • [bunkr] fix media domain for cdn9 (#4386, #4412)
  • [bunkr] fix extracting .wmv files (#4419)
  • [bunkr] fix media domain for cdn-pizza.bunkr.ru (#4489)
  • [bunkr] fix extraction (#4514, #4532, #4529, #4540)
  • [deviantart] fix full resolution URLs for non-downloadable images (#293, #4548, #4563)
  • [deviantart] fix shortened URLs (#4316)
  • [deviantart] fix search (#4384)
  • [deviantart] update Eclipse API endpoints (#4553, #4615)
  • [deviantart] use private tokens for is_mature posts (#4563)
  • [flickr] update default API credentials (#4332)
  • [giantessbooru] fix extraction (#4373)
  • [hiperdex] fix crash for titles containing Unicode characters (#4325)
  • [hiperdex] fix manga metadata
  • [imagefap] fix pagination (#3013)
  • [imagevenue] fix extraction (#4473)
  • [instagram] fix private posts with long shortcodes (#4362)
  • [instagram] fix video preview archive IDs (#2135, #4455)
  • [instagram] handle exceptions due to missing media (#4555)
  • [issuu] fix extraction (#4420)
  • [jpgfish] update domain to jpg1.su (#4494)
  • [kemonoparty] update favorite API endpoint (#4522)
  • [lensdump] fix extraction (#4352)
  • [mangakakalot] update domain
  • [reddit] fix preview.redd.it URLs (#4470)
  • [patreon] fix extraction (#4547)
  • [pixiv] handle errors for private novels (#4481)
  • [pornhub] fix extraction (#4301)
  • [pururin] fix extraction (#4375)
  • [subscribestar] fix preview detection (#4468)
  • [twitter] fix crash on private user (#4349)
  • [twitter] fix TweetWithVisibilityResults (#4369)
  • [twitter] fix crash when sortIndex is undefined (#4499)
  • [zerochan] fix tags extraction (#4315, #4319)

  Removals

  • [gfycat] remove module
  • [shimmie2] remove meme.museum

• Post Processors

  Changes

  • update finalize events
    • add finalize-error and finalize-success events that trigger
      depending on whether error(s) did or did not happen
    • change finalize to always trigger regardless of error status

  Additions

  • add python post processor
  • add prepare-after event (#4083)
  • [ugoira] add "framerate": "uniform" (#4421)

  Improvements

  • [ugoira] extend ffmpeg-output (#4421)

  Fixes

  • [ugoira] restore libx264-prevent-odd (#4407)
  • [ugoira] fix high frame rates (#4421)

• Downloaders

  Fixes

  • [http] close connection when file already exists (#4403)

• Options

  Additions

  • support parent>child categories for child extractor options,
    for example an imgur album from a reddit thread with reddit>imgur
  • implement subconfigs option (#4440)
  • add "ascii+" as a special path-restrict value (#4371)

  Removals

  • remove pyopenssl option

• Tests

  Improvements

  • move extractor results into their own, separate files (#4504)
  • include fallback URLs in content tests (#3163)
  • various test method improvements

• Miscellaneous

  Fixes

  • [formatter] use value of last alternative (#4492)
  • fix imports when running __main__.py (#4581)
  • fix symlink resolution in __main__.py
  • fix default Firefox user agent string

v1.25.8

Changes

• update default User-Agent header to Firefox 115 ESR

Additions

• [gfycat] support @me user (#3770, #4271)
• [gfycat] implement login support (#3770, #4271)
• [reddit] notify users about registering an OAuth application (#4292)
• [twitter] add ratelimit option (#4251)
• [twitter] use TweetResultByRestId endpoint that allows accessing single Tweets without login (#4250)

Fixes

• [bunkr] use .la TLD for media-files12 servers (#4147, #4276)
• [erome] ignore duplicate album IDs
• [fantia] send X-Requested-With header (#4273)
• [gelbooru_v01] fix source metadata (#4302, #4303)
• [gelbooru_v01] update vidyart domain
• [jpgfish] update domain to jpeg.pet
• [mangaread] fix tags metadata extraction
• [naverwebtoon] fix comic metadata extraction
• [newgrounds] extract & pass auth token during login (#4268)
• [paheal] fix extraction (#4262, #4293)
• [paheal] unescape source
• [philomena] fix --range (#4288)
• [philomena] handle 429 Too Many Requests errors (#4288)
• [pornhub] set accessAgeDisclaimerPH cookie (#4301)
• [reddit] use 0.6s delay between API requests (#4292)
• [seiga] set skip_fetish_warning cookie (#4242)
• [slideshare] fix extraction
• [twitter] fix following extractor not getting all users (#4287)
• [twitter] use GraphQL search endpoint by default (#4264)
• [twitter] do not treat missing TimelineAddEntries instruction as fatal (#4278)
• [weibo] fix cursor-based pagination
• [wikifeet] fix tag extraction (#4289, #4291)

Removals

• [bcy] remove module
• [lineblog] remove module

v1.25.7

Additions

• [flickr] add exif option
• [flickr] add metadata option (#4227)
• [mangapark] add source option (#3969)
• [twitter] extend conversations option (#4211)

Fixes

• [furaffinity] improve description HTML (#4224)
• [gelbooru_v01] fix --range (#4167)
• [hentaifox] fix titles containing @ (#4201)
• [mangapark] update to v5 (#3969)
• [piczel] update API server address (#4244)
• [poipiku] improve error detection (#4206)
• [sankaku] improve warnings for unavailable posts
• [senmanga] ensure download URLs have a scheme (#4235)

v1.25.6

Additions

• [blogger] download files from lh*.googleusercontent.com (#4070)
• [fantia] extract plan metadata (#2477)
• [fantia] emit warning for non-visible content sections (#4128)
• [furaffinity] extract favorite_id metadata (#4133)
• [jschan] add generic extractors for jschan image boards (#3447)
• [kemonoparty] support .su TLDs (#4139)
• [pixiv:novel] add novel-bookmark extractor (#4111)
• [pixiv:novel] add full-series option (#4111)
• [postimage] add gallery support, update image extractor (#3115, #4134)
• [redgifs] support galleries (#4021)
• [twitter] extract conversation_id metadata (#3839)
• [vipergirls] add login support (#4166)
• [vipergirls] use API endpoints (#4166)
• [formatter] implement H conversion (#4164)

Fixes

• [acidimg] fix extraction (#4136)
• [bunkr] update domain to bunkrr.su (#4159, #4189)
• [bunkr] fix video downloads
• [fanbox] prevent exception due to missing embeds (#4088)
• [instagram] fix retrieving /tagged posts (#4122)
• [jpgfish] update domain to jpg.pet (#4138)
• [pixiv:novel] fix error with embeds extraction (#4175)
• [pornhub] improve redirect handling (#4188)
• [reddit] fix crash due to empty crosspost_parent_lists (#4120, #4172)
• [redgifs] update search URL pattern (#4115, #4185)
• [senmanga] fix and update (#4160)
• [twitter] use GraphQL API search endpoint (#3942)
• [wallhaven] improve HTTP error handling (#4192)
• [weibo] prevent fatal exception due to missing video data (#4150)
• [weibo] fix .json extension for some videos

v1.25.5

Additions

• [8muses] add parts metadata field (#3329)
• [danbooru] add date metadata field (#4047)
• [e621] add date metadata field (#4047)
• [gofile] add basic password support (#4056)
• [imagechest] implement API support (#4065)
• [instagram] add order-files option (#3993, #4017)
• [instagram] add order-posts option (#3993, #4017)
• [instagram] add metadata option (#3107)
• [jpgfish] add jpg.fishing extractors (#2657, #2719)
• [lensdump] add lensdump.com extractors (#2078, #4104)
• [mangaread] add mangaread.org extractors (#2425, #2781)
• [misskey] add favorite extractor (#3950)
• [pixiv] add novel support (#1241, #4044)
• [reddit] support cross-posted media (#887, #3586, #3976)
• [postprocessor:exec] support tilde expansion for command
• [formatter] support slicing strings as bytes (#4087)

Fixes

• [8muses] fix value of album[url] (#3329)
• [danbooru] refactor pagination logic (#4002)
• [fanbox] skip invalid posts (#4088)
• [gofile] automatically fetch website-token
• [kemonoparty] fix kemono and coomer logins sharing the same cache (#4098)
• [newgrounds] add default delay between requests (#4046)
• [nsfwalbum] detect placeholder images
• [poipiku] extract full descriptions (#4066)
• [tcbscans] update domain to tcbscans.com (#4080)
• [twitter] extract TwitPic URLs in text (#3792, #3796)
• [weibo] require numeric IDs to have length >= 10 (#4059)
• [ytdl] fix crash due to removed no_color attribute
• [cookies] improve logging behavior (#4050)

v1.25.4

Additions

• [4chanarchives] add thread and board extractors (#4012)
• [foolfuuka] add archive.palanq.win
• [imgur] add favorite-folder extractor (#4016)
• [mangadex] add status and tags metadata (#4031)
• allow selecting a domain with --cookies-from-browser
• add --cookies-export command-line option
• add -C as short option for --cookies
• include exception type in config error messages

Fixes

• [exhentai] update sadpanda check
• [imagechest] load all images when a "Load More" button is present (#4028)
• [imgur] fix bug causing some images/albums from user profiles and favorites to be ignored
• [pinterest] update endpoint for related board pins
• [pinterest] fix pin.it extractor
• [ytdl] fix yt-dlp --xff/--geo-bypass tests (#3989)

Removals

• [420chan] remove module
• [foolfuuka] remove archive.alice.al and tokyochronos.net
• [foolslide] remove sensescans.com
• [nana] remove module

v1.25.3

Additions

• [imagefap] extract description and categories metadata (#3905)
• [imxto] add gallery extractor (#1289)
• [itchio] add game extractor (#3923)
• [nitter] extract user IDs from encoded banner URLs
• [pixiv] allow sorting search results by popularity (#3970)
• [reddit] match preview.redd.it URLs (#3935)
• [sankaku] support post URLs with MD5 hashes (#3952)
• [shimmie2] add generic extractors for Shimmie2 sites (#3734, #943)
• [tumblr] add day extractor (#3951)
• [twitter] support profile-conversation entries (#3938)
• [vipergirls] add thread and post extractors (#3812, #2720, #731)
• [downloader:http] add consume-content option (#3748)

Fixes

• [2chen] update domain to sturdychan.help
• [behance] fix extraction (#3980)
• [deviantart] retry downloads with private token (#3941)
• [imagefap] fix empty tags metadata
• [manganelo] support arbitrary minor version separators (#3972)
• [nozomi] fix file URLs (#3925)
• [oauth] catch exceptions from webbrowser.get() (#3947)
• [pixiv] fix pixivision extraction
• [reddit] ignore id-max value "zik0zj"/2147483647 (#3939, #3862, #3697, #3606, #3546, #3521, #3412)
• [sankaku] sanitize date: tags (#1790)
• [tumblr] fix and update pagination logic (#2191)
• [twitter] fix user metadata when downloading quoted Tweets (#3922)
• [ytdl] fix crash due to --geo-bypass deprecation (#3975)
• [postprocessor:metadata] support putting keys in quotes
• include more optional dependencies in executables (#3907)

v1.25.2

Additions

• [deviantart] add public option
• [nitter] extract videos from source elements (#3912)
• [twitter] add date_liked and date_bookmarked metadata for liked and bookmarked Tweets (#3816)
• [urlshortener] add support for bit.ly & t.co (#3841)
• [downloader:http] add MIME type and signature for .heic files (#3915)

Fixes

• [blogger] update regex to get the highest resolution URLs (#3863, #3870)
• [bunkr] update domain to bunkr.la (#3813, #3877)
• [deviantart] keep using private access tokens when requesting download URLs (#3845, #3857, #3896)
• [hentaifoundry] fix content filters (#3887)
• [hotleak] fix downloading of creators whose name starts with a category name (#3871)
• [imagechest] fix extraction (#3914)
• [realbooru] fix extraction (#2530)
• [sexcom] fix pagination (#3906)
• [sexcom] fix HD video extraction
• [shopify] fix collection extractor (#3866, #3868)
• [twitter] update to bookmark timeline v2 (#3859, #3854)
• [twitter] warn about "withheld" Tweets and users (#3864)

Improvements

• [danbooru] reduce number of API requests when fetching extended metadata
• [deviantart:search] detect login redirects (#3860)
• [generic] write regular expressions without x flags
• [mastodon] try to get account IDs without access token
• [twitter] calculate date from Tweet IDs

v1.25.1

Additions

• [nitter] support nitter.it (#3819)
• [twitter] add hashtag extractor (#3783)
• [twitter] support Tweet content with >280 characters
• [formatter] support loading f-strings from template files (#3800)
• [formatter] support filesystem paths for \fM modules (#3399)
• [formatter] support putting keys in quotes (e.g. user['name']) (#2559)
• [postprocessor:metadata] add skip option (#3786)

Fixes

• [output] set errors=replace for output streams (#3765)
• [gelbooru] extract favorites without needing cookies (#3704)
• [gelbooru] fix and improve --range for pools
• [hiperdex] fix extraction (#3768)
• [naverwebtoon] fix extraction (#3729)
• [nitter] fix extraction for instances without user banners
• [twitter] update API query hashes and parameters
• [weibo] support mix_media_info entries (#3793)
• fix circular reference detection for -K

Changes

• update globals instead of overwriting the default (#3773)

v1.25.0

Changes

• [e621] split e621 extractors from danbooru module (#3425)
• [deviantart] remove mature scraps warning (#3691)
• [deviantart] use /collections/all endpoint for favorites (#3666 ,#3668)
• [newgrounds] update default image and audio archive IDs to prevent ID overlap (#3681)
• rename --ignore-config to --config-ignore

Extractors

• [catbox] add file extractor (#3570)
• [deviantart] add search extractor (#538, #1264, #2954, #2970, #3577)
• [deviantart] add gallery-search extractor (#1695)
• [deviantart] support fxdeviantart.com URLs (##3740)
• [e621] implement notes and pools metadata extraction (#3425)
• [gelbooru] add favorite extractor (#3704)
• [imagetwist] support phun.imagetwist.com and imagehaha.com domains (#3622)
• [instagram] add user metadata field (#3107)
• [manganelo] update and fix metadata extraction
• [manganelo] support mobile-only chapters
• [mangasee] extract author and genre metadata (#3703)
• [misskey] add misskey extractors (#3717)
• [pornpics] add gallery and search extractors (#263, #3544, #3654)
• [redgifs] support v3 URLs (#3588. #3589)
• [redgifs] add collection extractors (#3427, #3662)
• [shopify] support ohpolly.com (#440, #3596)
• [szurubooru] add tag and post extractors (#3583, #3713)
• [twitter] add transform option

Options

• [postprocessor:metadata] add sort and separators options
• [postprocessor:exec] implement archive options (#3584)
• add --config-create command-line option (#2333)
• add --config-toml command-line option to load config files in TOML format
• add output.stdout, output.stdin, and output.stderr options (#1621, #2152, #2529)
• add hash_md5 and hash_sha1 functions (#3679)
• implement globals option to enable defining custom functions for eval statements
• implement archive-pragma option to use SQLite PRAGMA statements
• implement actions to trigger events on logging messages (#3338, #3630)
• implement ability to load external extractor classes
  • -X/--extractors command-line options
  • extractor.modules-sources config option

Fixes

• [bunkr] fix extraction (#3636, #3655)
• [danbooru] send gallery-dl User-Agent (#3665)
• [deviantart] fix crash when handling deleted deviations in status updates (#3656)
• [fanbox] fix crash with missing images (#3673)
• [imagefap] update gallery URLs (#3595)
• [imagefap] fix infinite pagination loop (#3594)
• [imagefap] fix metadata extraction
• [oauth] use default name for browsers without name attribute
• [pinterest] unescape search terms (#3621)
• [pixiv] fix --write-tags for "tags": "original" (#3675)
• [poipiku] warn about incorrect passwords (#3646)
• [reddit] update videos option (#3712)
• [soundgasm] rewrite (#3578)
• [telegraph] fix extraction when images are not in <figure> elements (#3590)
• [tumblr] raise more detailed errors for dashboard-only blogs (#3628)
• [twitter] fix some original retweets not downloading (#3744)
• [ytdl] fix --parse-metadata (#3663)
• [downloader:ytdl] prevent exception on empty results

Improvements

• [downloader:http] use time.monotonic()
• [downloader:http] update _http_retry to accept a Python function (#3569)
• [postprocessor:metadata] speed up JSON encoding
• replace json.loads/dumps with direct calls to JSONDecoder.decode/JSONEncoder.encode
• improve option.Formatter performance

Removals

• [nitter] remove nitter.pussthecat.org