Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6529 commits
Select commit Hold shift + click to select a range
e2134b3
replace '\f' in --print arguments with form feed character
mikf Feb 7, 2025
93adc86
improve '\f' format string handling for --print
mikf Feb 8, 2025
dae82f1
[b4k] update domain to arch.b4k.dev
NecRaul Feb 8, 2025
23c4bc8
[b4k] keep support for previous 'arch.b4k.co' domain
mikf Feb 9, 2025
3282025
merge #6956: [b4k] update domain to 'arch.b4k.dev' (#6955)
mikf Feb 9, 2025
6c2b6d5
[patreon] support '/profile/creators' URLs
mikf Feb 9, 2025
587205b
[pixiv] prevent exceptions during 'comments' extraction (#6965)
mikf Feb 10, 2025
be77465
[weebcentral] fix extracting wrong number of chapter pages (#6966)
mikf Feb 10, 2025
55034d9
[imhentai] add support (#1660 #3046 #3824 #4338 #5936)
mikf Feb 10, 2025
d1a8142
[bunkr] provide fallback URLs for 403 download links (#6732 #6972)
mikf Feb 12, 2025
f1f27eb
[vsco] support '/video/' URLs (#4295 #6973)
mikf Feb 12, 2025
6420210
[vsco] improve 'm3u8' handling
mikf Feb 12, 2025
1cf2870
[patreon] extract 'campaign' metadata (#6989)
mikf Feb 13, 2025
b8b541f
[itaku] support gallery section URLs (#6951)
mikf Feb 13, 2025
51f978e
[weibo] add 'movies' option (#6988)
mikf Feb 13, 2025
195b522
[tests] move 'e621:frontend' tests into regular results/e621.py
mikf Feb 14, 2025
cd9fa1e
[bunkr] implement fast '--range' support (#6985)
mikf Feb 14, 2025
7ae09c6
[imgur] add support for (hidden) personal posts (#6990)
mikf Feb 14, 2025
182b544
[ytdl] support specifying filesystem paths as 'module' (#6991)
mikf Feb 14, 2025
046ebb5
[imgur] replace AuthorizationError exception with logging message
mikf Feb 15, 2025
3530760
[dl:http] add 'sleep-429' option (#6996)
mikf Feb 15, 2025
800cf5b
replace 'print()' with 'output.stderr_write("\n")'
mikf Feb 15, 2025
b4eae65
[imhentai] avoid unnecessary HTTP request
mikf Feb 16, 2025
841bc9f
[archive] implement support for PostgreSQL databases (#6152)
mikf Feb 16, 2025
dac0c4a
[docs] add 'psycopg' to optional dependencies
mikf Feb 16, 2025
8daf496
[archive] add 'archive-table' option (#6152)
mikf Feb 17, 2025
fd4de02
[archive] support PostgreSQL archives for post processors (#6152)
mikf Feb 17, 2025
95c446f
[discord] add support (#6836)
zWolfrost Feb 18, 2025
82493a6
[hentairox] add support (#7003)
mikf Feb 18, 2025
67937d3
[archive] fix NameError when SQLite database path doesn't exist
mikf Feb 18, 2025
4396029
[furry34] add support (#1078 #7018)
mikf Feb 19, 2025
d4c56b0
[hentaiera] add support (#3046 #6952 #7020)
mikf Feb 19, 2025
7a11d02
[reddit] restrict subreddit search results (#7025)
mikf Feb 19, 2025
52d4e1a
[imhentai] inherit from BaseExtractor
mikf Feb 19, 2025
196aa26
[imhentai] improve pagination duplicate filtering
mikf Feb 20, 2025
57937c3
[newgrounds] provide 'comment_html' metadata (#7038)
mikf Feb 22, 2025
79dc04d
[subscribestar] fix 'post' extractor (#6582)
mikf Feb 22, 2025
4906541
[generic] fix config lookups by subcategory
mikf Feb 21, 2025
18ed39c
implement 'downloader' options per extractor category
mikf Feb 21, 2025
613f05a
fix cmdline arguments not overriding extractor-downloader options
mikf Feb 22, 2025
b1487df
[scripts/pull-request] handle branch already existing
mikf Feb 23, 2025
ff5f6fe
[boosty] added new direct message extractor
Jan 4, 2025
fe958ed
merge #6768: [boosty] add 'direct-messages' extractor
mikf Feb 23, 2025
e4cc341
add bilibili User Articles FavList support
mmmpipi Jan 6, 2025
a9853cd
merge #6781: [bilibili] add 'user-articles-favorite' extractor (#6725)
mikf Feb 23, 2025
daac2c6
[tiktok] add support (#3061 #4177 #5646 #6878 #6708)
CasualYT31 Feb 25, 2025
87e3cb1
[oauth] improve wording when opening URL
mikf Feb 25, 2025
876169d
[furaffinity] use a 1s delay between requests by default (#7054)
mikf Feb 25, 2025
d2cad59
[twitter] support 'grok' cards content (#7040)
mikf Feb 25, 2025
203c2e3
[vipergirls] change default 'domain' to 'viper.click' (4166)
mikf Feb 26, 2025
13c3fa4
[docs] add 'tiktok' options (#7060)
mikf Feb 26, 2025
5e87aee
[tiktok] add 'audio' option (#7060)
mikf Feb 26, 2025
afde4ad
[tiktok] add 'avatar' option
mikf Feb 26, 2025
4a3db15
[discord] do not match message URLs as channels (#7059)
mikf Feb 26, 2025
1d3b9a9
[tiktok] fix variable name
mikf Feb 26, 2025
95c1fea
[discord] add single message support
zWolfrost Feb 26, 2025
07a9dca
[erome] extract 'tags' metadata (#7076)
mikf Feb 28, 2025
8256a7a
[sankaku] fix extraction (#7071 #7072)
mikf Feb 28, 2025
aeb72de
[chevereto] fix extraction (#7073 #7079)
mikf Feb 28, 2025
fa7114e
[docs] update supportedsites
mikf Feb 28, 2025
f86ff15
merge #7067: [discord] add single message support (#7059)
mikf Feb 28, 2025
16c3352
[bunkr] bypassed URL encryption
okbuddyagi Feb 27, 2025
7c33b9f
merge #7070: [bunkr] decrypt file URLs (#7058 #7085)
mikf Feb 28, 2025
6c8c264
[dl:ytdl] log ytdl module and version when importing
mikf Feb 28, 2025
1a6fa70
[archive] fix exception when specifying path as list (#7088)
mikf Feb 28, 2025
71ebbfa
release version 1.29.0
mikf Mar 1, 2025
472ea73
[erome] fix AttributeError for albums without tags (#7076)
mikf Mar 1, 2025
db19990
[text] allow calling 'extract_iter' with invalid arguments
mikf Mar 2, 2025
4ecc40c
[docs] fix 'tiktok-range' default value (#7098)
mikf Mar 2, 2025
262e862
[tiktok] improve 'tiktok-range' parsing (#7098)
mikf Mar 2, 2025
f155389
[jpgfish] decrypt file URLs (#7073 #7079)
mikf Mar 2, 2025
c567bf1
[danbooru][e621] remove constructors
mikf Mar 2, 2025
f232a07
[danbooru:pool] download posts in pool order (#7091)
mikf Mar 3, 2025
4c601f9
[bunkr] update endpoint (#7097)
mikf Mar 3, 2025
4d2037f
[dl] warn about invalid 'subcategory' values (#7103)
mikf Mar 3, 2025
2f3265a
[tenor] add initial support (#6075)
mikf Mar 3, 2025
3a5adbf
[vsco] fix 'video' extractor (#7113)
mikf Mar 4, 2025
198593b
[vsco] fix extracting videos from '/gallery' results (#7113)
mikf Mar 4, 2025
b5c1bf3
[dl] improve invalid 'subcategory' value warning (#7103)
mikf Mar 4, 2025
5486a9c
[util] implement 'decrypt_xor()'
mikf Mar 4, 2025
b7bb40c
[furaffinity] fix 'artist' metadata (#7115 #6582)
mikf Mar 4, 2025
f507360
[tenor] add 'user' extractor (#6075)
mikf Mar 4, 2025
ecbad03
[workflows:tests] run on ubuntu-22.04
mikf Mar 5, 2025
984116a
[furaffinity] improve 'artist_url' extraction (#7115 #7123)
mikf Mar 6, 2025
c9488ce
[util] move Cloudflare/DDoS-Guard detection into 'detect_challenge()'
mikf Mar 6, 2025
4a2c860
[dl:http] detect Cloudflare/DDoS-Guard challenges (#7066 #7121)
mikf Mar 6, 2025
fac4ef5
[erome:search] recognize all input URL query parameters (#7125)
mikf Mar 7, 2025
639ddc9
[tenor] support URLs with language codes (#6075)
mikf Mar 7, 2025
8582af3
[tenor] support '/official/' user URLs (#6075)
mikf Mar 7, 2025
18b9ffe
[redgifs:search] support '/search?query=...' URLs (#7118)
mikf Mar 7, 2025
3ef23cc
[sankaku] fix search tag limit check
mikf Mar 7, 2025
7afd5ba
[sankaku] increase wait time on 429 errors (#7129)
mikf Mar 7, 2025
3b81d89
[bunkr] add 'endpoint' option (#7097)
mikf Mar 8, 2025
486e307
[reddit] add 'selftext' option (#7111)
mikf Mar 8, 2025
52aa5ba
[tenor] rename 'content_description' to just 'description'
mikf Mar 8, 2025
d6281b5
[tenor] relax '/view/' URL pattern (#6075)
mikf Mar 8, 2025
3eba856
release version 1.29.1
mikf Mar 8, 2025
a859abf
[twitter] prevent exception in '_extract_components()' (#7139)
mikf Mar 9, 2025
04464b6
[text] add second argument to 'parse_query_list()' (#7138)
mikf Mar 10, 2025
ce01835
[facebook] improve 'date' extraction (#7151)
mikf Mar 10, 2025
518865c
[civitai] fix/improve query parameter handling (#7138)
mikf Mar 10, 2025
1254c4e
[sankaku] update API URLs (#7154 #7155)
mikf Mar 11, 2025
94bbbbb
[sankaku] fix categorized tags for posts with >100 tags (#7155)
mikf Mar 11, 2025
859f1e7
[furaffinity] Adding a FuraffinityFolderExtractor, which extracts a s…
Deer-Spangle Mar 11, 2025
a12ff28
merge #7159: [furaffinity] add 'folder' extractor (#1817)
mikf Mar 12, 2025
e1bdcd9
[furaffinity] extract 'scraps' metadata (#7015)
mikf Mar 12, 2025
d40f8a8
[tests] add support for skipping an extractor result test
mikf Mar 12, 2025
898a09b
[sankaku] fix 'tags' metadata (#7155)
mikf Mar 12, 2025
3e64ec4
[tiktok] implement audio extraction without ytdl
mikf Mar 12, 2025
cd1ddb0
[wikimedia] add 'subcategories' option (#2340)
mikf Mar 12, 2025
f395a3e
[sankaku] fix potential infinite loop (#7155)
mikf Mar 14, 2025
22d46f2
[batoto] add 'domain' option (#7174)
mikf Mar 14, 2025
d900e86
[arcalive] add support (#5657 #7100)
hdk5 Mar 14, 2025
31e57ba
[arcalive] add 'user' extractor (#5657)
mikf Mar 14, 2025
5fa5a45
[tests] improve error message of multi type/value tests
mikf Mar 14, 2025
dbe8820
[arcalife] add 'gifs' option (#5657)
mikf Mar 14, 2025
063bf06
release version 1.29.2
mikf Mar 15, 2025
2001cf1
[tiktok] Add retry mechanism to rehydration data extraction
CasualYT31 Mar 16, 2025
bf927cb
[config] fix using same key multiple times with 'apply' (#7127)
mikf Mar 16, 2025
8bdd543
[deviantart:stash] fix legacy sta.sh links (#7181)
mikf Mar 16, 2025
6532cf9
[deviantart] match '/gallery/recommended-for-you' URLs (#7168)
mikf Mar 17, 2025
c7685bd
[tiktok] Incorporate --retries and --http-timeout into rehydration da…
CasualYT31 Mar 17, 2025
d746e02
[zerochan] parse JSON-LD data (#7178)
mikf Mar 17, 2025
567f5d0
[common] add 'subdomains' argument to 'cookies_check()' (#7188)
mikf Mar 18, 2025
d491590
[patreon] include subdomains in 'session_id' cookie check (#7188)
mikf Mar 18, 2025
a6b8569
[tests] implement expected failures
mikf Mar 18, 2025
070b75a
[patreon] do not match '/messages' URLs as creators (#7187)
mikf Mar 18, 2025
d6d2b1f
[tiktok] Queue links from user profiles instead of returning a list o…
CasualYT31 Mar 19, 2025
7b79140
[tiktok] Address review comments on _extract_rehydration_data()
CasualYT31 Mar 19, 2025
685660b
[tiktok] Don't skip last retry
CasualYT31 Mar 19, 2025
f8ef9a7
[kemonoparty] enable 'username'/'user_profile' metadata by default
mikf Mar 21, 2025
e454ded
Adding functionality for subfolder support from Deviant Art
BishopRed90 Mar 22, 2025
3973481
Merge pull request #2 from BishopRed90/DA_Subfolder_Support
BishopRed90 Mar 22, 2025
4807bc2
[subscribestar] extract 'title' metadata (#7219)
mikf Mar 22, 2025
af57ab3
[subscribestar] detect redirects to '/age_confirmation_warning' pages
mikf Mar 22, 2025
63a1052
[aes] simplify 'block_count' calculation
mikf Mar 22, 2025
4a74bc6
[kemonoparty] extract 'archives' metadata (#7195)
mikf Mar 22, 2025
62f9335
[kemonoparty] use '"type": "archive"' for 'archives' files (#7195)
mikf Mar 23, 2025
b20777e
[bbc] extract more metadata (#6582)
mikf Mar 23, 2025
9a64c3d
[deviantart] update code formatting
mikf Mar 23, 2025
ffdec0c
[deviantart] match subfolders by name
mikf Mar 23, 2025
b52c211
[deviantart] add 'subfolders' option
mikf Mar 23, 2025
5bceb0e
[deviantart] add 'subfolder' tests
mikf Mar 23, 2025
fd8f652
[hitomi] fix extractors (#7230)
mikf Mar 23, 2025
b3500b4
[bunkr] fix filename extraction (#7237)
mikf Mar 24, 2025
7a6899c
[imhentai] support 'hentaienvy.com' and 'hentaizap.com' (#7192 #7218)
mikf Mar 24, 2025
e1aabf0
merge #7220: [deviantart] add subfolder support (#4988 #7185)
mikf Mar 24, 2025
58e7808
[mangapark] utilizing more graphql and adjust functionality for new s…
Mar 23, 2025
0ffef58
merge #7232: [mangapark] fix extractors (#4999 #5883 #6507 #6908)
mikf Mar 24, 2025
03c87b7
[nozomi] fix extractors (#7242)
mikf Mar 25, 2025
ebf05e5
[mangapark] support mirror domains
mikf Mar 25, 2025
32a0696
[mangapark] support v3 URLs (#2072)
mikf Mar 25, 2025
a3a6131
[kemonoparty:discord] look up 'channel_name' when missing (#7245)
mikf Mar 26, 2025
93675b8
[skeb] make exceptions when extracting posts non-fatal (#7250)
mikf Mar 26, 2025
24bbcbc
[danbooru] add 'favgroup' extractor
mikf Mar 26, 2025
6894e0b
[arcalive] extend 'gifs' option
mikf Mar 26, 2025
81a1b64
[dl:ytdl] support processing inline HLS/DASH manifests (#6379 #7006)
mikf Mar 26, 2025
fcf76ba
[instagram] download videos from 'video_dash_manifest' data
mikf Mar 27, 2025
7ccf645
[mastodon] support '/statuses' URLs (#7255)
mikf Mar 27, 2025
fc868b0
[sexcom] remove constructors
mikf Mar 27, 2025
12327b0
[sexcom] fix 'gif' pin extraction (#7239)
mikf Mar 27, 2025
c6bc46f
[sexcom] support new-style '/gifs' URLs (#7239)
mikf Mar 27, 2025
d4ce8be
[sexcom] support new-style '/videos' URLs (#7239)
mikf Mar 27, 2025
6b4cd1b
[util] update auto Firefox version
mikf Mar 27, 2025
17c1bad
Adding support for the Multi Image Gallery
BishopRed90 Mar 28, 2025
3cbb88b
Adding support for the Multi Image Gallery
BishopRed90 Mar 28, 2025
a854d60
Removing a test flag (is_downloadable) to true.
BishopRed90 Mar 28, 2025
af23894
Fixing formatting of a line that is too long
BishopRed90 Mar 28, 2025
e381c48
[sexcom] extract 'date_url' metadata (#7239)
mikf Mar 28, 2025
8bfb505
merge #7191: [tiktok] add retry mechanism to rehydration data extraction
mikf Mar 28, 2025
1e498a0
[deviantart] update
mikf Mar 28, 2025
677c187
[deviantart] add multi-file test
mikf Mar 28, 2025
e0211bd
merge #7261: [deviantart] support multiple images for single posts
mikf Mar 28, 2025
c4d08b2
[pinterest] ignore 'story_pin_static_sticker_block' blocks (#7251)
mikf Mar 28, 2025
add52e6
release version 1.29.3
mikf Mar 29, 2025
4bcbc2d
[E621:pool] fix AttributeError (#7265)
mikf Mar 29, 2025
becdfbd
[readcomiconline] fix 'issue' extractor (#7269)
mikf Mar 30, 2025
4143e8e
[everia] improve/fix image extraction (#7270)
mikf Mar 30, 2025
1396ea6
[deviantart] fix "KeyError: 'has_subfolders'" (#7272)
mikf Mar 30, 2025
ee26f3c
[instagram] use Chrome User-Agent by default (#6379)
mikf Mar 30, 2025
fb6afb1
[webtoons] update & simplify code
mikf Mar 31, 2025
492ea46
[zerochan] fix "KeyError: 'author'" (#7282)
mikf Apr 1, 2025
015ba76
[webtoons] add 'artist' extractor (#7274)
mikf Apr 1, 2025
f7cd436
[chevereto] support 'imagepond.net' (#7278)
mikf Apr 1, 2025
0ea80c0
[zzup] support 'w.zzup.com' URLs
astringa1234 Apr 8, 2025
1772c23
[dl:ytdl] fix exception when ytdl reports an error (#7301)
mikf Apr 8, 2025
7f7af12
[kemonoparty] fix file paths with backslashes (#7321)
mikf Apr 8, 2025
76040f9
[issuu] fix extractors (#7317)
mikf Apr 8, 2025
f02dcb7
[rule34xyz] update to API v2 (#7289)
mikf Apr 8, 2025
8eb365d
[pp:metadata] fix handling of empty directory paths (#7296)
mikf Apr 8, 2025
a1bf71d
merge #7327: [zzup] support 'w.zzup.com' URLs
mikf Apr 8, 2025
edd90fb
[pp:metadata] add 'metadata-path' option (#6582)
mikf Apr 9, 2025
09d42b8
[webtoons] use a default delay pf 0.5-1.5s between requests (#7329)
mikf Apr 9, 2025
4916b4f
[webtoons] download JPEG files in better quality
mikf Apr 10, 2025
9ce6de6
[formatter] add 'i' and 'f' conversions (#6582)
mikf Apr 11, 2025
a08c35a
[pp:ugoira] preserve 'extension' when using "mode": "archive" (#7304)
mikf Apr 11, 2025
232d454
[tests] add reddit post with redgifs embed
mikf Apr 12, 2025
85808a9
[gelbooru] fix video URLs (#7345)
mikf Apr 12, 2025
1ee0572
[pixiv] match phixiv.net URLs
w4tchdoge Apr 12, 2025
2ef7a05
[pixiv] update 'work' extractor pattern & add tests
mikf Apr 12, 2025
8eba4e6
merge #7352: [pixiv] support 'phixiv.net' URLs
mikf Apr 12, 2025
a27bcf6
[discord] fix categories name bug
zWolfrost Apr 12, 2025
81ff021
merge #7353: [discord] fix "parent" keyword inconsistency (#7341)
mikf Apr 12, 2025
feacdd5
[hentai2read] fix exception for chapters without artist (#7355)
mikf Apr 12, 2025
21160a8
[tumblr] support URLs without subdomain (#7358)
mikf Apr 13, 2025
2674705
release version 1.29.4
mikf Apr 13, 2025
4c8c98a
use internal, non-caching version of re.compile for extractor patterns
mikf Apr 13, 2025
48053cf
[util] update auto Chrome version
mikf Apr 14, 2025
481ad1c
[bluesky:likes] use 'repo.listRecords' endpoint (#7194 #7287)
mikf Apr 15, 2025
397a795
[bluesky] use self.groups / remove constructors
mikf Apr 15, 2025
7d1321c
[bluesky:likes] fix infinite loop (#7194 #7287)
mikf Apr 16, 2025
dcaf729
[bluesky] add 'video' extractor (#4438)
mikf Apr 16, 2025
ecf6966
[everia] fix 'title' extraction
rrockey Apr 17, 2025
9abca62
merge #7379: [everia] fix 'title' extraction
mikf Apr 17, 2025
31fd613
[instagram] add 'followers' extractor (#7374)
mikf Apr 17, 2025
f3ababc
[pp:ugoira] use '_ugoira_frame_index' to detect '.zip' files
mikf Apr 17, 2025
fc561c2
[pp:ugoira] restore 'keep-files' functionality (#7304)
mikf Apr 17, 2025
763bbfa
[pp:ugoira] support '"keep-files": true' + custom extension (#7304)
mikf Apr 17, 2025
de72714
[fapello] stop pagination on empty results (#7385)
mikf Apr 18, 2025
d6311bb
[moebooru:pool] replace underscores in pool names (#4646)
mikf Apr 18, 2025
88f1541
[common] add 'request_location()' convenience function
mikf Apr 18, 2025
10f2e08
[fanbox] fix 'comments' extraction
mikf Apr 19, 2025
2798fb8
[twitter] update API endpoint query hashes (#7382 #7386)
mikf Apr 19, 2025
e6f4dcf
Handle instagram.com/share/ URLs
DjofPlusPlus Mar 25, 2025
4b11c72
merge #7241: [instagram] handle '/share/' URLs
mikf Apr 19, 2025
f0e7992
[twitter] added 'followers' extractor
stephanelsmith Oct 16, 2024
f0437a5
merge #6331: [twitter] add 'followers' extractor
mikf Apr 19, 2025
00f3b48
[gelbooru] don't hardcode image server domains (#7392)
mikf Apr 19, 2025
03b1d42
[deviantart] simplify 'additionalMedia' extraction
mikf Apr 20, 2025
99afe58
[deviantart] fix multi-image URLs (#6653)
mikf Apr 20, 2025
0579726
fix: Wikifeet - support redesign
parasiteoflife Apr 21, 2025
b71b5af
[wikifeet] fix 'bdate' parsing
mikf Apr 21, 2025
abf39b6
merge #7396: [wikifeet] support redesign
mikf Apr 21, 2025
4c974de
[naver] support video downloads (#7395)
Oct 18, 2023
25c2c3a
merge #4682: [naver] support videos & fix recent 'date' bug (#7395)
mikf Apr 22, 2025
ee1ba08
[deviantart:stash] extract more metadata (#7397)
mikf Apr 22, 2025
c3fb0f5
[kemonoparty] use '/posts-legacy' endpoint (#6780 #6931 #7404)
mikf Apr 22, 2025
4b855b1
[scrolller] retrieve data of single posts via GraphQL
mikf Apr 22, 2025
f342108
Adding in a first pass at a pictoa extractor
nuhonda Apr 24, 2025
6bbedda
[itaku] Add a new extractor for Itaku stars
Deer-Spangle Apr 24, 2025
db89f4a
[weasyl] Set gallery-dl user agent to match danbooru user agent, when…
Deer-Spangle Apr 24, 2025
25e8a98
[scrolller] support album posts (#7339)
mikf Apr 24, 2025
29a4444
[pictoa] update
mikf Apr 24, 2025
4de7c02
merge #7409: [pictoa] add support (#6683)
mikf Apr 24, 2025
31f85ab
merge #7412: [weasyl] use 'gallery-dl' User-Agent header
mikf Apr 24, 2025
f01b4ad
[itaku:stars] update
mikf Apr 24, 2025
754b704
merge #7411: [itaku] add 'stars' extractor
mikf Apr 24, 2025
3feb2c5
Add a "filtered" event
WyohKnott Dec 30, 2024
f5f5d9d
Add "filtered" event to archive too
WyohKnott Jan 4, 2025
071b113
Limit the "filtered" event to image-filter and chapter-filter...
WyohKnott Jan 4, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
15 changes: 15 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
root = true

[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true

[**.py]
indent_size = 4
indent_style = space
max_line_length = 79

[Makefile]
indent_style = tab
2 changes: 2 additions & 0 deletions .github/FUNDING.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
ko_fi: mikefaehrmann
custom: https://www.paypal.me/mikefaehrmann
69 changes: 69 additions & 0 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
name: Docker Images

on:
workflow_dispatch:
push:
branches:
- master
tags:
- v[0-9]+.[0-9]+.[0-9]+


permissions:
packages: write

concurrency:
group: docker
cancel-in-progress: false

jobs:
build:

runs-on: ubuntu-latest

# on release commits, run only for tag event
if: ${{
github.repository == 'mikf/gallery-dl' &&
( ! startsWith( github.event.head_commit.message , 'release version ' ) ||
startsWith( github.ref , 'refs/tags/v' ) )
}}

steps:
- uses: actions/checkout@v4

- uses: docker/metadata-action@v5
id: metadata
with:
images: |
mikf123/gallery-dl
ghcr.io/mikf/gallery-dl
tags: |
type=ref,event=tag
type=raw,value=dev
type=sha,format=long,prefix=
type=raw,priority=500,value={{date 'YYYY.MM.DD'}}

- uses: docker/setup-qemu-action@v3

- uses: docker/setup-buildx-action@v3

- name: Login to DockerHub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GHCR_TOKEN }}

- uses: docker/build-push-action@v5
with:
context: .
push: true
tags: ${{ steps.metadata.outputs.tags }}
labels: ${{ steps.metadata.outputs.labels }}
platforms: linux/amd64,linux/arm64
107 changes: 107 additions & 0 deletions .github/workflows/executables.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
name: Executables

on:
workflow_dispatch:
push:
branches:
- master
tags-ignore:
- "*"

env:
DATE_FORMAT: "%Y.%m.%d"

jobs:
build:

if: github.repository == 'mikf/gallery-dl'
runs-on: ${{ matrix.os }}
defaults:
run:
shell: bash
strategy:
fail-fast: false
matrix:
os: ["windows-latest", "macOS-latest"]
architecture: ["x64"]
python-version: ["3.13"]
python-packages: [""]
include:
- os: "ubuntu-latest"
architecture: "x64"
python-version: "3.13"
python-packages: "secretstorage"
- os: "windows-2019"
architecture: "x86"
python-version: "3.8"
python-packages: "toml"

steps:
- uses: actions/checkout@v4

- name: Set up Python ${{ matrix.python-version }} ${{ matrix.architecture }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
architecture: ${{ matrix.architecture }}

- name: Environment Variables
run: |
echo "DATE=$(date '+${{ env.DATE_FORMAT }}')" >> "$GITHUB_ENV"
echo "LABEL=$(python ./scripts/pyinstaller.py --print --os '${{ matrix.os }}' --arch '${{ matrix.architecture }}')" >> "$GITHUB_ENV"

- name: Update Version
# use Python since its behavior is consistent across operating systems
shell: python
run: |
import re
path = "./gallery_dl/version.py"
with open(path) as fp:
content = fp.read()
content = re.sub(
r'\b(__version__ = "[^"]+)',
r"\1:${{ env.DATE }}",
content)
content = re.sub(
r'\b(__variant__ =).+',
r'\1 "dev/${{ env.LABEL }}"',
content)
with open(path, "w") as fp:
fp.write(content)

- name: Build executable
run: |
pip install requests requests[socks] yt-dlp[default] pyyaml ${{ matrix.python-packages }} pyinstaller
python ./scripts/pyinstaller.py --label '${{ env.LABEL }}'

- uses: actions/upload-artifact@v4
with:
name: executable-${{ matrix.os }}-${{ matrix.architecture }}-${{ matrix.python-version }}
path: dist/*
retention-days: 1
compression-level: 0

release:

needs: build
runs-on: ubuntu-latest

steps:
- uses: actions/download-artifact@v4

- name: Environment Variables
run: echo "DATE=$(date '+${{ env.DATE_FORMAT }}')" >> "$GITHUB_ENV"

- name: Body
run: printf 'https://github.com/%s/commit/%s' '${{ github.repository }}' '${{ github.sha }}' > body.md

- uses: ncipollo/release-action@v1
with:
owner: gdl-org
repo: builds
tag: ${{ env.DATE }}
bodyFile: body.md
artifacts: "executable-*/*"
allowUpdates: true
makeLatest: true
token: ${{ secrets.REPO_TOKEN }}
58 changes: 58 additions & 0 deletions .github/workflows/pages.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
name: GitHub Pages

on:
workflow_dispatch:
push:
branches:
- master
paths:
- docs/**

permissions:
contents: read
pages: write
id-token: write

concurrency:
group: pages
cancel-in-progress: false

jobs:
dispatch:

if: github.repository == 'mikf/gallery-dl'
runs-on: ubuntu-latest

steps:
- name: Dispatch to gdl-org/docs
run: >
curl -L
-X POST
-H "Accept: application/vnd.github+json"
-H "Authorization: Bearer ${{ secrets.REPO_TOKEN }}"
-H "X-GitHub-Api-Version: 2022-11-28"
https://api.github.com/repos/gdl-org/docs/actions/workflows/pages.yml/dispatches
-d '{"ref":"master"}'

deploy:

if: github.repository == 'mikf/gallery-dl'
runs-on: ubuntu-latest

environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}

steps:
- uses: actions/checkout@v4
- uses: actions/configure-pages@v4

- name: Copy static files
run: |
mkdir --parents -- ./_site
cp --archive --target-directory=./_site -- \
./docs/oauth-redirect.html

- uses: actions/upload-pages-artifact@v3
- uses: actions/deploy-pages@v4
id: deployment
79 changes: 79 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
name: Tests

on:
workflow_dispatch:
push:
branches:
- master
pull_request:
branches:
- master

jobs:
test:

runs-on: ubuntu-22.04
strategy:
fail-fast: false
matrix:
python-version:
- "3.7"
- "3.8"
- "3.9"
- "3.10"
- "3.11"
- "3.12"
- "3.13"
- "pypy3.9"

steps:
- uses: actions/checkout@v4

- name: Check file permissions
run: |
if [[ "$(find ./gallery_dl -type f -not -perm 644)" ]]; then exit 1; fi

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

- name: Install dependencies
run: |
pip install -r requirements.txt
pip install flake8 youtube-dl

- name: Install yt-dlp
run: |
case "${{ matrix.python-version }}" in
3.4|3.5)
# don't install yt-dlp
;;
3.6|3.7|3.8)
# install from PyPI
pip install yt-dlp
;;
*)
# install from master
pip install https://github.com/yt-dlp/yt-dlp/archive/refs/heads/master.tar.gz
;;
esac

- name: Lint with flake8
run: |
case "${{ matrix.python-version }}" in
3.4|3.5|3.6|3.7)
flake8 --extend-exclude scripts/export_tests.py,scripts/pyprint.py .
;;
*)
flake8 .
;;
esac

- name: Run tests
run: |
make test

- name: Test autogeneration of man pages, bash/zsh/fish completion, etc
run: |
make
Loading