5. Optimizing Your First Config

Our config thus far looks like:

configuration:
  working_directory: '/tmp/ytdl-sub-downloads'
  dl_aliases:
    video: "--preset yt_video"
    video-url: "--youtube.video_url"
    type: "--overrides.video_type"
    single-music-video: "--preset yt_video --overrides.video_type 'Music Video' --youtube.video_url"

presets:
  yt_video:
    youtube:
      download_strategy: "video"

    output_options:
      output_directory: "/config/first_config/output_videos"
      file_name: "{video_path}.{ext}"
      thumbnail_name: "{video_path}.{thumbnail_ext}"

    overrides:
      video_name: "{upload_date_standardized}.{title_sanitized}"
      video_type: "{channel}"
      video_path: "{video_type_sanitized}/{video_name}"

  yt_playlist:
    preset: "yt_video"
    youtube:
      download_strategy: "playlist"

  yt_channel:
    preset: "yt_playlist"
    youtube:
      download_strategy: "channel"

In the last section, we previewed what it would look like if we downloaded Rick's music video playlist using the yt_playlist preset. Many videos were returned in the dry run.

Suppose we want to download this playlist and continually download new videos added to this playlist. Would ytdl-sub redownload videos that are already downloaded? With our current config, yes. Let's learn how we can optimize it to only download videos once.

Maintain Download Archive

In the output options, there is a field called maintain_download_archive that will record any entries we have successfully downloaded. This information gets stored in a JSON file in the output_directory. On successive downloads, ytdl-sub will look for this file and ensure any entry that has already been downloaded will not be downloaded again.

For YouTube playlists, channels, and SoundCloud artists, it's ideal to set this to True if you ever plan on downloading new videos from them.

Let's set this field to true in our yt_playlist preset.

  yt_playlist:
    preset: "yt_video"
    youtube:
      download_strategy: "playlist"
    output_options:
      maintain_download_archive: True

Now, let's download Rick's music video playlist using our subscription.yaml file:

rick_a_music_video_playlist:
  preset: "yt_playlist"
  youtube:
    playlist_url: "https://www.youtube.com/playlist?list=PLlaN88a7y2_plecYoJxvRFTLHVbIVAOoc"
  overrides:
    video_type: "Music Videos"

This will take some time depending on your download speed.

ytdl-sub sub

[ytdl-sub] Beginning subscription download for rick_a_official_music_videos
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Never Gonna Give You Up (Official Music Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Together Forever (Official Video) [Remastered in 4K]
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Unwanted (Official Song from the Podcast) (Lyric Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Angels On My Side (Official Music Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley ft.The Unsung Heroes - Every One of Us (Heroes Edit) (Official Music Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Love This Christmas (Official Music Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Take Me to Your Heart (Official Music Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Cry for Help (Official Music Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Whenever You Need Somebody (Official Music Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - She Wants To Dance With Me (Official Music Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Hold Me In Your Arms (Official Music Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Keep Singing (Official Music Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Never Gonna Give You Up (Live) (The Roxy 1987)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Beautiful Life (Official Music Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Dance (Official Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Giving Up On Love (Official Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Pray With Me (Official Music Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Hopelessly (Official Music Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Every One of Us (Rehearsal Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Never Gonna Give You Up (Live) (Top Of The Pops 1987)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Try (Official Music Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - She Makes Me (Official Music Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Walk Like A Panther (Official Music Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Lights Out (Official Music Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Giant (Tour Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Never Gonna Give You Up (Pianoforte) (Performance)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - It Would Take a Strong Strong Man (Official HD Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - The Ones You Love (Official HD Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - This Old House (Official Music Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley ft.The Unsung Heroes - Every One of Us (Lyric Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Whenever You Need Somebody (Live) (The Roxy 1987)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Love This Christmas (Lyric Video)
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Move Right Out (UK Version) [Official HD Video]
[ytdl-sub:downloader] Downloading metadata for Rick Astley - Giant (Official Audio)
[ytdl-sub:downloader] Downloading 1/36 Rick Astley - Beautiful Life (Official Music Video)
[ytdl-sub:downloader] Downloading 2/36 Rick Astley - Love This Christmas (Official Music Video)
[ytdl-sub:downloader] Downloading 4/36 Rick Astley - Together Forever (Official Video) [Remastered in 4K]
[ytdl-sub:downloader] Downloading 5/36 Rick Astley - Giving Up On Love (Official Video)
[ytdl-sub:downloader] Downloading 6/36 Rick Astley - Unwanted (Official Song from the Podcast) (Lyric Video)
[ytdl-sub:downloader] Downloading 7/36 Rick Astley - Whenever You Need Somebody (Official Music Video)
[ytdl-sub:downloader] Downloading 8/36 Rick Astley - Never Gonna Give You Up (Pianoforte) (Performance)
[ytdl-sub:downloader] Downloading 9/36 Rick Astley - She Makes Me (Official Music Video)
[ytdl-sub:downloader] Downloading 10/36 Rick Astley - The Ones You Love (Official HD Video)
[ytdl-sub:downloader] Downloading 11/36 Rick Astley - Never Gonna Give You Up (Live) (Top Of The Pops 1987)
[ytdl-sub:downloader] Downloading 12/36 Rick Astley - Lights Out (Official Music Video)
[ytdl-sub:downloader] Downloading 13/36 Rick Astley - Move Right Out (UK Version) [Official HD Video]
[ytdl-sub:downloader] Downloading 14/36 Rick Astley - She Wants To Dance With Me (Official Music Video)
[ytdl-sub:downloader] Downloading 15/36 Rick Astley - Hold Me In Your Arms (Official Music Video)
[ytdl-sub:downloader] Downloading 16/36 Rick Astley - Dance (Official Video)
[ytdl-sub:downloader] Downloading 17/36 Rick Astley ft.The Unsung Heroes - Every One of Us (Heroes Edit) (Official Music Video)
[ytdl-sub:downloader] Downloading 18/36 Rick Astley - Never Gonna Give You Up (Live) (The Roxy 1987)
[ytdl-sub:downloader] Downloading 19/36 Rick Astley - Whenever You Need Somebody (Live) (The Roxy 1987)
[ytdl-sub:downloader] Downloading 20/36 Rick Astley - Walk Like A Panther (Official Music Video)
[ytdl-sub:downloader] Downloading 21/36 Rick Astley - Hopelessly (Official Music Video)
[ytdl-sub:downloader] Downloading 22/36 Rick Astley - Keep Singing (Official Music Video)
[ytdl-sub:downloader] Downloading 23/36 Rick Astley - Never Gonna Give You Up (Official Music Video)
[ytdl-sub:downloader] Downloading 24/36 Rick Astley - Take Me to Your Heart (Official Music Video)
[ytdl-sub:downloader] Downloading 25/36 Rick Astley ft.The Unsung Heroes - Every One of Us (Lyric Video)
[ytdl-sub:downloader] Downloading 26/36 Rick Astley - Giant (Official Audio)
[ytdl-sub:downloader] Downloading 27/36 Rick Astley - Try (Official Music Video)
[ytdl-sub:downloader] Downloading 28/36 Rick Astley - Giant (Tour Video)
[ytdl-sub:downloader] Downloading 29/36 Rick Astley - Try (Champions League Tribute Version)
[ytdl-sub:downloader] Downloading 30/36 Rick Astley - This Old House (Official Music Video)
[ytdl-sub:downloader] Downloading 31/36 Rick Astley - Angels On My Side (Official Music Video)
[ytdl-sub:downloader] Downloading 32/36 Rick Astley - Cry for Help (Official Music Video)
[ytdl-sub:downloader] Downloading 33/36 Rick Astley - It Would Take a Strong Strong Man (Official HD Video)
[ytdl-sub:downloader] Downloading 34/36 Rick Astley - Love This Christmas (Lyric Video)
[ytdl-sub:downloader] Downloading 35/36 Rick Astley - Pray With Me (Official Music Video)
[ytdl-sub:downloader] Downloading 36/36 Rick Astley - Every One of Us (Rehearsal Video)
[ytdl-sub] Downloads for rick_a_official_music_videos:
Files created in '/config/first_config/output_videos'
----------------------------------------
.ytdl-sub-rick_a_official_music_videos-download-archive.json
Music Videos/2009-10-25.Rick Astley - Cry for Help (Official Music Video).jpg
Music Videos/2009-10-25.Rick Astley - Cry for Help (Official Music Video).webm
Music Videos/2009-10-25.Rick Astley - Hold Me In Your Arms (Official Music Video).jpg
Music Videos/2009-10-25.Rick Astley - Hold Me In Your Arms (Official Music Video).webm
Music Videos/2009-10-25.Rick Astley - Never Gonna Give You Up (Official Music Video).jpg
Music Videos/2009-10-25.Rick Astley - Never Gonna Give You Up (Official Music Video).mkv
Music Videos/2009-10-25.Rick Astley - Together Forever (Official Video) [Remastered in 4K].jpg
Music Videos/2009-10-25.Rick Astley - Together Forever (Official Video) [Remastered in 4K].webm
Music Videos/2009-10-25.Rick Astley - Whenever You Need Somebody (Official Music Video).jpg
Music Videos/2009-10-25.Rick Astley - Whenever You Need Somebody (Official Music Video).webm
Music Videos/2012-03-14.Rick Astley - Giving Up On Love (Official Video).jpg
Music Videos/2012-03-14.Rick Astley - Giving Up On Love (Official Video).webm
Music Videos/2012-03-14.Rick Astley - Hopelessly (Official Music Video).jpg
Music Videos/2012-03-14.Rick Astley - Hopelessly (Official Music Video).webm
Music Videos/2012-03-14.Rick Astley - She Wants To Dance With Me (Official Music Video).jpg
Music Videos/2012-03-14.Rick Astley - She Wants To Dance With Me (Official Music Video).webm
Music Videos/2012-03-14.Rick Astley - Take Me to Your Heart (Official Music Video).jpg
Music Videos/2012-03-14.Rick Astley - Take Me to Your Heart (Official Music Video).webm
Music Videos/2013-08-22.Rick Astley - Never Gonna Give You Up (Live) (Top Of The Pops 1987).jpg
Music Videos/2013-08-22.Rick Astley - Never Gonna Give You Up (Live) (Top Of The Pops 1987).webm
Music Videos/2015-02-01.Rick Astley - Lights Out (Official Music Video).jpg
Music Videos/2015-02-01.Rick Astley - Lights Out (Official Music Video).webm
Music Videos/2016-04-06.Rick Astley - Keep Singing (Official Music Video).jpg
Music Videos/2016-04-06.Rick Astley - Keep Singing (Official Music Video).webm
Music Videos/2016-06-09.Rick Astley - Angels On My Side (Official Music Video).jpg
Music Videos/2016-06-09.Rick Astley - Angels On My Side (Official Music Video).webm
Music Videos/2016-07-06.Rick Astley - Never Gonna Give You Up (Live) (The Roxy 1987).jpg
Music Videos/2016-07-06.Rick Astley - Never Gonna Give You Up (Live) (The Roxy 1987).webm
Music Videos/2016-07-06.Rick Astley - Whenever You Need Somebody (Live) (The Roxy 1987).jpg
Music Videos/2016-07-06.Rick Astley - Whenever You Need Somebody (Live) (The Roxy 1987).webm
Music Videos/2016-09-07.Rick Astley - Dance (Official Video).jpg
Music Videos/2016-09-07.Rick Astley - Dance (Official Video).webm
Music Videos/2016-11-10.Rick Astley - Pray With Me (Official Music Video).jpg
Music Videos/2016-11-10.Rick Astley - Pray With Me (Official Music Video).webm
Music Videos/2017-06-20.Rick Astley - This Old House (Official Music Video).jpg
Music Videos/2017-06-20.Rick Astley - This Old House (Official Music Video).webm
Music Videos/2018-03-09.Rick Astley - Walk Like A Panther (Official Music Video).jpg
Music Videos/2018-03-09.Rick Astley - Walk Like A Panther (Official Music Video).webm
Music Videos/2018-07-05.Rick Astley - Beautiful Life (Official Music Video).jpg
Music Videos/2018-07-05.Rick Astley - Beautiful Life (Official Music Video).webm
Music Videos/2018-09-13.Rick Astley - Try (Official Music Video).jpg
Music Videos/2018-09-13.Rick Astley - Try (Official Music Video).webm
Music Videos/2018-11-09.Rick Astley - She Makes Me (Official Music Video).jpg
Music Videos/2018-11-09.Rick Astley - She Makes Me (Official Music Video).webm
Music Videos/2019-05-31.Rick Astley - Try (Champions League Tribute Version).jpg
Music Videos/2019-05-31.Rick Astley - Try (Champions League Tribute Version).webm
Music Videos/2019-06-06.Rick Astley - Giant (Official Audio).jpg
Music Videos/2019-06-06.Rick Astley - Giant (Official Audio).webm
Music Videos/2019-06-19.Rick Astley - Giant (Tour Video).jpg
Music Videos/2019-06-19.Rick Astley - Giant (Tour Video).webm
Music Videos/2019-10-30.Rick Astley - Every One of Us (Rehearsal Video).jpg
Music Videos/2019-10-30.Rick Astley - Every One of Us (Rehearsal Video).webm
Music Videos/2019-11-19.Rick Astley - It Would Take a Strong Strong Man (Official HD Video).jpg
Music Videos/2019-11-19.Rick Astley - It Would Take a Strong Strong Man (Official HD Video).webm
Music Videos/2019-11-21.Rick Astley - Move Right Out (UK Version) [Official HD Video].jpg
Music Videos/2019-11-21.Rick Astley - Move Right Out (UK Version) [Official HD Video].webm
Music Videos/2019-11-28.Rick Astley - The Ones You Love (Official HD Video).jpg
Music Videos/2019-11-28.Rick Astley - The Ones You Love (Official HD Video).webm
Music Videos/2019-12-12.Rick Astley - Never Gonna Give You Up (Pianoforte) (Performance).jpg
Music Videos/2019-12-12.Rick Astley - Never Gonna Give You Up (Pianoforte) (Performance).webm
Music Videos/2020-10-15.Rick Astley ft.The Unsung Heroes - Every One of Us (Heroes Edit) (Official Music Video).jpg
Music Videos/2020-10-15.Rick Astley ft.The Unsung Heroes - Every One of Us (Heroes Edit) (Official Music Video).webm
Music Videos/2020-10-23.Rick Astley ft.The Unsung Heroes - Every One of Us (Lyric Video).jpg
Music Videos/2020-10-23.Rick Astley ft.The Unsung Heroes - Every One of Us (Lyric Video).webm
Music Videos/2020-11-30.Rick Astley - Love This Christmas (Lyric Video).jpg
Music Videos/2020-11-30.Rick Astley - Love This Christmas (Lyric Video).webm
Music Videos/2020-12-10.Rick Astley - Love This Christmas (Official Music Video).jpg
Music Videos/2020-12-10.Rick Astley - Love This Christmas (Official Music Video).webm
Music Videos/2021-04-08.Rick Astley - Unwanted (Official Song from the Podcast) (Lyric Video).jpg
Music Videos/2021-04-08.Rick Astley - Unwanted (Official Song from the Podcast) (Lyric Video).webm

Inspecting the Download Archive File

Notice the file .ytdl-sub-rick_a_official_music_videos-download-archive.json. This is the download archive JSON file that ytdl-sub will write and look for on a successive download.

Let's take a quick peek at what this file looks like:

head -n 10 output_videos/.ytdl-sub-rick_a_official_music_videos-download-archive.json 

{
  "3Z-O-XKL9cM": {
    "extractor": "youtube",
    "file_names": [
      "Music Videos/2019-05-31.Rick Astley - Try (Champions League Tribute Version).jpg",
      "Music Videos/2019-05-31.Rick Astley - Try (Champions League Tribute Version).webm"
    ],
    "upload_date": "2019-05-31"
  },
  "5o7WEv5Q5ZE": {

The structure is simple - it's a key-value store where the key is the entry's ytdl ID and the values include the ytdl extractor, files produced from it, and its upload date.