Release 2021.02.09

Closes: https://github.com/pukkandan/yt-dlp/issues/57#issuecomment-775227745

:ci skip dl

.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -21,7 +21,7 @@ ## Checklist
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.29. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.04. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ ## Checklist
 -->
 
 - [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running yt-dlp version **2021.01.29**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.04**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ ## Verbose log
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.01.29
+ [debug] yt-dlp version 2021.02.04
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -21,7 +21,7 @@ ## Checklist
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.29. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.04. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/pukkandan/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ ## Checklist
 -->
 
 - [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running yt-dlp version **2021.01.29**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.04**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -21,13 +21,13 @@ ## Checklist
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.29. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.04. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.01.29**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.04**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -21,7 +21,7 @@ ## Checklist
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.29. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.04. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -30,7 +30,7 @@ ## Checklist
 -->
 
 - [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running yt-dlp version **2021.01.29**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.04**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ ## Verbose log
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.01.29
+ [debug] yt-dlp version 2021.02.04
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -21,13 +21,13 @@ ## Checklist
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.29. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.04. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.01.29**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.04**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
 
 

.github/workflows/build.yml

@@ -84,14 +84,14 @@ jobs:
       with:
           python-version: '3.8'
     - name: Install Requirements
-      run: pip install pyinstaller mutagen
+      run: pip install pyinstaller mutagen Crypto
     - name: Bump version
       id: bump_version
       run: python devscripts/update-version.py
     - name: Print version
       run: echo "${{ steps.bump_version.outputs.ytdlc_version }}"
     - name: Run PyInstaller Script
-      run: python devscripts/pyinst.py 64
+      run: python pyinst.py 64
     - name: Upload youtube-dlc.exe Windows binary
       id: upload-release-windows
       uses: actions/upload-release-asset@v1
@@ -122,14 +122,14 @@ jobs:
           python-version: '3.4.4'
           architecture: 'x86'
     - name: Install Requirements for 32 Bit
-      run: pip install pyinstaller==3.5 mutagen
+      run: pip install pyinstaller==3.5 mutagen Crypto
     - name: Bump version
       id: bump_version
       run: python devscripts/update-version.py
     - name: Print version
       run: echo "${{ steps.bump_version.outputs.ytdlc_version }}"
     - name: Run PyInstaller Script for 32 Bit
-      run: python devscripts/pyinst.py 32
+      run: python pyinst.py 32
     - name: Upload Executable youtube-dlc_x86.exe
       id: upload-release-windows32
       uses: actions/upload-release-asset@v1

.gitignore

@@ -17,6 +17,7 @@ MANIFEST
 test/local_parameters.json
 .coverage
 cover/
+secrets/
 updates_key.pem
 *.egg-info
 .tox
@@ -58,6 +59,10 @@ youtube-dlc
 *.ogg
 *.opus
 *.info.json
+*.live_chat.json
+*.jpg
+*.png
+*.webp
 *.annotations.xml
 *.description
 

CONTRIBUTORS

@@ -17,3 +17,4 @@ alxnull
 FelixFrog
 Zocker1999NET
 nao20010128nao
+shirt-dev

Changelog.md

@@ -17,12 +17,31 @@ # Instuctions for creating release
 -->
 
 
+### 2021.02.09
+* **aria2c support for DASH/HLS**: by [shirt](https://github.com/shirt-dev)
+* **Implement Updater** (`-U`) by [shirt](https://github.com/shirt-dev)
+* [youtube] Fix comment extraction
+* [youtube_live_chat] Improve extraction
+* [youtube] Fix for channel URLs sometimes not downloading all pages
+* [aria2c] Changed default arguments to `--console-log-level=warn --summary-interval=0 --file-allocation=none -x16 -j16 -s16`
+* Add fallback for thumbnails
+* [embedthumbnail] Keep original thumbnail after conversion if write_thumbnail given
+* [embedsubtitle] Keep original subtitle after conversion if write_subtitles given
+* [pyinst.py] Move back to root dir
+* [youtube] Simplified renderer parsing and bugfixes
+* [movefiles] Fix compatibility with python2
+* [remuxvideo] Fix validation of conditional remux
+* [sponskrub] Don't raise error when the video does not exist
+* [documentation] Crypto is an optional dependency
+
+
 ### 2021.02.04
 * **Merge youtube-dl:** Upto [2021.02.04.1](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.02.04.1)
-* **Date/time formatting in output template:** You can now use [`strftime`](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) to format date/time fields. Example: `%(upload_date>%Y-%m-%d)s`
+* **Date/time formatting in output template:**
+    * You can use [`strftime`](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) to format date/time fields. Example: `%(upload_date>%Y-%m-%d)s`
 * **Multiple output templates:**
-    * Seperate output templates can be given for the different metadata files by using `-o TYPE:TEMPLATE`
+    * Separate output templates can be given for the different metadata files by using `-o TYPE:TEMPLATE`
-    * The alowed types are: `subtitle|thumbnail|description|annotation|infojson|pl_description|pl_infojson`
+    * The allowed types are: `subtitle|thumbnail|description|annotation|infojson|pl_description|pl_infojson`
 * [youtube] More metadata extraction for channel/playlist URLs (channel, uploader, thumbnail, tags)
 * New option `--no-write-playlist-metafiles` to prevent writing playlist metadata files
 * [audius] Fix extractor
@@ -38,7 +57,7 @@ ### 2021.02.04
 
 
 ### 2021.01.29
-* **Features from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl)**: Co-authored by @animelover1984 and @bbepis
+* **Features from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl)**: Co-authored by [animelover1984](https://github.com/animelover1984) and [bbepis](https://github.com/bbepis)
     * Add `--get-comments`
     * [youtube] Extract comments
     * [billibilli] Added BiliBiliSearchIE, BilibiliChannelIE
@@ -78,7 +97,7 @@ ### 2021.01.24
     * Valid types are: home, temp, description, annotation, subtitle, infojson, thumbnail
     * Additionally, configuration file is taken from home directory or current directory ([documentation](https://github.com/pukkandan/yt-dlp#:~:text=Home%20Configuration))
 * Allow passing different arguments to different external downloaders ([documentation](https://github.com/pukkandan/yt-dlp#:~:text=--downloader-args%20NAME:ARGS))
-* [mildom] Add extractor by @nao20010128nao
+* [mildom] Add extractor by [nao20010128nao](https://github.com/nao20010128nao)
 * Warn when using old style `--external-downloader-args` and `--post-processor-args`
 * Fix `--no-overwrite` when using `--write-link`
 * [sponskrub] Output `unrecognized argument` error message correctly
@@ -109,7 +128,7 @@ ### 2021.01.16
 
 ### 2021.01.14
 * Added option `--break-on-reject`
-* [roosterteeth.com] Fix for bonus episodes by @Zocker1999NET
+* [roosterteeth.com] Fix for bonus episodes by [Zocker1999NET](https://github.com/Zocker1999NET)
 * [tiktok] Fix for when share_info is empty
 * [EmbedThumbnail] Fix bug due to incorrect function name
 * [documentation] Changed sponskrub links to point to [pukkandan/sponskrub](https://github.com/pukkandan/SponSkrub) since I am now providing both linux and windows releases
@@ -118,18 +137,18 @@ ### 2021.01.14
 
 
 ### 2021.01.12
-* [roosterteeth.com] Add subtitle support by @samiksome
+* [roosterteeth.com] Add subtitle support by [samiksome](https://github.com/samiksome)
-* Added `--force-overwrites`, `--no-force-overwrites` by @alxnull
+* Added `--force-overwrites`, `--no-force-overwrites` by [alxnull](https://github.com/alxnull)
 * Changed fork name to `yt-dlp`
-* Fix typos by @FelixFrog
+* Fix typos by [FelixFrog](https://github.com/FelixFrog)
 * [ci] Option to skip
 * [changelog] Added unreleased changes in blackjack4494/yt-dlc
 
 
 ### 2021.01.10
-* [archive.org] Fix extractor and add support for audio and playlists by @wporr
+* [archive.org] Fix extractor and add support for audio and playlists by [wporr](https://github.com/wporr)
-* [Animelab] Added by @mariuszskon
+* [Animelab] Added by [mariuszskon](https://github.com/mariuszskon)
-* [youtube:search] Fix view_count by @ohnonot
+* [youtube:search] Fix view_count by [ohnonot](https://github.com/ohnonot)
 * [youtube] Show if video is embeddable in info
 * Update version badge automatically in README
 * Enable `test_youtube_search_matching`
@@ -138,11 +157,11 @@ ### 2021.01.10
 
 ### 2021.01.09
 * [youtube] Fix bug in automatic caption extraction
-* Add `post_hooks` to YoutubeDL by @alexmerkel
+* Add `post_hooks` to YoutubeDL by [alexmerkel](https://github.com/alexmerkel)
-* Batch file enumeration improvements by @glenn-slayden
+* Batch file enumeration improvements by [glenn-slayden](https://github.com/glenn-slayden)
-* Stop immediately when reaching `--max-downloads` by @glenn-slayden
+* Stop immediately when reaching `--max-downloads` by [glenn-slayden](https://github.com/glenn-slayden)
-* Fix incorrect ANSI sequence for restoring console-window title by @glenn-slayden
+* Fix incorrect ANSI sequence for restoring console-window title by [glenn-slayden](https://github.com/glenn-slayden)
-* Kill child processes when yt-dlc is killed by @Unrud
+* Kill child processes when yt-dlc is killed by [Unrud](https://github.com/Unrud)
 
 
 ### 2021.01.08
@@ -152,11 +171,11 @@ ### 2021.01.08
 
 
 ### 2021.01.07-1
-* [Akamai] fix by @nixxo
+* [Akamai] fix by [nixxo](https://github.com/nixxo)
-* [Tiktok] merge youtube-dl tiktok extractor by @GreyAlien502
+* [Tiktok] merge youtube-dl tiktok extractor by [GreyAlien502](https://github.com/GreyAlien502)
-* [vlive] add support for playlists by @kyuyeunk
+* [vlive] add support for playlists by [kyuyeunk](https://github.com/kyuyeunk)
-* [youtube_live_chat] make sure playerOffsetMs is positive by @siikamiika
+* [youtube_live_chat] make sure playerOffsetMs is positive by [siikamiika](https://github.com/siikamiika)
-* Ignore extra data streams in ffmpeg by @jbruchon
+* Ignore extra data streams in ffmpeg by [jbruchon](https://github.com/jbruchon)
 * Allow passing different arguments to different postprocessors using `--postprocessor-args`
 * Deprecated `--sponskrub-args`. The same can now be done using `--postprocessor-args "sponskrub:<args>"`
 * [CI] Split tests into core-test and full-test
@@ -186,15 +205,15 @@ ### 2021.01.05
     * Changed video format sorting to show video only files and video+audio files together.
     * Added `--video-multistreams`, `--no-video-multistreams`, `--audio-multistreams`, `--no-audio-multistreams`
     * Added `b`,`w`,`v`,`a` as alias for `best`, `worst`, `video` and `audio` respectively
-* **Shortcut Options:** Added `--write-link`, `--write-url-link`, `--write-webloc-link`, `--write-desktop-link` by @h-h-h-h - See [Internet Shortcut Options]README.md(#internet-shortcut-options) for details
+* **Shortcut Options:** Added `--write-link`, `--write-url-link`, `--write-webloc-link`, `--write-desktop-link` by [h-h-h-h](https://github.com/h-h-h-h) - See [Internet Shortcut Options]README.md(#internet-shortcut-options) for details
 * **Sponskrub integration:** Added `--sponskrub`, `--sponskrub-cut`, `--sponskrub-force`, `--sponskrub-location`, `--sponskrub-args` - See [SponSkrub Options](README.md#sponskrub-options-sponsorblock) for details
-* Added `--force-download-archive` (`--force-write-archive`) by @h-h-h-h
+* Added `--force-download-archive` (`--force-write-archive`) by [h-h-h-h](https://github.com/h-h-h-h)
 * Added `--list-formats-as-table`,  `--list-formats-old`
 * **Negative Options:** Makes it possible to negate most boolean options by adding a `no-` to the switch. Usefull when you want to reverse an option that is defined in a config file
     * Added `--no-ignore-dynamic-mpd`, `--no-allow-dynamic-mpd`, `--allow-dynamic-mpd`, `--youtube-include-hls-manifest`, `--no-youtube-include-hls-manifest`, `--no-youtube-skip-hls-manifest`, `--no-download`, `--no-download-archive`, `--resize-buffer`, `--part`, `--mtime`, `--no-keep-fragments`, `--no-cookies`, `--no-write-annotations`, `--no-write-info-json`, `--no-write-description`, `--no-write-thumbnail`, `--youtube-include-dash-manifest`, `--post-overwrites`, `--no-keep-video`, `--no-embed-subs`, `--no-embed-thumbnail`, `--no-add-metadata`, `--no-include-ads`, `--no-write-sub`, `--no-write-auto-sub`, `--no-playlist-reverse`, `--no-restrict-filenames`, `--youtube-include-dash-manifest`, `--no-format-sort-force`, `--flat-videos`, `--no-list-formats-as-table`, `--no-sponskrub`, `--no-sponskrub-cut`, `--no-sponskrub-force`
     * Renamed: `--write-subs`, `--no-write-subs`, `--no-write-auto-subs`, `--write-auto-subs`. Note that these can still be used without the ending "s"
 * Relaxed validation for format filters so that any arbitrary field can be used
-* Fix for embedding thumbnail in mp3 by @pauldubois98 ([ytdl-org/youtube-dl#21569](https://github.com/ytdl-org/youtube-dl/pull/21569))
+* Fix for embedding thumbnail in mp3 by [pauldubois98](https://github.com/pauldubois98) ([ytdl-org/youtube-dl#21569](https://github.com/ytdl-org/youtube-dl/pull/21569))
 * Make Twitch Video ID output from Playlist and VOD extractor same. This is only a temporary fix
 * **Merge youtube-dl:** Upto [2021.01.03](https://github.com/ytdl-org/youtube-dl/commit/8e953dcbb10a1a42f4e12e4e132657cb0100a1f8) - See [blackjack4494/yt-dlc#280](https://github.com/blackjack4494/yt-dlc/pull/280) for details
     * Extractors [tiktok](https://github.com/ytdl-org/youtube-dl/commit/fb626c05867deab04425bad0c0b16b55473841a2) and [hotstar](https://github.com/ytdl-org/youtube-dl/commit/bb38a1215718cdf36d73ff0a7830a64cd9fa37cc) have not been merged
@@ -211,7 +230,7 @@ ### Unreleased changes in [blackjack4494/yt-dlc](https://github.com/blackjack449
     * Redirect channel home to /video
     * Print youtube's warning message
     * Multiple pages are handled better for feeds
-* Add --break-on-existing by @gergesh
+* Add --break-on-existing by [gergesh](https://github.com/gergesh)
 * Pre-check video IDs in the archive before downloading
 * [bitwave.tv] New extractor
 * [Gedi] Add extractor

README.md

@@ -3,6 +3,8 @@ # YT-DLP
 [![Release version](https://img.shields.io/github/v/release/pukkandan/yt-dlp?color=brightgreen&label=Release)](https://github.com/pukkandan/yt-dlp/releases/latest)
 [![License: Unlicense](https://img.shields.io/badge/License-Unlicense-blue.svg)](LICENSE)
 [![CI Status](https://github.com/pukkandan/yt-dlp/workflows/Core%20Tests/badge.svg?branch=master)](https://github.com/pukkandan/yt-dlp/actions)
+[![Discord](https://img.shields.io/discord/807245652072857610?color=blue&label=discord&logo=discord)](https://discord.gg/S75JaBna)  
+
 [![Commits](https://img.shields.io/github/commit-activity/m/pukkandan/yt-dlp?label=commits)](https://github.com/pukkandan/yt-dlp/commits)
 [![Last Commit](https://img.shields.io/github/last-commit/pukkandan/yt-dlp/master)](https://github.com/pukkandan/yt-dlp/commits)
 [![Downloads](https://img.shields.io/github/downloads/pukkandan/yt-dlp/total)](https://github.com/pukkandan/yt-dlp/releases/latest)
@@ -102,11 +104,11 @@ ### UPDATE
 ### COMPILE
 
 **For Windows**:
-To build the Windows executable, you must have pyinstaller (and optionally mutagen for embedding thumbnail in opus/ogg files)
+To build the Windows executable, you must have pyinstaller (and optionally mutagen and Crypto)
 
-    python -m pip install --upgrade pyinstaller mutagen
+    python -m pip install --upgrade pyinstaller mutagen Crypto
 
-Once you have all the necessary dependancies installed, just run `py devscripts\pyinst.py`. The executable will be built for the same architecture (32/64 bit) as the python used to build it. It is strongly reccomended to use python3 although python2.6+ is supported.
+Once you have all the necessary dependancies installed, just run `py pyinst.py`. The executable will be built for the same architecture (32/64 bit) as the python used to build it. It is strongly reccomended to use python3 although python2.6+ is supported.
 
 You can also build the executable without any version info or metadata by using:
 

devscripts/update-version.py

@@ -8,7 +8,7 @@
 exec(compile(open('youtube_dlc/version.py').read(), 'youtube_dlc/version.py', 'exec'))
 old_version = locals()['__version__']
 
-old_version_list = old_version.replace('-', '.').split(".", 4)
+old_version_list = old_version.split(".", 4)
 
 old_ver = '.'.join(old_version_list[:3])
 old_rev = old_version_list[3] if len(old_version_list) > 3 else ''

docs/supportedsites.md

@@ -1218,9 +1218,9 @@ # Supported sites
  - **youtube:history**: Youtube watch history, ":ythistory" for short (requires authentication)
  - **youtube:playlist**: YouTube.com playlists
  - **youtube:recommended**: YouTube.com recommended videos, ":ytrec" for short (requires authentication)
- - **youtube:search**: YouTube.com searches
+ - **youtube:search**: YouTube.com searches, "ytsearch" keyword
  - **youtube:search:date**: YouTube.com searches, newest videos first, "ytsearchdate" keyword
- - **youtube:search_url**: YouTube.com searches, "ytsearch" keyword
+ - **youtube:search_url**: YouTube.com search URLs
  - **youtube:subscriptions**: YouTube.com subscriptions feed, ":ytsubs" for short (requires authentication)
  - **youtube:tab**: YouTube.com tab
  - **youtube:watchlater**: Youtube watch later list, ":ytwatchlater" for short (requires authentication)

pyinst.py

@@ -3,7 +3,7 @@
 
 from __future__ import unicode_literals
 import sys
-import os
+# import os
 import platform
 
 from PyInstaller.utils.win32.versioninfo import (
@@ -18,16 +18,15 @@
 _x86 = '_x86' if arch == '32' else ''
 
 FILE_DESCRIPTION = 'Media Downloader%s' % (' (32 Bit)' if _x86 else '')
-SHORT_URLS = {'32': 'git.io/JUGsM', '64': 'git.io/JLh7K'}
 
-root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
+# root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
-print('Changing working directory to %s' % root_dir)
+# print('Changing working directory to %s' % root_dir)
-os.chdir(root_dir)
+# os.chdir(root_dir)
 
 exec(compile(open('youtube_dlc/version.py').read(), 'youtube_dlc/version.py', 'exec'))
 VERSION = locals()['__version__']
 
-VERSION_LIST = VERSION.replace('-', '.').split('.')
+VERSION_LIST = VERSION.split('.')
 VERSION_LIST = list(map(int, VERSION_LIST)) + [0] * (4 - len(VERSION_LIST))
 
 print('Version: %s%s' % (VERSION, _x86))
@@ -49,7 +48,7 @@
             StringTable(
                 '040904B0', [
                     StringStruct('Comments', 'Youtube-dlc%s Command Line Interface.' % _x86),
-                    StringStruct('CompanyName', 'pukkandan@gmail.com'),
+                    StringStruct('CompanyName', 'https://github.com/pukkandan/yt-dlp'),
                     StringStruct('FileDescription', FILE_DESCRIPTION),
                     StringStruct('FileVersion', VERSION),
                     StringStruct('InternalName', 'youtube-dlc%s' % _x86),
@@ -59,7 +58,7 @@
                     ),
                     StringStruct('OriginalFilename', 'youtube-dlc%s.exe' % _x86),
                     StringStruct('ProductName', 'Youtube-dlc%s' % _x86),
-                    StringStruct('ProductVersion', '%s%s | %s' % (VERSION, _x86, SHORT_URLS[arch])),
+                    StringStruct('ProductVersion', '%s%s' % (VERSION, _x86)),
                 ])]),
         VarFileInfo([VarStruct('Translation', [0, 1200])])
     ]
@@ -73,6 +72,7 @@
     '--exclude-module=test',
     '--exclude-module=ytdlp_plugins',
     '--hidden-import=mutagen',
+    '--hidden-import=Crypto',
     'youtube_dlc/__main__.py',
 ])
 SetVersion('dist/youtube-dlc%s.exe' % _x86, VERSION_FILE)

requirements.txt

@@ -1 +1,2 @@
 mutagen
+Crypto

setup.py

@@ -20,7 +20,7 @@
     '**PS**: Many links in this document will not work since this is a copy of the README.md from Github',
     open("README.md", "r", encoding="utf-8").read()))
 
-REQUIREMENTS = ['mutagen']
+REQUIREMENTS = ['mutagen', 'Crypto']
 
 
 if len(sys.argv) >= 2 and sys.argv[1] == 'py2exe':

youtube_dlc/YoutubeDL.py

@@ -2292,12 +2292,9 @@ def existing_file(*filepaths):
                     downloaded = []
                     merger = FFmpegMergerPP(self)
                     if not merger.available:
-                        postprocessors = []
                         self.report_warning('You have requested multiple '
                                             'formats but ffmpeg is not installed.'
                                             ' The formats won\'t be merged.')
-                    else:
-                        postprocessors = [merger]
 
                     def compatible_formats(formats):
                         # TODO: some formats actually allow this (mkv, webm, ogg, mp4), but not all of them.
@@ -2349,7 +2346,8 @@ def correct_ext(filename):
                             downloaded.append(fname)
                             partial_success, real_download = dl(fname, new_info)
                             success = success and partial_success
-                        info_dict['__postprocessors'] = postprocessors
+                        if merger.available:
+                            info_dict['__postprocessors'].append(merger)
                         info_dict['__files_to_merge'] = downloaded
                         # Even if there were no downloads, it is being merged only now
                         info_dict['__real_download'] = True
@@ -2895,20 +2893,17 @@ def get_encoding(self):
         return encoding
 
     def _write_thumbnails(self, info_dict, filename):  # return the extensions
-        if self.params.get('writethumbnail', False):
+        write_all = self.params.get('write_all_thumbnails', False)
-            thumbnails = info_dict.get('thumbnails')
-            if thumbnails:
-                thumbnails = [thumbnails[-1]]
-        elif self.params.get('write_all_thumbnails', False):
-            thumbnails = info_dict.get('thumbnails') or []
-        else:
         thumbnails = []
+        if write_all or self.params.get('writethumbnail', False):
+            thumbnails = info_dict.get('thumbnails') or []
+        multiple = write_all and len(thumbnails) > 1
 
         ret = []
-        for t in thumbnails:
+        for t in thumbnails[::1 if write_all else -1]:
             thumb_ext = determine_ext(t['url'], 'jpg')
-            suffix = '%s.' % t['id'] if len(thumbnails) > 1 else ''
+            suffix = '%s.' % t['id'] if multiple else ''
-            thumb_display_id = '%s ' % t['id'] if len(thumbnails) > 1 else ''
+            thumb_display_id = '%s ' % t['id'] if multiple else ''
             t['filename'] = thumb_filename = replace_extension(filename, suffix + thumb_ext, info_dict.get('ext'))
 
             if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(thumb_filename)):
@@ -2928,4 +2923,6 @@ def _write_thumbnails(self, info_dict, filename):  # return the extensions
                 except (compat_urllib_error.URLError, compat_http_client.HTTPException, socket.error) as err:
                     self.report_warning('Unable to download thumbnail "%s": %s' %
                                         (t['url'], error_to_compat_str(err)))
+            if ret and not write_all:
+                break
         return ret

youtube_dlc/__init__.py

@@ -15,7 +15,6 @@
 
 from .options import (
     parseOpts,
-    _remux_formats,
 )
 from .compat import (
     compat_getpass,
@@ -24,7 +23,6 @@
 from .utils import (
     DateRange,
     decodeOption,
-    DEFAULT_OUTTMPL,
     DownloadError,
     ExistingVideoReached,
     expand_path,
@@ -33,11 +31,12 @@
     preferredencoding,
     read_batch_urls,
     RejectedVideoReached,
+    REMUX_EXTENSIONS,
+    render_table,
     SameFileError,
     setproctitle,
     std_headers,
     write_string,
-    render_table,
 )
 from .update import update_self
 from .downloader import (
@@ -211,13 +210,15 @@ def parse_retries(retries):
         if not opts.audioquality.isdigit():
             parser.error('invalid audio quality specified')
     if opts.recodevideo is not None:
-        if opts.recodevideo not in _remux_formats:
+        if opts.recodevideo not in REMUX_EXTENSIONS:
             parser.error('invalid video recode format specified')
     if opts.remuxvideo and opts.recodevideo:
         opts.remuxvideo = None
         write_string('WARNING: --remux-video is ignored since --recode-video was given\n', out=sys.stderr)
     if opts.remuxvideo is not None:
-        if opts.remuxvideo not in _remux_formats:
+        opts.remuxvideo = opts.remuxvideo.replace(' ', '')
+        remux_regex = r'{0}(?:/{0})*$'.format(r'(?:\w+>)?(?:%s)' % '|'.join(REMUX_EXTENSIONS))
+        if not re.match(remux_regex, opts.remuxvideo):
             parser.error('invalid video remux format specified')
     if opts.convertsubtitles is not None:
         if opts.convertsubtitles not in ['srt', 'vtt', 'ass', 'lrc']:
@@ -232,11 +233,6 @@ def parse_retries(retries):
     if opts.extractaudio and not opts.keepvideo and opts.format is None:
         opts.format = 'bestaudio/best'
 
-    # --all-sub automatically sets --write-sub if --write-auto-sub is not given
-    # this was the old behaviour if only --all-sub was given.
-    if opts.allsubtitles and not opts.writeautomaticsub:
-        opts.writesubtitles = True
-
     outtmpl = opts.outtmpl
     if not outtmpl:
         outtmpl = {'default': (
@@ -310,9 +306,17 @@ def parse_retries(retries):
             'format': opts.convertsubtitles,
         })
     if opts.embedsubtitles:
+        already_have_subtitle = opts.writesubtitles
         postprocessors.append({
             'key': 'FFmpegEmbedSubtitle',
+            'already_have_subtitle': already_have_subtitle
         })
+        if not already_have_subtitle:
+            opts.writesubtitles = True
+    # --all-sub automatically sets --write-sub if --write-auto-sub is not given
+    # this was the old behaviour if only --all-sub was given.
+    if opts.allsubtitles and not opts.writeautomaticsub:
+        opts.writesubtitles = True
     if opts.embedthumbnail:
         already_have_thumbnail = opts.writethumbnail or opts.write_all_thumbnails
         postprocessors.append({
@@ -353,7 +357,11 @@ def parse_retries(retries):
         opts.postprocessor_args.setdefault('sponskrub', [])
         opts.postprocessor_args['default'] = opts.postprocessor_args['default-compat']
 
-    audio_ext = opts.audioformat if (opts.extractaudio and opts.audioformat != 'best') else None
+    final_ext = (
+        opts.recodevideo
+        or (opts.remuxvideo in REMUX_EXTENSIONS) and opts.remuxvideo
+        or (opts.extractaudio and opts.audioformat != 'best') and opts.audioformat
+        or None)
 
     match_filter = (
         None if opts.match_filter is None
@@ -474,7 +482,7 @@ def parse_retries(retries):
         'extract_flat': opts.extract_flat,
         'mark_watched': opts.mark_watched,
         'merge_output_format': opts.merge_output_format,
-        'final_ext': opts.recodevideo or opts.remuxvideo or audio_ext,
+        'final_ext': final_ext,
         'postprocessors': postprocessors,
         'fixup': opts.fixup,
         'source_address': opts.source_address,

youtube_dlc/downloader/__init__.py

@@ -1,11 +1,24 @@
 from __future__ import unicode_literals
 
+from ..utils import (
+    determine_protocol,
+)
+
+
+def _get_real_downloader(info_dict, protocol=None, *args, **kwargs):
+    info_copy = info_dict.copy()
+    if protocol:
+        info_copy['protocol'] = protocol
+    return get_suitable_downloader(info_copy, *args, **kwargs)
+
+
+# Some of these require _get_real_downloader
 from .common import FileDownloader
+from .dash import DashSegmentsFD
 from .f4m import F4mFD
 from .hls import HlsFD
 from .http import HttpFD
 from .rtmp import RtmpFD
-from .dash import DashSegmentsFD
 from .rtsp import RtspFD
 from .ism import IsmFD
 from .youtube_live_chat import YoutubeLiveChatReplayFD
@@ -14,10 +27,6 @@
     FFmpegFD,
 )
 
-from ..utils import (
-    determine_protocol,
-)
-
 PROTOCOL_MAP = {
     'rtmp': RtmpFD,
     'm3u8_native': HlsFD,
@@ -31,7 +40,7 @@
 }
 
 
-def get_suitable_downloader(info_dict, params={}):
+def get_suitable_downloader(info_dict, params={}, default=HttpFD):
     """Get the downloader class that can handle the info dict."""
     protocol = determine_protocol(info_dict)
     info_dict['protocol'] = protocol
@@ -45,16 +54,17 @@ def get_suitable_downloader(info_dict, params={}):
         if ed.can_download(info_dict):
             return ed
 
-    if protocol.startswith('m3u8') and info_dict.get('is_live'):
+    if protocol.startswith('m3u8'):
+        if info_dict.get('is_live'):
             return FFmpegFD
-
+        elif _get_real_downloader(info_dict, 'frag_urls', params, None):
-    if protocol == 'm3u8' and params.get('hls_prefer_native') is True:
             return HlsFD
-
+        elif params.get('hls_prefer_native') is True:
-    if protocol == 'm3u8_native' and params.get('hls_prefer_native') is False:
+            return HlsFD
+        elif params.get('hls_prefer_native') is False:
             return FFmpegFD
 
-    return PROTOCOL_MAP.get(protocol, HttpFD)
+    return PROTOCOL_MAP.get(protocol, default)
 
 
 __all__ = [

youtube_dlc/downloader/dash.py

@@ -1,6 +1,8 @@
 from __future__ import unicode_literals
 
+from ..downloader import _get_real_downloader
 from .fragment import FragmentFD
+
 from ..compat import compat_urllib_error
 from ..utils import (
     DownloadError,
@@ -20,31 +22,42 @@ def real_download(self, filename, info_dict):
         fragments = info_dict['fragments'][:1] if self.params.get(
             'test', False) else info_dict['fragments']
 
+        real_downloader = _get_real_downloader(info_dict, 'frag_urls', self.params, None)
+
         ctx = {
             'filename': filename,
             'total_frags': len(fragments),
         }
 
+        if real_downloader:
+            self._prepare_external_frag_download(ctx)
+        else:
             self._prepare_and_start_frag_download(ctx)
 
         fragment_retries = self.params.get('fragment_retries', 0)
         skip_unavailable_fragments = self.params.get('skip_unavailable_fragments', True)
 
+        fragment_urls = []
         frag_index = 0
         for i, fragment in enumerate(fragments):
             frag_index += 1
             if frag_index <= ctx['fragment_index']:
                 continue
+            fragment_url = fragment.get('url')
+            if not fragment_url:
+                assert fragment_base_url
+                fragment_url = urljoin(fragment_base_url, fragment['path'])
+
+            if real_downloader:
+                fragment_urls.append(fragment_url)
+                continue
+
             # In DASH, the first segment contains necessary headers to
             # generate a valid MP4 file, so always abort for the first segment
             fatal = i == 0 or not skip_unavailable_fragments
             count = 0
             while count <= fragment_retries:
                 try:
-                    fragment_url = fragment.get('url')
-                    if not fragment_url:
-                        assert fragment_base_url
-                        fragment_url = urljoin(fragment_base_url, fragment['path'])
                     success, frag_content = self._download_fragment(ctx, fragment_url, info_dict)
                     if not success:
                         return False
@@ -75,6 +88,16 @@ def real_download(self, filename, info_dict):
                 self.report_error('giving up after %s fragment retries' % fragment_retries)
                 return False
 
+        if real_downloader:
+            info_copy = info_dict.copy()
+            info_copy['url_list'] = fragment_urls
+            fd = real_downloader(self.ydl, self.params)
+            # TODO: Make progress updates work without hooking twice
+            # for ph in self._progress_hooks:
+            #     fd.add_progress_hook(ph)
+            success = fd.real_download(filename, info_copy)
+            if not success:
+                return False
+        else:
             self._finish_frag_download(ctx)
-
         return True

youtube_dlc/downloader/external.py

@@ -5,6 +5,13 @@
 import subprocess
 import sys
 import time
+import shutil
+
+try:
+    from Crypto.Cipher import AES
+    can_decrypt_frag = True
+except ImportError:
+    can_decrypt_frag = False
 
 from .common import FileDownloader
 from ..compat import (
@@ -18,15 +25,19 @@
     cli_bool_option,
     cli_configuration_args,
     encodeFilename,
+    error_to_compat_str,
     encodeArgument,
     handle_youtubedl_headers,
     check_executable,
     is_outdated_version,
     process_communicate_or_kill,
+    sanitized_Request,
 )
 
 
 class ExternalFD(FileDownloader):
+    SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps')
+
     def real_download(self, filename, info_dict):
         self.report_destination(filename)
         tmpfilename = self.temp_name(filename)
@@ -79,7 +90,7 @@ def available(cls):
 
     @classmethod
     def supports(cls, info_dict):
-        return info_dict['protocol'] in ('http', 'https', 'ftp', 'ftps')
+        return info_dict['protocol'] in cls.SUPPORTED_PROTOCOLS
 
     @classmethod
     def can_download(cls, info_dict):
@@ -109,8 +120,47 @@ def _call_downloader(self, tmpfilename, info_dict):
         _, stderr = process_communicate_or_kill(p)
         if p.returncode != 0:
             self.to_stderr(stderr.decode('utf-8', 'replace'))
+
+        if 'url_list' in info_dict:
+            file_list = []
+            for [i, url] in enumerate(info_dict['url_list']):
+                tmpsegmentname = '%s_%s.frag' % (tmpfilename, i)
+                file_list.append(tmpsegmentname)
+            with open(tmpfilename, 'wb') as dest:
+                for i in file_list:
+                    if 'decrypt_info' in info_dict:
+                        decrypt_info = info_dict['decrypt_info']
+                        with open(i, 'rb') as src:
+                            if decrypt_info['METHOD'] == 'AES-128':
+                                iv = decrypt_info.get('IV')
+                                decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
+                                    self._prepare_url(info_dict, info_dict.get('_decryption_key_url') or decrypt_info['URI'])).read()
+                                encrypted_data = src.read()
+                                decrypted_data = AES.new(
+                                    decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(encrypted_data)
+                                dest.write(decrypted_data)
+                            else:
+                                shutil.copyfileobj(open(i, 'rb'), dest)
+                    else:
+                        shutil.copyfileobj(open(i, 'rb'), dest)
+            if not self.params.get('keep_fragments', False):
+                for file_path in file_list:
+                    try:
+                        os.remove(file_path)
+                    except OSError as ose:
+                        self.report_error("Unable to delete file %s; %s" % (file_path, error_to_compat_str(ose)))
+                try:
+                    file_path = '%s.frag.urls' % tmpfilename
+                    os.remove(file_path)
+                except OSError as ose:
+                    self.report_error("Unable to delete file %s; %s" % (file_path, error_to_compat_str(ose)))
+
         return p.returncode
 
+    def _prepare_url(self, info_dict, url):
+        headers = info_dict.get('http_headers')
+        return sanitized_Request(url, None, headers) if headers else url
+
 
 class CurlFD(ExternalFD):
     AVAILABLE_OPT = '-V'
@@ -186,15 +236,17 @@ def _make_cmd(self, tmpfilename, info_dict):
 
 class Aria2cFD(ExternalFD):
     AVAILABLE_OPT = '-v'
+    SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps', 'frag_urls')
 
     def _make_cmd(self, tmpfilename, info_dict):
         cmd = [self.exe, '-c']
-        cmd += self._configuration_args([
-            '--min-split-size', '1M', '--max-connection-per-server', '4'])
         dn = os.path.dirname(tmpfilename)
+        if 'url_list' not in info_dict:
+            cmd += ['--out', os.path.basename(tmpfilename)]
+        verbose_level_args = ['--console-log-level=warn', '--summary-interval=0']
+        cmd += self._configuration_args(['--file-allocation=none', '-x16', '-j16', '-s16'] + verbose_level_args)
         if dn:
             cmd += ['--dir', dn]
-        cmd += ['--out', os.path.basename(tmpfilename)]
         if info_dict.get('http_headers') is not None:
             for key, val in info_dict['http_headers'].items():
                 cmd += ['--header', '%s: %s' % (key, val)]
@@ -202,6 +254,20 @@ def _make_cmd(self, tmpfilename, info_dict):
         cmd += self._option('--all-proxy', 'proxy')
         cmd += self._bool_option('--check-certificate', 'nocheckcertificate', 'false', 'true', '=')
         cmd += self._bool_option('--remote-time', 'updatetime', 'true', 'false', '=')
+        cmd += ['--auto-file-renaming=false']
+        if 'url_list' in info_dict:
+            cmd += verbose_level_args
+            cmd += ['--uri-selector', 'inorder', '--download-result=hide']
+            url_list_file = '%s.frag.urls' % tmpfilename
+            url_list = []
+            for [i, url] in enumerate(info_dict['url_list']):
+                tmpsegmentname = '%s_%s.frag' % (os.path.basename(tmpfilename), i)
+                url_list.append('%s\n\tout=%s' % (url, tmpsegmentname))
+            with open(url_list_file, 'w') as f:
+                f.write('\n'.join(url_list))
+
+            cmd += ['-i', url_list_file]
+        else:
             cmd += ['--', info_dict['url']]
         return cmd
 
@@ -221,9 +287,7 @@ def _make_cmd(self, tmpfilename, info_dict):
 
 
 class FFmpegFD(ExternalFD):
-    @classmethod
+    SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps', 'm3u8', 'rtsp', 'rtmp', 'mms')
-    def supports(cls, info_dict):
-        return info_dict['protocol'] in ('http', 'https', 'ftp', 'ftps', 'm3u8', 'rtsp', 'rtmp', 'mms')
 
     @classmethod
     def available(cls):

youtube_dlc/downloader/fragment.py

@@ -277,3 +277,24 @@ def _finish_frag_download(self, ctx):
             'status': 'finished',
             'elapsed': elapsed,
         })
+
+    def _prepare_external_frag_download(self, ctx):
+        if 'live' not in ctx:
+            ctx['live'] = False
+        if not ctx['live']:
+            total_frags_str = '%d' % ctx['total_frags']
+            ad_frags = ctx.get('ad_frags', 0)
+            if ad_frags:
+                total_frags_str += ' (not including %d ad)' % ad_frags
+        else:
+            total_frags_str = 'unknown (live)'
+        self.to_screen(
+            '[%s] Total fragments: %s' % (self.FD_NAME, total_frags_str))
+
+        tmpfilename = self.temp_name(ctx['filename'])
+
+        # Should be initialized before ytdl file check
+        ctx.update({
+            'tmpfilename': tmpfilename,
+            'fragment_index': 0,
+        })

youtube_dlc/downloader/hls.py

@@ -8,6 +8,7 @@
 except ImportError:
     can_decrypt_frag = False
 
+from ..downloader import _get_real_downloader
 from .fragment import FragmentFD
 from .external import FFmpegFD
 
@@ -73,10 +74,13 @@ def real_download(self, filename, info_dict):
                 'hlsnative has detected features it does not support, '
                 'extraction will be delegated to ffmpeg')
             fd = FFmpegFD(self.ydl, self.params)
-            for ph in self._progress_hooks:
+            # TODO: Make progress updates work without hooking twice
-                fd.add_progress_hook(ph)
+            # for ph in self._progress_hooks:
+            #     fd.add_progress_hook(ph)
             return fd.real_download(filename, info_dict)
 
+        real_downloader = _get_real_downloader(info_dict, 'frag_urls', self.params, None)
+
         def is_ad_fragment_start(s):
             return (s.startswith('#ANVATO-SEGMENT-INFO') and 'type=ad' in s
                     or s.startswith('#UPLYNK-SEGMENT') and s.endswith(',ad'))
@@ -85,6 +89,8 @@ def is_ad_fragment_end(s):
             return (s.startswith('#ANVATO-SEGMENT-INFO') and 'type=master' in s
                     or s.startswith('#UPLYNK-SEGMENT') and s.endswith(',segment'))
 
+        fragment_urls = []
+
         media_frags = 0
         ad_frags = 0
         ad_frag_next = False
@@ -109,6 +115,9 @@ def is_ad_fragment_end(s):
             'ad_frags': ad_frags,
         }
 
+        if real_downloader:
+            self._prepare_external_frag_download(ctx)
+        else:
             self._prepare_and_start_frag_download(ctx)
 
         fragment_retries = self.params.get('fragment_retries', 0)
@@ -140,6 +149,11 @@ def is_ad_fragment_end(s):
                         else compat_urlparse.urljoin(man_url, line))
                     if extra_query:
                         frag_url = update_url_query(frag_url, extra_query)
+
+                    if real_downloader:
+                        fragment_urls.append(frag_url)
+                        continue
+
                     count = 0
                     headers = info_dict.get('http_headers', {})
                     if byte_range:
@@ -168,6 +182,7 @@ def is_ad_fragment_end(s):
                         self.report_error(
                             'giving up after %s fragment retries' % fragment_retries)
                         return False
+
                     if decrypt_info['METHOD'] == 'AES-128':
                         iv = decrypt_info.get('IV') or compat_struct_pack('>8xq', media_sequence)
                         decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
@@ -211,6 +226,17 @@ def is_ad_fragment_end(s):
                 elif is_ad_fragment_end(line):
                     ad_frag_next = False
 
+        if real_downloader:
+            info_copy = info_dict.copy()
+            info_copy['url_list'] = fragment_urls
+            info_copy['decrypt_info'] = decrypt_info
+            fd = real_downloader(self.ydl, self.params)
+            # TODO: Make progress updates work without hooking twice
+            # for ph in self._progress_hooks:
+            #     fd.add_progress_hook(ph)
+            success = fd.real_download(filename, info_copy)
+            if not success:
+                return False
+        else:
             self._finish_frag_download(ctx)
-
         return True

youtube_dlc/downloader/youtube_live_chat.py

@@ -50,7 +50,16 @@ def download_and_parse_fragment(url, frag_index):
                     success, raw_fragment = dl_fragment(url)
                     if not success:
                         return False, None, None
-                    data = parse_yt_initial_data(raw_fragment) or json.loads(raw_fragment)['response']
+                    data = parse_yt_initial_data(raw_fragment)
+                    if not data:
+                        raw_data = json.loads(raw_fragment)
+                        # sometimes youtube replies with a list
+                        if not isinstance(raw_data, list):
+                            raw_data = [raw_data]
+                        try:
+                            data = next(item['response'] for item in raw_data if 'response' in item)
+                        except StopIteration:
+                            data = {}
 
                     live_chat_continuation = try_get(
                         data,

youtube_dlc/extractor/common.py

@@ -975,7 +975,7 @@ def playlist_result(entries, playlist_id=None, playlist_title=None, playlist_des
             video_info['id'] = playlist_id
         if playlist_title:
             video_info['title'] = playlist_title
-        if playlist_description:
+        if playlist_description is not None:
             video_info['description'] = playlist_description
         return video_info
 

youtube_dlc/extractor/youtube.py

@@ -1999,8 +1999,10 @@ def get_continuation(continuation, session_token, replies=False):
                     raise ExtractorError('Unexpected HTTP error code: %s' % response_code)
 
             first_continuation = True
+            chain_msg = ''
+            self.to_screen('Downloading comments')
             while continuations:
-                continuation, itct = continuations.pop()
+                continuation = continuations.pop()
                 comment_response = get_continuation(continuation, xsrf_token)
                 if not comment_response:
                     continue
@@ -2046,9 +2048,10 @@ def get_continuation(continuation, session_token, replies=False):
                             continue
 
                         if self._downloader.params.get('verbose', False):
-                            self.to_screen('[debug] Comments downloaded (chain %s) %s of ~%s' % (comment['commentId'], len(video_comments), expected_video_comment_count))
+                            chain_msg = ' (chain %s)' % comment['commentId']
+                        self.to_screen('Comments downloaded: %d of ~%d%s' % (len(video_comments), expected_video_comment_count, chain_msg))
                         reply_comment_meta = replies_data[1]['response']['continuationContents']['commentRepliesContinuation']
-                        for reply_meta in replies_data[1]['response']['continuationContents']['commentRepliesContinuation']['contents']:
+                        for reply_meta in reply_comment_meta.get('contents', {}):
                             reply_comment = reply_meta['commentRenderer']
                             video_comments.append({
                                 'id': reply_comment['commentId'],
@@ -2063,12 +2066,12 @@ def get_continuation(continuation, session_token, replies=False):
                             continue
                         reply_continuations += [rcn['nextContinuationData']['continuation'] for rcn in reply_comment_meta['continuations']]
 
-                self.to_screen('Comments downloaded %s of ~%s' % (len(video_comments), expected_video_comment_count))
+                self.to_screen('Comments downloaded: %d of ~%d' % (len(video_comments), expected_video_comment_count))
                 if 'continuations' in item_section:
                     continuations += [ncd['nextContinuationData']['continuation'] for ncd in item_section['continuations']]
                 time.sleep(1)
 
-            self.to_screen('Total comments downloaded %s of ~%s' % (len(video_comments), expected_video_comment_count))
+            self.to_screen('Total comments downloaded: %d of ~%d' % (len(video_comments), expected_video_comment_count))
             info.update({
                 'comments': video_comments,
                 'comment_count': expected_video_comment_count
@@ -2108,6 +2111,8 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             'id': 'UCqj7Cz7revf5maW9g5pgNcg',
             'title': 'Игорь Клейнер - Playlists',
             'description': 'md5:be97ee0f14ee314f1f002cf187166ee2',
+            'uploader': 'Игорь Клейнер',
+            'uploader_id': 'UCqj7Cz7revf5maW9g5pgNcg',
         },
     }, {
         # playlists, multipage, different order
@@ -2117,6 +2122,8 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             'id': 'UCqj7Cz7revf5maW9g5pgNcg',
             'title': 'Игорь Клейнер - Playlists',
             'description': 'md5:be97ee0f14ee314f1f002cf187166ee2',
+            'uploader_id': 'UCqj7Cz7revf5maW9g5pgNcg',
+            'uploader': 'Игорь Клейнер',
         },
     }, {
         # playlists, singlepage
@@ -2126,6 +2133,8 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             'id': 'UCAEtajcuhQ6an9WEzY9LEMQ',
             'title': 'ThirstForScience - Playlists',
             'description': 'md5:609399d937ea957b0f53cbffb747a14c',
+            'uploader': 'ThirstForScience',
+            'uploader_id': 'UCAEtajcuhQ6an9WEzY9LEMQ',
         }
     }, {
         'url': 'https://www.youtube.com/c/ChristophLaimer/playlists',
@@ -2157,6 +2166,8 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             'id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
             'title': 'lex will - Home',
             'description': 'md5:2163c5d0ff54ed5f598d6a7e6211e488',
+            'uploader': 'lex will',
+            'uploader_id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
         },
         'playlist_mincount': 2,
     }, {
@@ -2166,6 +2177,8 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             'id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
             'title': 'lex will - Videos',
             'description': 'md5:2163c5d0ff54ed5f598d6a7e6211e488',
+            'uploader': 'lex will',
+            'uploader_id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
         },
         'playlist_mincount': 975,
     }, {
@@ -2175,6 +2188,8 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             'id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
             'title': 'lex will - Videos',
             'description': 'md5:2163c5d0ff54ed5f598d6a7e6211e488',
+            'uploader': 'lex will',
+            'uploader_id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
         },
         'playlist_mincount': 199,
     }, {
@@ -2184,6 +2199,8 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             'id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
             'title': 'lex will - Playlists',
             'description': 'md5:2163c5d0ff54ed5f598d6a7e6211e488',
+            'uploader': 'lex will',
+            'uploader_id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
         },
         'playlist_mincount': 17,
     }, {
@@ -2193,6 +2210,8 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             'id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
             'title': 'lex will - Community',
             'description': 'md5:2163c5d0ff54ed5f598d6a7e6211e488',
+            'uploader': 'lex will',
+            'uploader_id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
         },
         'playlist_mincount': 18,
     }, {
@@ -2202,8 +2221,10 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             'id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
             'title': 'lex will - Channels',
             'description': 'md5:2163c5d0ff54ed5f598d6a7e6211e488',
+            'uploader': 'lex will',
+            'uploader_id': 'UCKfVa3S1e4PHvxWcwyMMg8w',
         },
-        'playlist_mincount': 138,
+        'playlist_mincount': 12,
     }, {
         'url': 'https://invidio.us/channel/UCmlqkdCBesrv2Lak1mF_MxA',
         'only_matching': True,
@@ -2221,6 +2242,7 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             'id': 'PLwP_SiAcdui0KVebT0mU9Apz359a4ubsC',
             'uploader': 'Christiaan008',
             'uploader_id': 'UCEPzS1rYsrkqzSLNp76nrcg',
+            'description': 'md5:a14dc1a8ef8307a9807fe136a0660268',
         },
         'playlist_count': 96,
     }, {
@@ -2255,6 +2277,7 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             'id': 'PLzH6n4zXuckpfMu_4Ff8E7Z1behQks5ba',
             'uploader_id': 'UC9-y-6csu5WGm29I7JiwpnA',
             'uploader': 'Computerphile',
+            'description': 'md5:7f567c574d13d3f8c0954d9ffee4e487',
         },
         'playlist_mincount': 11,
     }, {
@@ -2295,12 +2318,12 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
         'info_dict': {
             'id': '9Auq9mYxFEE',
             'ext': 'mp4',
-            'title': 'Watch Sky News live',
+            'title': compat_str,
             'uploader': 'Sky News',
             'uploader_id': 'skynews',
             'uploader_url': r're:https?://(?:www\.)?youtube\.com/user/skynews',
             'upload_date': '20191102',
-            'description': 'md5:78de4e1c2359d0ea3ed829678e38b662',
+            'description': 'md5:85ddd75d888674631aaf9599a9a0b0ae',
             'categories': ['News & Politics'],
             'tags': list,
             'like_count': int,
@@ -2574,7 +2597,7 @@ def _extract_continuation(cls, renderer):
         next_continuation = cls._extract_next_continuation_data(renderer)
         if next_continuation:
             return next_continuation
-        contents = renderer.get('contents')
+        contents = renderer.get('contents') or renderer.get('items')
         if not isinstance(contents, list):
             return
         for content in contents:
@@ -2611,35 +2634,22 @@ def extract_entries(parent_renderer):  # this needs to called again for continua
                 for isr_content in isr_contents:
                     if not isinstance(isr_content, dict):
                         continue
-                    renderer = isr_content.get('playlistVideoListRenderer')
+
-                    if renderer:
+                    known_renderers = {
-                        for entry in self._playlist_entries(renderer):
+                        'playlistVideoListRenderer': self._playlist_entries,
-                            yield entry
+                        'gridRenderer': self._grid_entries,
-                        continuation_list[0] = self._extract_continuation(renderer)
+                        'shelfRenderer': lambda x: self._shelf_entries(x, tab.get('title') != 'Channels'),
+                        'backstagePostThreadRenderer': self._post_thread_entries,
+                        'videoRenderer': lambda x: [self._video_entry(x)],
+                    }
+                    for key, renderer in isr_content.items():
+                        if key not in known_renderers:
                             continue
-                    renderer = isr_content.get('gridRenderer')
+                        for entry in known_renderers[key](renderer):
-                    if renderer:
-                        for entry in self._grid_entries(renderer):
-                            yield entry
-                        continuation_list[0] = self._extract_continuation(renderer)
-                        continue
-                    renderer = isr_content.get('shelfRenderer')
-                    if renderer:
-                        is_channels_tab = tab.get('title') == 'Channels'
-                        for entry in self._shelf_entries(renderer, not is_channels_tab):
-                            yield entry
-                        continue
-                    renderer = isr_content.get('backstagePostThreadRenderer')
-                    if renderer:
-                        for entry in self._post_thread_entries(renderer):
-                            yield entry
-                        continuation_list[0] = self._extract_continuation(renderer)
-                        continue
-                    renderer = isr_content.get('videoRenderer')
-                    if renderer:
-                        entry = self._video_entry(renderer)
                             if entry:
                                 yield entry
+                        continuation_list[0] = self._extract_continuation(renderer)
+                        break
 
                 if not continuation_list[0]:
                     continuation_list[0] = self._extract_continuation(is_renderer)
@@ -2692,47 +2702,46 @@ def extract_entries(parent_renderer):  # this needs to called again for continua
             if not response:
                 break
 
+            known_continuation_renderers = {
+                'playlistVideoListContinuation': self._playlist_entries,
+                'gridContinuation': self._grid_entries,
+                'itemSectionContinuation': self._post_thread_continuation_entries,
+                'sectionListContinuation': extract_entries,  # for feeds
+            }
             continuation_contents = try_get(
-                response, lambda x: x['continuationContents'], dict)
+                response, lambda x: x['continuationContents'], dict) or {}
-            if continuation_contents:
+            continuation_renderer = None
-                continuation_renderer = continuation_contents.get('playlistVideoListContinuation')
+            for key, value in continuation_contents.items():
-                if continuation_renderer:
+                if key not in known_continuation_renderers:
-                    for entry in self._playlist_entries(continuation_renderer):
-                        yield entry
-                    continuation = self._extract_continuation(continuation_renderer)
                     continue
-                continuation_renderer = continuation_contents.get('gridContinuation')
+                continuation_renderer = value
-                if continuation_renderer:
-                    for entry in self._grid_entries(continuation_renderer):
-                        yield entry
-                    continuation = self._extract_continuation(continuation_renderer)
-                    continue
-                continuation_renderer = continuation_contents.get('itemSectionContinuation')
-                if continuation_renderer:
-                    for entry in self._post_thread_continuation_entries(continuation_renderer):
-                        yield entry
-                    continuation = self._extract_continuation(continuation_renderer)
-                    continue
-                continuation_renderer = continuation_contents.get('sectionListContinuation')  # for feeds
-                if continuation_renderer:
                 continuation_list = [None]
-                    for entry in extract_entries(continuation_renderer):
+                for entry in known_continuation_renderers[key](continuation_renderer):
                     yield entry
-                    continuation = continuation_list[0]
+                continuation = continuation_list[0] or self._extract_continuation(continuation_renderer)
+                break
+            if continuation_renderer:
                 continue
 
+            known_renderers = {
+                'gridPlaylistRenderer': (self._grid_entries, 'items'),
+                'gridVideoRenderer': (self._grid_entries, 'items'),
+                'playlistVideoRenderer': (self._playlist_entries, 'contents'),
+                'itemSectionRenderer': (self._playlist_entries, 'contents'),
+            }
             continuation_items = try_get(
                 response, lambda x: x['onResponseReceivedActions'][0]['appendContinuationItemsAction']['continuationItems'], list)
-            if continuation_items:
+            continuation_item = try_get(continuation_items, lambda x: x[0], dict) or {}
-                continuation_item = continuation_items[0]
+            video_items_renderer = None
-                if not isinstance(continuation_item, dict):
+            for key, value in continuation_item.items():
+                if key not in known_renderers:
                     continue
-                renderer = continuation_item.get('playlistVideoRenderer') or continuation_item.get('itemSectionRenderer')
+                video_items_renderer = {known_renderers[key][1]: continuation_items}
-                if renderer:
+                for entry in known_renderers[key][0](video_items_renderer):
-                    video_list_renderer = {'contents': continuation_items}
-                    for entry in self._playlist_entries(video_list_renderer):
                     yield entry
-                    continuation = self._extract_continuation(video_list_renderer)
+                continuation = self._extract_continuation(video_items_renderer)
+                break
+            if video_items_renderer:
                 continue
             break
 
@@ -2784,12 +2793,15 @@ def _extract_from_tabs(self, item_id, webpage, data, tabs, identity_token):
                 data, lambda x: x['metadata']['playlistMetadataRenderer'], dict)
         if renderer:
             title = renderer.get('title')
-            description = renderer.get('description')
+            description = renderer.get('description', '')
             playlist_id = channel_id
             tags = renderer.get('keywords', '').split()
             thumbnails_list = (
                 try_get(renderer, lambda x: x['avatar']['thumbnails'], list)
-                or data['sidebar']['playlistSidebarRenderer']['items'][0]['playlistSidebarPrimaryInfoRenderer']['thumbnailRenderer']['playlistVideoThumbnailRenderer']['thumbnail']['thumbnails']
+                or try_get(
+                    data,
+                    lambda x: x['sidebar']['playlistSidebarRenderer']['items'][0]['playlistSidebarPrimaryInfoRenderer']['thumbnailRenderer']['playlistVideoThumbnailRenderer']['thumbnail']['thumbnails'],
+                    list)
                 or [])
 
         thumbnails = []
@@ -3089,7 +3101,7 @@ def _real_extract(self, url):
 
 
 class YoutubeSearchIE(SearchInfoExtractor, YoutubeBaseInfoExtractor):
-    IE_DESC = 'YouTube.com searches'
+    IE_DESC = 'YouTube.com searches, "ytsearch" keyword'
     # there doesn't appear to be a real limit, for example if you search for
     # 'python' you get more than 8.000.000 results
     _MAX_RESULTS = float('inf')
@@ -3178,7 +3190,7 @@ class YoutubeSearchDateIE(YoutubeSearchIE):
 
 
 class YoutubeSearchURLIE(YoutubeSearchIE):
-    IE_DESC = 'YouTube.com searches, "ytsearch" keyword'
+    IE_DESC = 'YouTube.com search URLs'
     IE_NAME = YoutubeSearchIE.IE_NAME + '_url'
     _VALID_URL = r'https?://(?:www\.)?youtube\.com/results\?(.*?&)?(?:search_query|q)=(?:[^&]+)(?:[&]|$)'
     # _MAX_RESULTS = 100

youtube_dlc/options.py

@@ -18,14 +18,12 @@
     get_executable_path,
     OUTTMPL_TYPES,
     preferredencoding,
+    REMUX_EXTENSIONS,
     write_string,
 )
 from .version import __version__
 
 
-_remux_formats = ('mp4', 'mkv', 'flv', 'webm', 'mov', 'avi', 'mp3', 'mka', 'm4a', 'ogg', 'opus')
-
-
 def _hide_login_info(opts):
     PRIVATE_OPTS = set(['-p', '--password', '-u', '--username', '--video-password', '--ap-password', '--ap-username'])
     eqre = re.compile('^(?P<key>' + ('|'.join(re.escape(po) for po in PRIVATE_OPTS)) + ')=.+$')
@@ -1042,7 +1040,7 @@ def _dict_from_multiple_values_options_callback(
             'Remux the video into another container if necessary (currently supported: %s). '
             'If target container does not support the video/audio codec, remuxing will fail. '
             'You can specify multiple rules; eg. "aac>m4a/mov>mp4/mkv" will remux aac to m4a, mov to mp4 '
-            'and anything else to mkv.' % '|'.join(_remux_formats)))
+            'and anything else to mkv.' % '|'.join(REMUX_EXTENSIONS)))
     postproc.add_option(
         '--recode-video',
         metavar='FORMAT', dest='recodevideo', default=None,

youtube_dlc/postprocessor/embedthumbnail.py

@@ -10,9 +10,9 @@
 
 try:
     import mutagen
-    _has_mutagen = True
+    has_mutagen = True
 except ImportError:
-    _has_mutagen = False
+    has_mutagen = False
 
 from .ffmpeg import FFmpegPostProcessor
 
@@ -42,13 +42,12 @@ def __init__(self, downloader=None, already_have_thumbnail=False):
     def run(self, info):
         filename = info['filepath']
         temp_filename = prepend_extension(filename, 'temp')
-        files_to_delete = []
 
         if not info.get('thumbnails'):
             self.to_screen('There aren\'t any thumbnails to embed')
             return [], info
 
-        thumbnail_filename = info['thumbnails'][-1]['filename']
+        original_thumbnail = thumbnail_filename = info['thumbnails'][-1]['filename']
 
         if not os.path.exists(encodeFilename(thumbnail_filename)):
             self.report_warning('Skipping embedding the thumbnail because the file is missing.')
@@ -67,7 +66,7 @@ def is_webp(path):
                 self.to_screen('Correcting extension to webp and escaping path for thumbnail "%s"' % thumbnail_filename)
                 thumbnail_webp_filename = replace_extension(thumbnail_filename, 'webp')
                 os.rename(encodeFilename(thumbnail_filename), encodeFilename(thumbnail_webp_filename))
-                thumbnail_filename = thumbnail_webp_filename
+                original_thumbnail = thumbnail_filename = thumbnail_webp_filename
                 thumbnail_ext = 'webp'
 
         # Convert unsupported thumbnail formats to JPEG (see #25687, #25717)
@@ -79,9 +78,9 @@ def is_webp(path):
             escaped_thumbnail_jpg_filename = replace_extension(escaped_thumbnail_filename, 'jpg')
             self.to_screen('Converting thumbnail "%s" to JPEG' % escaped_thumbnail_filename)
             self.run_ffmpeg(escaped_thumbnail_filename, escaped_thumbnail_jpg_filename, ['-bsf:v', 'mjpeg2jpeg'])
-            files_to_delete.append(escaped_thumbnail_filename)
             thumbnail_jpg_filename = replace_extension(thumbnail_filename, 'jpg')
             # Rename back to unescaped for further processing
+            os.rename(encodeFilename(escaped_thumbnail_filename), encodeFilename(thumbnail_filename))
             os.rename(encodeFilename(escaped_thumbnail_jpg_filename), encodeFilename(thumbnail_jpg_filename))
             thumbnail_filename = thumbnail_jpg_filename
             thumbnail_ext = 'jpg'
@@ -153,7 +152,7 @@ def is_webp(path):
                     success = False
 
         elif info['ext'] in ['ogg', 'opus']:
-            if not _has_mutagen:
+            if not has_mutagen:
                 raise EmbedThumbnailPPError('module mutagen was not found. Please install using `python -m pip install mutagen`')
             self.to_screen('Adding thumbnail to "%s"' % filename)
 
@@ -184,9 +183,11 @@ def is_webp(path):
         if success and temp_filename != filename:
             os.remove(encodeFilename(filename))
             os.rename(encodeFilename(temp_filename), encodeFilename(filename))
+
+        files_to_delete = [thumbnail_filename]
         if self._already_have_thumbnail:
-            info['__files_to_move'][thumbnail_filename] = replace_extension(
+            info['__files_to_move'][original_thumbnail] = replace_extension(
-                info['__thumbnail_filename'], os.path.splitext(thumbnail_filename)[1][1:])
+                info['__thumbnail_filename'], os.path.splitext(original_thumbnail)[1][1:])
-        else:
+            if original_thumbnail == thumbnail_filename:
-            files_to_delete.append(thumbnail_filename)
+                files_to_delete = []
         return files_to_delete, info

youtube_dlc/postprocessor/ffmpeg.py

@@ -442,6 +442,10 @@ def run(self, information):
 
 
 class FFmpegEmbedSubtitlePP(FFmpegPostProcessor):
+    def __init__(self, downloader=None, already_have_subtitle=False):
+        super(FFmpegEmbedSubtitlePP, self).__init__(downloader)
+        self._already_have_subtitle = already_have_subtitle
+
     def run(self, information):
         if information['ext'] not in ('mp4', 'webm', 'mkv'):
             self.to_screen('Subtitles can only be embedded in mp4, webm or mkv files')
@@ -501,7 +505,8 @@ def run(self, information):
         os.remove(encodeFilename(filename))
         os.rename(encodeFilename(temp_filename), encodeFilename(filename))
 
-        return sub_filenames, information
+        files_to_delete = [] if self._already_have_subtitle else sub_filenames
+        return files_to_delete, information
 
 
 class FFmpegMetadataPP(FFmpegPostProcessor):

youtube_dlc/postprocessor/movefilesafterdownload.py

@@ -4,11 +4,11 @@
 
 from .common import PostProcessor
 from ..utils import (
+    decodeFilename,
     encodeFilename,
     make_dir,
     PostProcessingError,
 )
-from ..compat import compat_str
 
 
 class MoveFilesAfterDownloadPP(PostProcessor):
@@ -26,12 +26,12 @@ def run(self, info):
         finaldir = info.get('__finaldir', dl_path)
         finalpath = os.path.join(finaldir, dl_name)
         self.files_to_move.update(info['__files_to_move'])
-        self.files_to_move[info['filepath']] = finalpath
+        self.files_to_move[info['filepath']] = decodeFilename(finalpath)
 
+        make_newfilename = lambda old: decodeFilename(os.path.join(finaldir, os.path.basename(encodeFilename(old))))
         for oldfile, newfile in self.files_to_move.items():
             if not newfile:
-                newfile = os.path.join(finaldir, os.path.basename(encodeFilename(oldfile)))
+                newfile = make_newfilename(oldfile)
-            oldfile, newfile = compat_str(oldfile), compat_str(newfile)
             if os.path.abspath(encodeFilename(oldfile)) == os.path.abspath(encodeFilename(newfile)):
                 continue
             if not os.path.exists(encodeFilename(oldfile)):
@@ -50,5 +50,5 @@ def run(self, info):
             self.to_screen('Moving file "%s" to "%s"' % (oldfile, newfile))
             shutil.move(oldfile, newfile)  # os.rename cannot move between volumes
 
-        info['filepath'] = compat_str(finalpath)
+        info['filepath'] = finalpath
         return [], info

youtube_dlc/postprocessor/sponskrub.py

@@ -43,6 +43,10 @@ def run(self, information):
         if self.path is None:
             return [], information
 
+        filename = information['filepath']
+        if not os.path.exists(encodeFilename(filename)):  # no download
+            return [], information
+
         if information['extractor_key'].lower() != 'youtube':
             self.to_screen('Skipping sponskrub since it is not a YouTube video')
             return [], information
@@ -58,7 +62,6 @@ def run(self, information):
             if not information.get('__real_download', False):
                 self.report_warning('If sponskrub is run multiple times, unintended parts of the video could be cut out.')
 
-        filename = information['filepath']
         temp_filename = prepend_extension(filename, self._temp_ext)
         if os.path.exists(encodeFilename(temp_filename)):
             os.remove(encodeFilename(temp_filename))

youtube_dlc/update.py

@@ -15,6 +15,7 @@
 from .version import __version__
 
 
+'''  # Not signed
 def rsa_verify(message, signature, key):
     from hashlib import sha256
     assert isinstance(message, bytes)
@@ -27,17 +28,13 @@ def rsa_verify(message, signature, key):
         return False
     expected = b'0001' + (byte_size - len(asn1) // 2 - 3) * b'ff' + b'00' + asn1
     return expected == signature
+'''
 
 
 def update_self(to_screen, verbose, opener):
     """Update the program file with the latest version from the repository"""
 
-    return to_screen('Update is currently broken.\nVisit  https://github.com/pukkandan/yt-dlp/releases/latest  to get the latest version')
+    JSON_URL = 'https://api.github.com/repos/pukkandan/yt-dlp/releases/latest'
-
-    UPDATE_URL = 'https://blackjack4494.github.io//update/'
-    VERSION_URL = UPDATE_URL + 'LATEST_VERSION'
-    JSON_URL = UPDATE_URL + 'versions.json'
-    UPDATES_RSA_KEY = (0x9d60ee4d8f805312fdb15a62f87b95bd66177b91df176765d13514a0f1754bcd2057295c5b6f1d35daa6742c3ffc9a82d3e118861c207995a8031e151d863c9927e304576bc80692bc8e094896fcf11b66f3e29e04e3a71e9a11558558acea1840aec37fc396fb6b65dc81a1c4144e03bd1c011de62e3f1357b327d08426fe93, 65537)
 
     def sha256sum():
         h = hashlib.sha256()
@@ -54,55 +51,36 @@ def sha256sum():
         to_screen('It looks like you installed youtube-dlc with a package manager, pip, setup.py or a tarball. Please use that to update.')
         return
 
-    # compiled file.exe can find itself by
-    # to_screen(os.path.basename(sys.executable))
-    # and path to py or exe
-    # to_screen(os.path.realpath(sys.executable))
-
-    # Check if there is a new version
-    try:
-        newversion = opener.open(VERSION_URL).read().decode('utf-8').strip()
-    except Exception:
-        if verbose:
-            to_screen(encode_compat_str(traceback.format_exc()))
-        to_screen('ERROR: can\'t find the current version. Please try again later.')
-        to_screen('Visit https://github.com/blackjack4494/yt-dlc/releases/latest')
-        return
-    if newversion == __version__:
-        to_screen('youtube-dlc is up-to-date (' + __version__ + ')')
-        return
-
     # Download and check versions info
     try:
-        versions_info = opener.open(JSON_URL).read().decode('utf-8')
+        version_info = opener.open(JSON_URL).read().decode('utf-8')
-        versions_info = json.loads(versions_info)
+        version_info = json.loads(version_info)
     except Exception:
         if verbose:
             to_screen(encode_compat_str(traceback.format_exc()))
         to_screen('ERROR: can\'t obtain versions info. Please try again later.')
-        to_screen('Visit https://github.com/blackjack4494/yt-dlc/releases/latest')
+        to_screen('Visit https://github.com/pukkandan/yt-dlp/releases/lastest')
-        return
-    if 'signature' not in versions_info:
-        to_screen('ERROR: the versions file is not signed or corrupted. Aborting.')
-        return
-    signature = versions_info['signature']
-    del versions_info['signature']
-    if not rsa_verify(json.dumps(versions_info, sort_keys=True).encode('utf-8'), signature, UPDATES_RSA_KEY):
-        to_screen('ERROR: the versions file signature is invalid. Aborting.')
         return
 
-    version_id = versions_info['latest']
+    version_id = version_info['tag_name']
+    if version_id == __version__:
+        to_screen('youtube-dlc is up-to-date (' + __version__ + ')')
+        return
 
     def version_tuple(version_str):
         return tuple(map(int, version_str.split('.')))
+
     if version_tuple(__version__) >= version_tuple(version_id):
         to_screen('youtube-dlc is up to date (%s)' % __version__)
         return
 
     to_screen('Updating to version ' + version_id + ' ...')
-    version = versions_info['versions'][version_id]
 
-    print_notes(to_screen, versions_info['versions'])
+    version = {
+        'bin': next(i for i in version_info['assets'] if i['name'] == 'youtube-dlc'),
+        'exe': next(i for i in version_info['assets'] if i['name'] == 'youtube-dlc.exe'),
+        'exe_x86': next(i for i in version_info['assets'] if i['name'] == 'youtube-dlc_x86.exe'),
+    }
 
     # sys.executable is set to the full pathname of the exe-file for py2exe
     # though symlinks are not followed so that we need to do this manually
@@ -113,7 +91,7 @@ def version_tuple(version_str):
         to_screen('ERROR: no write permissions on %s' % filename)
         return
 
-    # Py2EXE
+    # PyInstaller
     if hasattr(sys, 'frozen'):
         exe = filename
         directory = os.path.dirname(exe)
@@ -122,19 +100,14 @@ def version_tuple(version_str):
             return
 
         try:
-            urlh = opener.open(version['exe'][0])
+            urlh = opener.open(version['exe']['browser_download_url'])
             newcontent = urlh.read()
             urlh.close()
         except (IOError, OSError):
             if verbose:
                 to_screen(encode_compat_str(traceback.format_exc()))
             to_screen('ERROR: unable to download latest version')
-            to_screen('Visit https://github.com/blackjack4494/yt-dlc/releases/latest')
+            to_screen('Visit https://github.com/pukkandan/yt-dlp/releases/lastest')
-            return
-
-        newcontent_hash = hashlib.sha256(newcontent).hexdigest()
-        if newcontent_hash != version['exe'][1]:
-            to_screen('ERROR: the downloaded file hash does not match. Aborting.')
             return
 
         try:
@@ -147,16 +120,17 @@ def version_tuple(version_str):
             return
 
         try:
-            bat = os.path.join(directory, 'youtube-dlc-updater.bat')
+            bat = os.path.join(directory, 'yt-dlp-updater.cmd')
             with io.open(bat, 'w') as batfile:
                 batfile.write('''
-@echo off
+@(
-echo Waiting for file handle to be closed ...
+    echo.Waiting for file handle to be closed ...
-ping 127.0.0.1 -n 5 -w 1000 > NUL
+    ping 127.0.0.1 -n 5 -w 1000 > NUL
-move /Y "%s.new" "%s" > NUL
+    move /Y "%s.new" "%s" > NUL
-echo Updated youtube-dlc to version %s.
+    echo.Updated youtube-dlc to version %s.
-start /b "" cmd /c del "%%~f0"&exit /b"
+)
-                \n''' % (exe, exe, version_id))
+@start /b "" cmd /c del "%%~f0"&exit /b
+                ''' % (exe, exe, version_id))
 
             subprocess.Popen([bat])  # Continues to run in the background
             return  # Do not show premature success messages
@@ -169,19 +143,14 @@ def version_tuple(version_str):
     # Zip unix package
     elif isinstance(globals().get('__loader__'), zipimporter):
         try:
-            urlh = opener.open(version['bin'][0])
+            urlh = opener.open(version['bin']['browser_download_url'])
             newcontent = urlh.read()
             urlh.close()
         except (IOError, OSError):
             if verbose:
                 to_screen(encode_compat_str(traceback.format_exc()))
             to_screen('ERROR: unable to download latest version')
-            to_screen('Visit https://github.com/blackjack4494/yt-dlc/releases/latest')
+            to_screen('Visit https://github.com/pukkandan/yt-dlp/releases/lastest')
-            return
-
-        newcontent_hash = hashlib.sha256(newcontent).hexdigest()
-        if newcontent_hash != version['bin'][1]:
-            to_screen('ERROR: the downloaded file hash does not match. Aborting.')
             return
 
         try:
@@ -196,6 +165,7 @@ def version_tuple(version_str):
     to_screen('Updated youtube-dlc. Restart youtube-dlc to use the new version.')
 
 
+'''  # UNUSED
 def get_notes(versions, fromVersion):
     notes = []
     for v, vdata in sorted(versions.items()):
@@ -210,3 +180,4 @@ def print_notes(to_screen, versions, fromVersion=__version__):
         to_screen('PLEASE NOTE:')
         for note in notes:
             to_screen(note)
+'''

youtube_dlc/utils.py

@@ -1715,6 +1715,8 @@ def random_user_agent():
     'wav',
     'f4f', 'f4m', 'm3u8', 'smil')
 
+REMUX_EXTENSIONS = ('mp4', 'mkv', 'flv', 'webm', 'mov', 'avi', 'mp3', 'mka', 'm4a', 'ogg', 'opus')
+
 # needed for sanitizing filenames in restricted mode
 ACCENT_CHARS = dict(zip('ÂÃÄÀÁÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖŐØŒÙÚÛÜŰÝÞßàáâãäåæçèéêëìíîïðñòóôõöőøœùúûüűýþÿ',
                         itertools.chain('AAAAAA', ['AE'], 'CEEEEIIIIDNOOOOOOO', ['OE'], 'UUUUUY', ['TH', 'ss'],
@@ -4690,9 +4692,7 @@ def cli_configuration_args(params, arg_name, key, default=[], exe=None):  # retu
         return default, False
     assert isinstance(argdict, dict)
 
-    assert isinstance(key, compat_str)
     key = key.lower()
-
     args = exe_args = None
     if exe is not None:
         assert isinstance(exe, compat_str)

youtube_dlc/version.py

@@ -1,3 +1,3 @@
 from __future__ import unicode_literals
 
-__version__ = '2021.01.29'
+__version__ = '2021.02.04'