[youtube:tab] Fallback to API when webpage fails to download #1122

coletdjnz · 2021-09-29T06:59:57Z

Please follow the guide below

You will be asked some questions, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your pull request (like that [x])
Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
Searched the bugtracker for similar pull requests
Checked the code with flake8

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check one of the following options:

I am the original author of this code and I am willing to release it under Unlicense
I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

Bug fix
Improvement
New extractor
New feature

Description of your pull request and other information

This supersedes #682. (I'm making a new PR to keep things on topic - the other PR is a bit messy due to be experimental).

This PR adds support for falling back to the Innertube API when the webpage fails to download for whatever reason (429, network error, initial data extraction failure, etc.)

It is also possible to manually skip the webpage download by using --extractor-args youtubetab:skip=webpage

Also added a quick fix for the resolve MP to OLAK playlist handling, since that was broken.

At least at the time of writing, yt-dlp should now have a complete* workaround for the common** YouTube HTTP 429 error reported, for both video and tab extraction.
_{*except private content accessible only through a secondary or greater account/channel}
_{**common referring to the webpage request getting 429nd only, but the API still works.}

resolves #926, resolves ytdl-org/youtube-dl#23638 (and most of the other 429 reports upstream)

Known issues:

When using cookies, the webpage is required to extract the required variables. Without this, the API requests will be for the first channel of the first account.
- In the case of the user passing cookies, and the webpage failing to download, the tab extractor will raise an error. This is to stop unwanted behavior in variety of situations (e.g. an automated script to download a secondary channel liked playlist, but the webpage fails and starts downloading the first channel liked playlist).
- Currently, the user can skip this check by passing --extractor-args youtubetab:skip=authcheck
channel search does not work (returns 0 results)
- I think this is a YouTube a bug. YouTube appears to produce params for itself that it cannot understand.
- See [youtube:tab] Fallback to API when webpage fails to download #1122 (comment)

TODO:

Improve the error message when authenticated but webpage fails.
Documentation
Add ~~(and probably fixup)~~ tests
fix https://www.youtube.com/feed/recommended
~~Fix unavailable video messages showing - test: https://www.youtube.com/playlist?list=UUTYLiWFZy8xtPwxFwX9rV7Q~~ for another PR

…n index without ytcfg

… existing

This reverts commit 7a05226

This reverts commit 6126633

# Conflicts: # yt_dlp/extractor/youtube.py

coletdjnz · 2021-09-30T08:24:03Z

Had to fix and improve the visitorData handling since the home/recommended was broken.

Some info on visitorData for the sake of documentation:

When YT gives visitorData in the responseContext of any API request, from what I see, is the most current one (so we need to update our visitorData to it). visitorData can change in some situations, such as ytdl-org/youtube-dl#28702.

If YouTube doesn't provide visitorData in responseContext, visitorData is the same as before (i.e., does not need updating).

We've seen it been used to keep track of what comment section version to use (ensure consistency) when YT was switching from the old to new API a while back. Along with the recommend feed and the playlists in the above issue, maybe this suggests it is some sort of token that identifies a session state?

After this PR I do need to go through YoutubeIE and clean up it's visitorData handling. I notice the comment one no longer works. Even though it may not be required as of now, it is good to have just in case (e.g A/B testing as mentioned before).

Note: probably not a good idea to share visitorData across different clients.

Co-authored-by: pukkandan <[email protected]>

yt_dlp/extractor/youtube.py

…allback

… retrying

yt_dlp/extractor/youtube.py

coletdjnz · 2021-10-04T06:21:11Z

Some things we'll move to a future cleanup PR(s):

generalize extract_response and extract_webpage
fixup extract_response & extract webpage INFO alerts (i.e., unavailable videos):
- show only if incomplete data
- otherwise, show in debug mode
fixup tab extractor tests (needs above)
remove old python2 hacks from tab entries
ensure consistency with visitorData

coletdjnz · 2021-10-04T07:23:12Z

Some changes:

added a few tests for API Fallback mode
raise an error when url is not resolved (no endpoint provided) instead of unexpected tab not found error. This usually happens when, for example, an invalid playlist link is provided. Normally on YouTube it redirects to the home page.

Issue: channel search (e.g. https://www.youtube.com/user/theCodyReeder/search?query=chicken):

resolve_url endpoint resolves this correctly, but the query parameter that would usually be in plaintext json is instead in the protobuf params. YouTube's backend doesn't understand itself in this situation so the API fallback does not work in this case. 😆

yt_dlp/extractor/youtube.py

…1122) and add some extractor_args to force this mode Authored by: coletdjnz

vivandiu · 2023-01-22T02:48:17Z

Ok

coletdjnz added 30 commits August 12, 2021 19:03

api-only channel extraction

db89662

add extractor arg and support mixes

54161df

bug and linter

841fe34

use hacky getAccountSwitcherEndpoint to get account syncid and sessio…

3549b38

…n index without ytcfg

linter

59e5b21

Merge remote-tracking branch 'upstream/master' into api-only-real

6d5344e

failing to extract resolve_url is usually due to playlist/channel not…

6ee625e

… existing

initial implementation of multi-js-player support (not tested)

6126633

Fix player url related logic

7a05226

Revert "Fix player url related logic"

11fde94

This reverts commit 7a05226

Revert "initial implementation of multi-js-player support (not tested)"

9f712a0

This reverts commit 6126633

Use iframe API to get latest player version

16af686

don't unnecessary fetch for player js if the client doesn't require it

534f3fb

remove unused self._player_url

0c2fe14

linter and bug fix

7b6894c

revert accidental change

ba2065b

add player_js=require/no_require to allow pure api-only mode

87019f0

remove redundant check

6fb686f

Add is_authenticated wrapper for sapisidhash function, and a warning.

371ca87

remove warning for now as it may cause issues

f08e1fa

[youtube] remove annotations and xsrf token extraction

3c63303

Merge remote-tracking branch 'upstream/master' into api-only-real

263dd0b

player_skip=player option

aad471c

Merge remote-tracking branch 'upstream/master' into api-only-real

134c781

cleanup

df861a6

Merge remote-tracking branch 'upstream/master' into api-only-real

14ca1e8

# Conflicts: # yt_dlp/extractor/youtube.py

move code to _extract_tab_endpoint

0014400

use is_authenticated property

785b7d8

create _extract_data to handle webpage to api fallbacks

efd5f28

pass ytcfg instead of webpage

37c274c

coletdjnz added 2 commits September 30, 2021 18:15

errors from API pages are expected most of the time

66ab5e6

Fix home page extraction and improve visitorData handling

9541b78

coletdjnz and others added 2 commits September 30, 2021 21:37

visitorData should be extracted from the latest data

76c5d1c

very minor bracket cleanup

09bf547

Co-authored-by: pukkandan <[email protected]>

coletdjnz commented Sep 30, 2021

View reviewed changes

yt_dlp/extractor/youtube.py Outdated Show resolved Hide resolved

pukkandan reviewed Sep 30, 2021

View reviewed changes

yt_dlp/extractor/youtube.py Show resolved Hide resolved

coletdjnz added 7 commits October 2, 2021 13:45

Merge remote-tracking branch 'upstream/master' into tab-api-fallback

6510871

fix extract_from_tabs thumbnails and tags being same list

01c28fa

Lists -> Playlists

cfb5b69

Failing to resolve album to playlist should be fatal

b5c47f3

Merge remote-tracking branch 'origin/tab-api-fallback' into tab-api-f…

2451c74

…allback

visitor data doc improve

d6e21bc

webpage: break like extract_response when receive an error instead of…

465e71a

… retrying

pukkandan reviewed Oct 2, 2021

View reviewed changes

yt_dlp/extractor/youtube.py Show resolved Hide resolved

yt_dlp/extractor/youtube.py Show resolved Hide resolved

Add tests and raise error when no endpoint resolved

1712bf4

coletdjnz added 2 commits October 5, 2021 18:30

Add documentation

f294ced

allow non-fatal exit of extract_tab_endpoint

7e90581

coletdjnz commented Oct 5, 2021

View reviewed changes

yt_dlp/extractor/youtube.py Show resolved Hide resolved

coletdjnz requested a review from pukkandan October 5, 2021 06:01

coletdjnz added pending-review PR needs a review and removed pending-fixes PR has had changes requested labels Oct 5, 2021

pukkandan merged commit ac56cf3 into yt-dlp:master Oct 8, 2021

Nov 22, 2021

[youtube:tab] Fallback to API when webpage fails to download (yt-dlp#…

38fa36f

…1122) and add some extractor_args to force this mode Authored by: coletdjnz

coletdjnz deleted the tab-api-fallback branch March 4, 2022 06:03

pukkandan added site-enhancement Feature request for some website and removed pending-review PR needs a review labels Jun 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[youtube:tab] Fallback to API when webpage fails to download #1122

[youtube:tab] Fallback to API when webpage fails to download #1122

coletdjnz commented Sep 29, 2021 •

edited

Loading

coletdjnz commented Sep 30, 2021 •

edited

Loading

coletdjnz commented Oct 4, 2021 •

edited

Loading

coletdjnz commented Oct 4, 2021 •

edited

Loading

vivandiu commented Jan 22, 2023

[youtube:tab] Fallback to API when webpage fails to download #1122

[youtube:tab] Fallback to API when webpage fails to download #1122

Conversation

coletdjnz commented Sep 29, 2021 • edited Loading

Please follow the guide below

Before submitting a pull request make sure you have:

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check one of the following options:

What is the purpose of your pull request?

Description of your pull request and other information

coletdjnz commented Sep 30, 2021 • edited Loading

Some info on visitorData for the sake of documentation:

coletdjnz commented Oct 4, 2021 • edited Loading

coletdjnz commented Oct 4, 2021 • edited Loading

vivandiu commented Jan 22, 2023

coletdjnz commented Sep 29, 2021 •

edited

Loading

coletdjnz commented Sep 30, 2021 •

edited

Loading

coletdjnz commented Oct 4, 2021 •

edited

Loading

coletdjnz commented Oct 4, 2021 •

edited

Loading