Error for tt_search() with hidden API

[jobreu]

Hi Johannes, first of all thx for the great package!

I just wanted to try out tt_search() for the hidden API. However, I get an error message, no matter what search terms I try.

Error in `resp_body_raw()`:
! Can not retrieve empty body

Reprex:

library(traktok)

test_df <- tt_search("europawahl", max_pages = 2)

[JBGruber]

Cheers! I can't reproduce this. did you authenticate with your cookies? You can check which cookies are saved with cookiemonster::get_cookies("tiktok"). I should be around 30 rows.

But I'm fishing in the dark here. Even if I delete everything, I don't get that error. Maybe try to also update httr2 and curl, just for good measure.

[jobreu]

Thx for the quick reply! cookiemonster::get_cookies("tiktok") gives me 13 rows of data. I also updated httr2 and curl but still get the same error.

[jobreu]

Problem solved! It seems that I was not logged into my account when I exported the cookies.txt before. It worked when I exported the cookie file again after logging in.

[JBGruber]

Excellent! Was this user error or should I make it clearer somewhere that one needs to be logged in before exporting the cookies?

[jobreu]

It was a user error. Still, if you find the time, maybe in the documentation for the unofficial API under Authentication you could possibly change the wording to be more explicit about being logged in. Maybe a simple fix could be to move the statement "after logging in at TikTok.com at least once" from the parentheses to the beginning of that section? E.g.: "After logging in at TikTok.com at least once, you need to export the cookies needed for authentication from your browser. The easiest way to do so..."

[jobreu]

When I tried using the hidden API again today, I received the same error I received before (see 1st post in this issue), even though I refreshed the cookies, the exported .txt file had > 30 entries/rows, and I was logged in during the cookie export and while sending the request. I also deleted the cookies from my browser and exported them again, which, unfortunately, did not solve the issue.

[jobreu]

I also tried gathering data with pyktok and the user metadata request did not work. Maybe this issue is also related to this? dfreelon/pyktok#41

[JBGruber]

I'm afraid this time it's not a user error. The endpoint seems to have changed again and the API now return 200 OK, but with an empty body, no matter what I try. I will have another look soon, but the function does not seem usable at the moment.

[jobreu]

That's what I feared based on the pyktok issue. Thanks for looking into this!

[JBGruber]

FYI, this happened before and then the endpoint suddenly started working again (at least the search). I might at some point also figure out how to properly sign requests, which would make this easier as well.

[michaelgoodier]

Also having this issue - was planning on analysing search results about the UK general election and unfortunately tiktok seems to have changed the endpoint almost exactly after it was announced.

Looking at the successful request to the API in-browser it has a couple of extra params that this package doesn't seem to add (possibly the most notable are web_search_code, msToken, X-Bogus, and _signature) but how these are generated is beyond my level of skill. Do you have any idea if/when you'll get a chance to take a look at this?

Thanks for all your work on this package - for now the individual video metadata still works which will still be useful for another idea we had!

[JBGruber]

I try to use as few of these headers as possible, so I usally include everything the browser does and then reduce the headers until I find the required ones. The ones that break the function are X-Bogus (and maybe _signature), which were not required. I don't really see a path to resolve that, unless someone figures out how to produce the X-Bogus values.

[JBGruber]

Just a small update of what isn't working:

I still don't know how one would sign a request with that X-Bogus parameter. Most others seem to be using browser automation, which I've now also done with e.g., tt_user_info_hidden and tt_user_videos_hidden. But for the search it's not working properly. What works is to get the first six videos:

sess <- rvest::read_html_live("https://www.tiktok.com/search?q=rstats&t=1730277472984")
sess |> 
  rvest::html_elements("a") |> 
  rvest::html_attr("href") |> 
  stringr::str_subset("/video/")
#> [1] "https://www.tiktok.com/@smooth.learning.c/video/7351237579729145096"
#> [2] "https://www.tiktok.com/@programandoenr/video/7141887653603020037"   
#> [3] "https://www.tiktok.com/@statisticsglobe/video/7420679794028776736"  
#> [4] "https://www.tiktok.com/@statisticsglobe/video/7428218915173535008"  
#> [5] "https://www.tiktok.com/@rajistics/video/7074268172777573675"        
#> [6] "https://www.tiktok.com/@statisticsglobe/video/7421368928707923233"

^{Created on 2024-11-04 with reprex v2.1.1}

But scrolling down, like in tt_user_videos_hidden, is only possible after logging in, which is not working with the chromote package that powers rvest::read_html_live.

[deboutlesdamnes]

Hi Johannes,

Thanks for the great package! Just wondering what the status was on adapting the package to the API change and what the roadblocks are? Happy to try and help get around it.