"Forbidden" error when adding or refreshing APKMirror track-only apps

[bossanovaorca]

Prerequisites

Describe the bug

APKMirror applications are track-only on Obtainium, but when refreshing or trying to install new apps on APKMirror, Obtainium returns "Forbidden". It doesn't seem to be related to my network so I'm not sure why Obtainium is having trouble with this site.

To Reproduce

1. Add an APKMirror application to Obtainium, for example https://www.apkmirror.com/apk/x-corp/twitter/ (it will be "track-only")
2. look for any "Forbidden" errors

Please complete the following information:

• Device: Samsung Galaxy S23 Ultra
• OS: Android 14
• Obtainium Version: 1.1.30

I can provide any logs or screenshots if needed

[leo-liar]

Can report the same behaviour since a few days ago (with a different app). Update check suddenly failed with "Forbidden". Removed the app from Obtainium but could not add it again because this action is also "Forbidden".

• Device: OnePlus 8T
• OS: Android 13
• Obtainium version: 1.1.30

[cyberboh]

I already reported it some months ago in #1097
He closed it and talked about my IP, but the e issue actually from Obtainium itself can not parse HTML from APKMIRROR which is protected by cloudflare. I already ping him but no response till now.

[ImranR98]

Yeah it seems like cloudflare protection has gone from being inconsistent to always on. May have to remove the apkmirror source if a solution is not found.

[bossanovaorca]

oh ok I did see #1097 when searching for previous tickets but I wasn't aware that my current issue was Cloudflare related. I may have to look for other sources to migrate to then

[DwainZwerg]

@archon810 Do you have an idea?

[DwainZwerg]

Do APK Grabber (archived in November 2024) and APK Updater (possibly this post deals with the same problem in the latest issue) have the same problem? If not, we could use their source code.

[archon810]

Try setting a proper user agent that references the app being used to query ("Obtainium" with the version would be a good start).

[ImranR98]

@archon810 I've changed the user agent to Obtainium/<version> (currently Obtainium/1.1.31. Still seeing the same error.
Btw we currently rely on web scraping but it sounds like (searching through rumboalla/apkupdater#16) like there is an APKMirror API. Do you have any documentation on it? I can't find much about it.

[fahim-ahmed05]

Try with common browser user agent

[aleks01100001]

Try with common browser user agent

Can we change the user agent ourselves?

[DwainZwerg]

Yes, in the settings of the respective app.

[leo-liar]

Yes, in the settings of the respective app.

When adding a non-HTML-source app (like one from APKMirror) this is not possible.

[fahim-ahmed05]

Yes, in the settings of the respective app.

Don't see any option to change the user agent

[cyberboh]

1.1.32 still forbidden error appear

[ImranR98]

Latest version now correctly identifies Obtainium in the user agent (with no option to override). @archon810 please let us know if there's anything else that can be done. If not the source should be removed.

[archon810]

@ImranR98 Now that there's a proper UA, I took a look at the Cloudflare WAF stats. I found that the vast majority of "Obtainium/*" user-agent requests (over 10k over the past 24h) were allowed, with only several blocked or challenged. Is the version with this user-agent already released out into the wild?

The challenge reasons were:

• usage of tor exit nodes (which have extremely high malicious traffic rates and are therefore challenged or blocked)
• some urls are challenged to make certain bot access to them harder. I can explain a little more privately if you contact me on Telegram with the same username as I have here on Github.

Perhaps your test includes one of the challenged urls or you're using tor? Otherwise, please let me know on Telegram what you're querying for and perhaps your IP so I could look into it.

Furthermore, I can explain how APKUpdater/APKGrabber use our API and perhaps generate an API key for Obtainium so that you don't have to ping our feeds at all and instead batch request 100 apps at a time. In return, I ask only that Obtainium, just like APKGrabber and APKUpdater, continues to show users our download pages so that we can generate some ad revenue and pay for our server and operational costs.

[ImranR98]

Thanks @archon810, good to see that it's working for most people (I'm still seeing "Forbidden" with the couple of apps I just tested, "Garmin Connect IQ" and Netflix). The RSS feed request fails, for example https://www.apkmirror.com/apk/netflix-inc/netflix/feed/.

Furthermore, I can explain how APKUpdater/APKGrabber use our API and perhaps generate an API key for Obtainium so that you don't have to ping our feeds at all and instead batch request 100 apps at a time.

Yes, a proper API would probably be better than using the RSS feed. Obtainium currently only queries for one app at a time - so using the API would not be more efficient in that sense - but I'm guessing the API provides more info, is easier to parse, and may allow for other features like search.

Do you know how we could include the API key in Obtainium without making it public? Not sure how APKUpdater/APKGrabber do it. Or would it be okay to make that public?

I ask only that Obtainium, just like APKGrabber and APKUpdater, continues to show users our download pages so that we can generate some ad revenue and pay for our server and operational costs.

Yes, APKMirror is a "track-only" source and will stay that way.

[DwainZwerg]

So at least I still have forbidden errors too, all the time.

[FangPam]

Latest version now correctly identifies Obtainium in the user agent (with no option to override). @archon810 please let us know if there's anything else that can be done. If not the source should be removed.

Hello, how to set the "Custom user-agent for APKMirror",I wasn't able to find the button.I'd be grateful if you could advise me.