***IMPORTANT***: Instagram's latest changes and how we're reacting to them

[chris-greening]

Hey everyone!

Many have you have likely noticed the uptick of InstagramLoginRedirectError messages..... 😅

Instagram's been making it increasingly harder to scrape data for the last few months now. I've been able to roll with the punches almost immediately in all instances but their latest changes this last week have been the hardest yet.

THAT BEING SAID I've spent 12+ hours the last week experimenting and investigating and I have come up with a solution that I am implementing in the code. I will be rolling out incremental, non-breaking changes as I make them and will be as transparent as possible in the version notes. I will be blogging extensively about how you can use these new tools and features and will post plenty of reference material/documentation.

Thanks so much for your patience and I think you're going to really like the new features that are coming 😏

Sneak peak of (some) things to expect:

• Fixes to the HTTP request for all scrapers, Instagram is now enforcing the need to pass a sessionid cookie for requests
• Dedicated selenium support (will still avoid selenium as a dependency but will instead allow for passing webdrivers as an arg)
• Increased flexibility, mix-and-match tools/functions and all sorts of neat things for getting data

Under the hood, I will also be reimplementing the mapping of data points to their scrapers so that the codebase will be drastically easier to understand. As it stands now, a lot of the algorithm is based off of old code that isn't really necessary anymore so that's gonna be cleaned up.

Cheers and let me know if you have any questions!
Chris

[chris-greening]

Here is a blog post with more info:

https://dev.to/chrisgreening/so-it-goes-instascrape-v2-0-0-is-in-the-works-3l8n

[chris-greening]

In case you missed it, instascrape 2.x.x is out! Check out the version release notes. If you're going to pip install it, make sure to grab insta-scrape==2.0.2 to get the latest patch I just pushed this morning!

[maheshs11]

InstagramLoginRedirectError

[maheshs11]

import requests

from instascrape import Profile

headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.74 Safari/537.36 Edg/79.0.309.43",
"cookie":'INSERT_SESSION_ID_HERE'
}

resp = requests.get("https://www.instagram.com/google/", headers=headers)

google = Profile(resp.text).scrape()

in output:
InstagramLoginRedirectError

in dev tool:
Refused to load the font 'https://fonts.gstatic.com/s/opensans/v13/cJZKeOuBrn4kERxqtaUH3VtXRa8TVwTICgirnJhmVJw.woff2' because it violates the following Content Security Policy directive: "font-src data: https://.fbcdn.net https://.instagram.com https://*.cdninstagram.com".

[chris-greening]

Hello! You're so close, you just have to specify in headers["cookie"] that you are passing a sessionid.

Check out this code for the solution:

from instascrape import Profile

# PASTE YOUR SESSIONID HERE
SESSIONID = ''

headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.74 Safari/537.36 Edg/79.0.309.43",
"cookie":f'sessionid={SESSIONID};'
}
google = Profile("google").scrape()

[danielorlando97]

In that code, I think you forgot to set the 'header' parameter of the scrape function. I had the same problem and changing this line solved it

google = Profile("google").scrape(headers=headres)

[franzea123]

Hi Chris, thanks for your awesome development. I've just installed insta-scrape-2.1.1 using the console at pycharm by typing "pip install insta-scrape". I'm running Python 3.8.7 . The installation completed successfully, but when i tried to download the photos from an account, i got the following error:

InstagramLoginRedirectError instascrape.exceptions.exceptions.InstagramLoginRedirectError: Instagram is redirecting you to the login page instead of the page you are trying to scrape. This could be occuring because you made too many requests too quickly or are not logged into Instagram on your machine. Try passing a valid session ID to the scrape method as a cookie to bypass the login requirement
As i read you're already working in this issue, thank you very much. I'll definitely stay updated on your advances.

I also have a question. Does the profile scraping method downloads all of the instagram photos from a profile from the first photo till the last or only the latest very recent photos? if not, can it be combined with selenium for infinite scrolling till the bottom of the posts in order to get all the photos?

[chris-greening]

Hey! I wrote a blog post pretty recently that should answer your questions!

Instagram recently made it harder to scrape data and you now have to pass a valid session ID almost all the time. You can find details on obtaining your session id here and details on passing it to the scrape method are in the blog post I listed above (see issue #78 as well, @beckettnormington provided a perfect example)

In terms of getting all images from an Instagram profile, you will have to follow along with everything the blog post does (copied below for convenience)

from selenium.webdriver import Chrome 
from instascrape import Profile, scrape_posts

webdriver = Chrome("path/to/chromedriver.exe")

headers = {
    "user-agent": "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Mobile Safari/537.36 Edg/87.0.664.57",
    "cookie": "sessionid=PASTE_YOUR_SESSIONID_HERE;"
}
joe = Profile("joebiden")
joe.scrape(headers=headers)

posts = joe.get_posts(webdriver=webdriver, login_first=True)

scraped_posts, unscraped_posts = scrape_posts(posts, headers=headers, pause=10)

This script will infinite scroll to the bottom of a profile using a selenium webdriver and scrape every post. From here you can loop through each scraped Post in scraped_posts and use the download method as you normally would.

Hope this helps!

[RichReads]

Hi Chris

I've copied the code you posted for scraping all Joe Biden Instagram images from his profile and changed it to try and scrape all posts from the profile, menwithstreetstyle.
In the pasted code below, I've changed the actual variables for webdriver, SESSIONID, headers and filepath in case it's sensitive information.

The script is working except it stops scraping after doing 12 images usually.
It'll scrape the latest 12 images from the profile then stop.
The time before last, it scraped 36 images then stopped. I'm not sure why it did 36 images that time.
But anyways, my main point is it stops well before scraping all the images from the profile.
Could you give any suggestions on how I could get it to scrape all the images from the profile?
Thanks for any help.

from selenium.webdriver import Chrome
from instascrape import Profile, scrape_posts

# Creating our webdriver
webdriver = Chrome(absolutefilepath)

# Scraping profile
SESSIONID = 'mysessionid'
headers = copiedfromyourblogpost
mwss = Profile("menwithstreetstyle")
mwss.scrape(headers=headers)

# Scraping the posts
posts = mwss.get_posts(webdriver=webdriver, amount=6967, login_first=True)
scraped, unscraped = scrape_posts(posts, silent=False, headers=headers, pause=10)

counter = 1
for i in scraped:
    filepath = 'absolutefilepath'
    i.download(filepath + str(counter) + '.jpg')
    counter += 1

[rgyllenhammer]

Hello @RichReads ! Hopefully you have a solution by now, but in case anyone else was wondering I was able to get something working by manually scraping them like this:

from selenium.webdriver import Chrome
from instascrape import Profile, scrape_posts
import time

# Creating our webdriver
webdriver = Chrome(absolutefilepath)

# Scraping profile
SESSIONID = 'mysessionid'
headers = copiedfromyourblogpost
mwss = Profile("menwithstreetstyle")
mwss.scrape(headers=headers)

# Scraping the posts
posts = mwss.get_posts(webdriver=webdriver, amount=6967, login_first=True)

# Mutates posts inplace so after this for loop complets you may access caption, url, etc
post_count = 0
for post in posts:
    post.scrape(headers=headers)
    time.sleep(5)
    print('scraping post', post_count)
    post_count += 1

print(posts[0]['caption'])

Eventually you still get rate limited by instagram after about 250 posts, but I found that deleting and re-installing your chromedriver file allows you to bypass this Nevermind, deleting and re-installing only worked a single time, I have not found a solution to being rate limited by instagram after scraping 250 posts. Switching accounts and machines seems to do SOMETHING, but sometimes only allows 30 - 40 more accounts to be scraped, sometimes lets another 250 through.

[kenho211]

Will we get Instagram account blocked if overusing a sessionid for scraping? Is it possible to get sessionid without using an account?

[tarob0ba]

To answer your questions:

1. Instagram will not ban or suspend your account, but it may temporarily have you verify that you are not a robot.
2. No, a session id is linked to an account. This is how Instagram "remembers" you.

[kenho211]

Thanks a lot!

[AnhNgDo]

Hi @chris-greening,

Thanks again for sharing your work with us.

I've been trying to scrape post counts for a number of hashtags, but always run into InstagramLoginRedirectError after about 10 requests. I tried using header and session-id but that doesn't seem to help. Is there anything else I could do?

Here is the code:

start_time = time()

tag_data = []

for idx, row in tag_df.iterrows():
    tag = row['Instagram Tag']
    related = row ['Related Tag']
    full_url = base_url + tag + '/'
    insta_tag = Hashtag(full_url)
    print(full_url)
    
    #Wait for  normally randomized amount of time 
    rand_time = abs(np.random.normal(WAIT_TIME, 1.5))
    sleep(rand_time)
    
    insta_tag.scrape(headers=headers, webdriver=browser)
    tag_data.append(('#'+tag, insta_tag.amount_of_posts, related))
    print(('#'+tag, insta_tag.amount_of_posts, related))

print("--- %s seconds ---" % (time() - start_time))

I used Chrome webdriver and headers as per your instructional code:

headers = {"user-agent": "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Mobile Safari/537.36 Edg/87.0.664.57",
           "cookie": f"sessionid={SESSIONID};"}

Thank you!

[sahiljena]

Hi, @chris-greening I have been using this sessionid method for scrapping posts and other things. But after some requests, the account gets disabled.
Is there any way of solving this?

[Chocomagma]

Sir is it still no solution for Post ?
"HACK: This isn't a very clean solution and there is certainly a better
way to deal with returning a Post object with only partial data"

if hasattr(return_instance, "timestamp"):
    return_instance.upload_date = datetime.datetime.fromtimestamp(return_instance.timestamp)

[asecuesu]

After

Installing collected packages: insta-scrape
Successfully installed insta-scrape-2.0.2

I try to use it :

from selenium.webdriver import Chrome from instascrape import Profile, scrape_posts

And I get error :

---

ImportError Traceback (most recent call last)
in ()
1 from selenium.webdriver import Chrome
----> 2 from instascrape import Profile, scrape_posts

ImportError: cannot import name 'scrape_posts' from 'instascrape' (/usr/local/lib/python3.7/dist-packages/instascrape/init.py)