HOW TO FIX - Traceback (most recent call last): File "/usr/local/bin/twint", line 33, in <module> && File "/home/rangersmyth_74/.local/lib/python3.7/site-packages/twint/token.py", line 69, in refresh raise RefreshTokenException('Could not find the Guest token in HTML') twint.token.RefreshTokenException: Could not find the Guest token in HTML

[rangersmyth74]

I really love this app and I have only used it once. But now I want to use it and I can't. I have searched so much that my eyes are sore, why am I getting these errors and why can't I find a fix for it!! hehe...

Linux Kali-ROG 5.14.0-kali4-amd64 #1 SMP Debian 5.14.16-1kali1 (2021-11-05) x86_64 GNU/Linux

Please help not just me but anyone who searches the 400 pages of information to find an answer. 9 hours of work to try this, the time I will never get back and I just want to chuck in Linux altogether and do anything else.

• [] Python version is 3.6;
• [] Updated Twint with pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint;

These are the steps I took to install twint and get try and get it to work on 2 versions of Linux and 1 Deb. I have spent 9 hours trying my best, but no fix anywhere, none at all, not one.

What errors am I getting, all listed below

1: Install on Kali. This did not work for me.

Linux Kali-ROG 5.14.0-kali4-amd64 #1 SMP Debian 5.14.16-1kali1 (2021-11-05) x86_64 GNU/Linux

Instructions for install : https://github.com/twintproject/twint

    git clone –depth=1 https://github.com/twintproject/twint.git
    cd twint
    pip3 install . -r requirements.txt        

Try and run these commands to see if it works.

   $ twint -s pineapple
    $ twint -u networkchuck -s "raspberry pi" 

2: Install on Kali AWS. This did not work for me.

Linux kali 5.15.0-kali2-cloud-amd64 #1 SMP Debian 5.15.5-2kali2 (2021-12-22) x86_64 GNU/Linux

It seems that twint and AWS do not work due to

Instructions for install : https://www.geeksforgeeks.org/how-to-use-twint-osint-tool-on-google-cloud-console/

    sudo git clone –depth=1 https://github.com/twintproject/twint.git
    sudo cd twint
    sudo pip3 install . -r requirements.txt
    sudo pip3 install twint

Try and run these commands to see if it works.

    $ twint -s pineapple
    $ twint -u networkchuck -s "raspberry pi" 

3: Install on Google Cloud Shell and This worked for me.

Lightbox Linux cs-899333161534-default-boost-h8jtm 5.10.68+ #1 SMP Wed Dec 1 10:07:21 UTC 2021 x86_64 GNU/Linux

Instructions for install: https://www.geeksforgeeks.org/how-to-use-twint-osint-tool-on-google-cloud-console/

FYI 1 - Google didn't like the command git clone --depth=1

So I removed it and it installed.

FY 2 - Using pip3 install twint command said all was installed, but I added sudo

$ sudo git clone https://github.com/twintproject/twint.git
$ ls
$ cd twint
$ ls
$ sudo pip3 install . -r requirements.txt
$ sudo pip3 install twint

Ran commands to see if it works.

    $ twint -s pineapple
    $ twint -u networkchuck -s "raspberry pi"

I was able to run twint until I logged out and logged back in, it didn't work anymore.

The Kali twint install journey.

My Install on Kali Linux.

┌──(kali㉿kali)-[~]
└─$ git clone --depth=1 https://github.com/twintproject/twint.git
Cloning into 'twint'...
remote: Enumerating objects: 47, done.
remote: Counting objects: 100% (47/47), done.
remote: Compressing objects: 100% (44/44), done.
remote: Total 47 (delta 3), reused 14 (delta 0), pack-reused 0
Receiving objects: 100% (47/47), 42.95 KiB | 1.59 MiB/s, done.
Resolving deltas: 100% (3/3), done.

┌──(kali㉿kali)-[~]
└─$ ls
Desktop Documents Downloads Music Pictures Public Templates Videos bash.txt nmap twint

┌──(kali㉿kali)-[~]
└─$ cd twint

┌──(kali㉿kali)-[~/twint]
└─$cd twint

┌──(kali㉿kali)-[~/twint]
└─$ pip3 install . -r requirements.txt
Command 'pip3' not found, but can be installed with:
sudo apt install python3-pip
Do you want to install it? (N/y)y
sudo apt install python3-pip

┌──(kali㉿kali)-[~/twint]
└─$ pip3 install . -r requirements.txt

WARNING: The script twint is installed in '/home/kali/.local/bin' which is not on PATH.

Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed aiohttp-socks-0.4.1 cchardet-2.1.7 dataclasses-0.6 elasticsearch-7.16.2 fake-useragent-0.1.11 geographiclib-1.52 geopy-2.2.0 googletransx-2.4.2 pandas-1.3.5 schedule-1.1.0 twint-2.1.21

(I added two commands for the path later on in the install, see far below).

┌──(kali㉿kali)-[~/.local]
└─$ cd bin

┌──(kali㉿kali)-[~/.local/bin]
└─$ ls
cchardetect translate twint

┌──(kali㉿kali)-[~/.local/bin]
└─$ twint -u networkchuck
Traceback (most recent call last):
File "/usr/local/bin/twint", line 33, in
sys.exit(load_entry_point('twint==2.1.21', 'console_scripts', 'twint')())
File "/usr/local/bin/twint", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/lib/python3.9/importlib/metadata.py", line 77, in load
module = import_module(match.group('module'))
File "/usr/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 972, in _find_and_load_unlocked
File "", line 228, in _call_with_frames_removed
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 850, in exec_module
File "", line 228, in _call_with_frames_removed
File "/home/kali/.local/lib/python3.9/site-packages/twint/init.py", line 14, in
from . import run
File "/home/kali/.local/lib/python3.9/site-packages/twint/run.py", line 4, in
from . import datelock, feed, get, output, verbose, storage
File "/home/kali/.local/lib/python3.9/site-packages/twint/get.py", line 12, in
from aiohttp_socks import ProxyConnector, ProxyType
File "/home/kali/.local/lib/python3.9/site-packages/aiohttp_socks/init.py", line 5, in
from .connector import (
File "/home/kali/.local/lib/python3.9/site-packages/aiohttp_socks/connector.py", line 8, in
from aiohttp.helpers import CeilTimeout # noqa
ImportError: cannot import name 'CeilTimeout' from 'aiohttp.helpers' (/usr/lib/python3/dist-packages/aiohttp/helpers.py)

┌──(kali㉿kali)-[~/.local/bin]
└─$ echo $SHELL

/bin/zsh

┌──(kali㉿kali)-[~/.local/bin]
└─$ bash

┌──(kali㉿kali)-[~/.local/bin]
└─$ myip
Command 'myip' not found, did you mean:
command 'mzip' from deb mtools
Try: sudo apt install

┌──(kali㉿kali)-[~/.local/bin]
└─$ twint -u networkchuck -s "raspberry pi"
Traceback (most recent call last):
File "/usr/local/bin/twint", line 33, in
sys.exit(load_entry_point('twint==2.1.21', 'console_scripts', 'twint')())
File "/usr/local/bin/twint", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/lib/python3.9/importlib/metadata.py", line 77, in load
module = import_module(match.group('module'))
File "/usr/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 972, in _find_and_load_unlocked
File "", line 228, in _call_with_frames_removed
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 850, in exec_module
File "", line 228, in _call_with_frames_removed
File "/home/kali/.local/lib/python3.9/site-packages/twint/init.py", line 14, in
from . import run
File "/home/kali/.local/lib/python3.9/site-packages/twint/run.py", line 4, in
from . import datelock, feed, get, output, verbose, storage
File "/home/kali/.local/lib/python3.9/site-packages/twint/get.py", line 12, in
from aiohttp_socks import ProxyConnector, ProxyType
File "/home/kali/.local/lib/python3.9/site-packages/aiohttp_socks/init.py", line 5, in
from .connector import (
File "/home/kali/.local/lib/python3.9/site-packages/aiohttp_socks/connector.py", line 8, in
from aiohttp.helpers import CeilTimeout # noqa
ImportError: cannot import name 'CeilTimeout' from 'aiohttp.helpers' (/usr/lib/python3/dist-packages/aiohttp/helpers.py)

──(kali㉿kali)-[~/.local/bin]
└─$ cd ..

┌──(kali㉿kali)-[~/.local]
└─$ cd..
cd..: command not found

┌──(kali㉿kali)-[~/.local]
└─$ cd ..

┌──(kali㉿kali)-[~]
└─$

┌──(kali㉿kali)-[~]
└─$ twint -u networkchuck -s "raspberry pi"
Command 'twint' not found, did you mean:
command 'twine' from deb twine
Try: sudo apt install

┌──(kali㉿kali)-[~/twint]
└─$ python3
Python 3.9.9 (main, Dec 16 2021, 23:13:29)
[GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

┌──(kali㉿kali)-[~/twint]
└─$ twint -u networkchuck -s "raspberry pi"
Command 'twint' not found, did you mean:
command 'twine' from deb twine
Try: sudo apt install

┌──(kali㉿kali)-[~]
└─$ sudo apt update && sudo apt upgrade

┌──(kali㉿kali)-[~]
└─$ systemctl reboot -i

┌──(kali㉿kali)-[~/twint]
└─$cd twint

┌──(kali㉿kali)-[~/twint]
└─$ chmod +x setup.py

┌──(kali㉿kali)-[~/twint]
└─$ python3 setup.py
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help

error: no commands supplied
──(kali㉿kali)-[~/twint]
└─$ python3 setup.py --help
Common commands: (see '--help-commands' for more)

setup.py build will build the package underneath 'build/'
setup.py install will install the package

┌──(kali㉿kali)-[~/twint]
└─$ sudo python3 setup.py install

running install

┌──(kali㉿kali)-[~/twint]
└─$ sudo python3 setup.py build
running build
running build_py

┌──(kali㉿kali)-[~/twint]
└─$ twint -u networkchuck -s "raspberry pi"
Traceback (most recent call last):
File "/usr/local/bin/twint", line 33, in
sys.exit(load_entry_point('twint==2.1.21', 'console_scripts', 'twint')())
File "/usr/local/bin/twint", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/lib/python3.9/importlib/metadata.py", line 77, in load
module = import_module(match.group('module'))
File "/usr/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 972, in _find_and_load_unlocked
File "", line 228, in _call_with_frames_removed
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 850, in exec_module
File "", line 228, in _call_with_frames_removed
File "/home/kali/.local/lib/python3.9/site-packages/twint/init.py", line 14, in
from . import run
File "/home/kali/.local/lib/python3.9/site-packages/twint/run.py", line 4, in
from . import datelock, feed, get, output, verbose, storage
File "/home/kali/.local/lib/python3.9/site-packages/twint/get.py", line 12, in
from aiohttp_socks import ProxyConnector, ProxyType
File "/home/kali/.local/lib/python3.9/site-packages/aiohttp_socks/init.py", line 5, in
from .connector import (
File "/home/kali/.local/lib/python3.9/site-packages/aiohttp_socks/connector.py", line 8, in
from aiohttp.helpers import CeilTimeout # noqa
ImportError: cannot import name 'CeilTimeout' from 'aiohttp.helpers' (/usr/lib/python3/dist-packages/aiohttp/helpers.py)

Research Issue.

Fix 1: I searched on Google with the error and found,

#1071

Asked to run the command below to check if there was a .local/bin folder created.

$ echo $PATH | tr -s ":" "\n" | sort

┌──(kali㉿kali)-[~/Downloads/twint]
└─$ echo $PATH | tr -s ":" "\n" | sort 1 ⨯
/bin
/home/kali/.dotnet/tools
/home/kali/.local/bin
/sbin
/usr/bin
/usr/games
/usr/local/bin
/usr/local/games
/usr/local/sbin
/usr/sbin

The advice was to add these two lines to the .zshrc script as I am using zsh shell.
I am having the same issue in bash, so I don't think it matters at all for the moment.

$ export PYTHON_BIN_PATH="$(python3 -m site --user-base)/bin"

$ export PATH="$PATH:$PYTHON_BIN_PATH"

This method didn't work for me.
Still have the error.

Fix2: $ pip3 install yarl --force-reinstall --no-cache-dir

https://stackoverflow.com/questions/64747304/twint-python-library-is-causing-exception-for-search-query?rq=1

A fix was mentioned by a user - "I had the same issue with python3.7 and resolve it by reinstalling yarl, like this way:"

$ pip3 install yarl --force-reinstall --no-cache-dir

This method didn't work for me.

Fix3: To uninstall and reinstall twint with the upgrade.

#915

    pip3 uninstall twint
    pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint
    sudo pip3 install twint

This method didn't work for me.

Fix4: Install on Google Cloud Shell.

Google Cloud New build

    sudo git clone –depth=1 https://github.com/twintproject/twint.git
    cd twint
    sudo pip3 install . -r requirements.txt
    pip3 install twint

This worked for me but once you leave the session, twint seems to be causing the same issue as I had on the Kali machine.

twint -u networkchuck –limit 20 (Did not recognise -limit 20)

twint -u networkchuck -s “raspberry pi”

New Error message.

$ twint -u networkchuck -s crypto -o rightnow.json–json

Traceback (most recent call last):
File "/home/rangersmyth_74/.local/bin/twint", line 10, in
sys.exit(run_as_command())
File "/home/rangersmyth_74/.local/lib/python3.7/site-packages/twint/cli.py", line 339, in run_as_command
main()
File "/home/rangersmyth_74/.local/lib/python3.7/site-packages/twint/cli.py", line 330, in main
run.Search(c)
File "/home/rangersmyth_74/.local/lib/python3.7/site-packages/twint/run.py", line 410, in Search
run(config, callback)
File "/home/rangersmyth_74/.local/lib/python3.7/site-packages/twint/run.py", line 329, in run
get_event_loop().run_until_complete(Twint(config).main(callback))
File "/home/rangersmyth_74/.local/lib/python3.7/site-packages/twint/run.py", line 36, in init
self.token.refresh()
File "/home/rangersmyth_74/.local/lib/python3.7/site-packages/twint/token.py", line 69, in refresh
raise RefreshTokenException('Could not find the Guest token in HTML')
twint.token.RefreshTokenException: Could not find the Guest token in HTML

Searching for twint.token

#1146
Fix 5. Add to PATH.
https://issueexplorer.com/issue/twintproject/twint/1189

"" MikeTheScriptKid wrote this answer on 2021-05-08 ""
So I finally got the twint command working in Kali.
I added the following to /etc/environment
/your_user/.local/bin

"" HarryWestFord wrote this answer on 2021-05-10 ""
"sorry for being such a newbie but how excalty do you add this on the command line?

"" hackingbutlegal wrote this answer on 2021-05-15 ""

nano /etc/environment

This didn't work for me.

Found a fix to sort out the token problem

SCRIPT from >> #1146

import re
import time
import logging as logme
import requests

class TokenExpiryException(Exception):
def init(self, msg):
super().init(msg)

class RefreshTokenException(Exception):
def init(self, msg):
super().init(msg)

class Token:
def init(self, config):
self._session = requests.Session()
self._session.headers.update(
{'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0'})
self.config = config
self._proxies = self._set_proxies()
self._retries = 5
self._timeout = 10
self.url = 'https://twitter.com'

def _set_proxies(self) -> dict:
    settings = [self.config.Proxy_type, self.config.Proxy_host, self.config.Proxy_port]
    if not all(settings):
        logme.debug(f"No proxy in config")
        return {}

    proxy_type = self.config.Proxy_type.lower()
    proxy_val = f"{self.config.Proxy_host}:{self.config.Proxy_port}"
    proxies = {proxy_type: proxy_val}
    if proxy_type == 'http':
        proxies['https'] = proxy_val
    return proxies

def _request(self):
    for attempt in range(self._retries + 1):
        # The request is newly prepared on each retry because of potential cookie updates.
        req = self._session.prepare_request(requests.Request('GET', self.url))
        logme.debug(f'Retrieving {req.url}')
        try:
            if self._proxies:
                r = self._session.send(
                    req,
                    allow_redirects=True,
                    timeout=self._timeout,
                    proxies=self._proxies,
                    verify=False
                )
            else:
                r = self._session.send(req, allow_redirects=True, timeout=self._timeout)
        except requests.exceptions.RequestException as exc:
            if attempt < self._retries:
                retrying = ', retrying'
                level = logme.WARNING
            else:
                retrying = ''
                level = logme.ERROR
            logme.log(level, f'Error retrieving {req.url}: {exc!r}{retrying}')
        else:
            success, msg = (True, None)
            msg = f': {msg}' if msg else ''

            if success:
                logme.debug(f'{req.url} retrieved successfully{msg}')
                return r
        if attempt < self._retries:
            # TODO : might wanna tweak this back-off timer
            sleep_time = 2.0 * 2 ** attempt
            logme.info(f'Waiting {sleep_time:.0f} seconds')
            time.sleep(sleep_time)
    else:
        msg = f'{self._retries + 1} requests to {self.url} failed, giving up.'
        logme.fatal(msg)
        self.config.Guest_token = None
        raise RefreshTokenException(msg)

def refresh(self):
    logme.debug('Retrieving guest token')
    res = self._request()
    match = re.search(r'\("gt=(\d+);', res.text)
    if match:
        logme.debug('Found guest token in HTML')
        self.config.Guest_token = str(match.group(1))
    else:
        self.config.Guest_token = None
        raise RefreshTokenException('Could not find the Guest token in HTML')

I ran this script and got back nothing, no error, but when I did a search it didn't work.

Back to the Drawing Board.

References:

https://github.com/twintproject/twint/wiki/Setup

https://github.com/twintproject/twint/wiki/Configuration

#468

#915

#917

#944

#980

#1114 - Has a fix to upgrade twint && I also repored my issue here.

#1146 - Token Fix

https://github.com/Altimis/Scweet

http://saka.docsio.net/66892325/scrape-join-dates-user-info-from-a-list-csv-of-twitter-users

http://saka.docsio.net/64747304/twint-python-library-is-causing-exception-for-search-qeruy

https://www.geeksforgeeks.org/how-to-use-twint-osint-tool-on-google-cloud-console/

https://github.com/himanshudabas/twint/tree/twint-fixes

https://www.kaggle.com/zelinngilo/punya-nadia?scriptVersionId=57423357

https://www.geeksforgeeks.org/how-to-use-twint-osint-tool-on-google-cloud-console/

https://stackoverflow.com/questions/64747304/twint-python-library-is-causing-exception-for-search-query?rq=1

https://www.bountysource.com/teams/tweep/issues?tracker_ids=65007358

https://issueexplorer.com/issue/twintproject/twint/1266

[minamotorin]

Solution

The following patch works for me.

# This patch is WTFPL (http://www.wtfpl.net/txt/copying/) and no warranty.
diff --git a/twint/token.py b/twint/token.py
index ae66a24..2eedcee 100644
--- a/twint/token.py
+++ b/twint/token.py
@@ -65,5 +65,30 @@ class Token:
             logme.debug('Found guest token in HTML')
             self.config.Guest_token = str(match.group(1))
         else:
-            self.config.Guest_token = None
-            raise RefreshTokenException('Could not find the Guest token in HTML')
+            headers = {
+                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0',
+                'authority': 'api.twitter.com',
+                'content-length': '0',
+                'authorization': self.config.Bearer_token,
+                'x-twitter-client-language': 'en',
+                'x-csrf-token': res.cookies.get("ct0"),
+                'x-twitter-active-user': 'yes',
+                'content-type': 'application/x-www-form-urlencoded',
+                'accept': '*/*',
+                'sec-gpc': '1',
+                'origin': 'https://twitter.com',
+                'sec-fetch-site': 'same-site',
+                'sec-fetch-mode': 'cors',
+                'sec-fetch-dest': 'empty',
+                'referer': 'https://twitter.com/',
+                'accept-language': 'en-US',
+            }
+            self._session.headers.update(headers)
+            req = self._session.prepare_request(requests.Request('POST', 'https://api.twitter.com/1.1/guest/activate.json'))
+            res = self._session.send(req, allow_redirects=True, timeout=self._timeout)
+            match = re.search(r'{"guest_token":"(\d+)"}', res.text)
+            if match:
+                logme.debug('Found guest token in JSON')
+                self.config.Guest_token = str(match.group(1))
+            else:
+                self.config.Guest_token = None
+                raise RefreshTokenException('Could not find the Guest token in JSON')

I don't understand session of requests, so the code may be not good.
I hope someone rewrite the patch better and create a pull request.

About the problem

In my environment, this problem has recently begun to occur.
This doesn't happen every time, so if you are lucky, you don't get the error.

The cause is literally literally that twint could not find the Guest token in HTML.
Actually, sometimes token isn't included in HTML recently.

#!/usr/bin/env python3
# This program is WTFPL.
import requests

res = requests.get('https://twitter.com')
print(res.text.split('\n')[-1])

twint require the result of running the above code is })();</script><script nonce="VALUE">document.cookie = decodeURIComponent("gt=VALUE; Max-Age=VALUE; Domain=.twitter.com; Path=/; Secure");</script>.
However, sometimes the result is only })();</script> and missing the Guest token.

About the solution

In my patch, twint get the Guest token from https://api.twitter.com/1.1/guest/activate.json if could not find the one.
I referred to the code of gallery-dl.

These are the steps I took to install twint and get try and get it to work on 2 versions of Linux and 1 Deb. I have spent 9 hours trying my best, but no fix anywhere, none at all, not one.

Unfortunately, twint hasn't been updated for a long time.
If the problem is new, updating will not solve the problem.

In this case, the problem is too new to find a solution.
However, the problem may not happen in other environments, so I want reports.

[yvessaintlaureint]

tbh, I've got the same problem like this. My OS is Windows and I'm trying to scrap twitter's tweet in google colab. In the end, RefreshTokenException: Could not find the Guest token in HTML appear everytime I ran twint.run.Search(c). I though that it was just from google colab but after I tried it in Jupyter, the result still the same which is couldn't find the guest token. Any solution for this?

[LinqLover]

Thank you for the patch @minamotorin which worked for me too! I have created a PR #1322, hopefully any maintainer will be able to merge it soon.

[minamotorin]

@LinqLover @gabrielruoff

I forgot to update log message, so I updated my patch.
Please confirm it.

The changes are the following two points.

• logme.debug('Found guest token in JSON') was added.
• Error message of RefreshTokenException was changed.

[aaabbbbbbb]

@minamotorin @LinqLover
although i put your solution part in my token.py file, i seem to get this error 'twint.token.RefreshTokenException: Could not find the Guest token in JSON'
can you please guide me through the solution process or even share your token.py file

The OS that i am currently using is Kali Linux

[prhbrt]

Additionally to @minamotorin's response, I use this hotfix in my ipynbs. It also adds a response field to the exception, to simplify debugging.

import nest_asyncio
nest_asyncio.apply()
import re
import twint
from twint.token import Token as OriginalToken, RefreshTokenException

class Token(OriginalToken):
  def __init__(self, config):
    super().__init__(config)
    self._session.headers.update({
      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'
    })
  
  def refresh(self):
    res = self._request()
    match = re.search(r'\("gt=(\d+);', res.text)
    if match:
      self.config.Guest_token = str(match.group(1))
    else:
      headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0',
        'authority': 'api.twitter.com',
        'content-length': '0',
        'authorization': self.config.Bearer_token,
        'x-twitter-client-language': 'en',
        'x-csrf-token': res.cookies.get("ct0"),
        'x-twitter-active-user': 'yes',
        'content-type': 'application/x-www-form-urlencoded',
        'accept': '*/*',
        'sec-gpc': '1',
        'origin': 'https://twitter.com',
        'sec-fetch-site': 'same-site',
        'sec-fetch-mode': 'cors',
        'sec-fetch-dest': 'empty',
        'referer': 'https://twitter.com/',
        'accept-language': 'en-US',
      }
      self._session.headers.update(headers)
      req = self._session.prepare_request(requests.Request('POST', 'https://api.twitter.com/1.1/guest/activate.json'))
      res = self._session.send(req, allow_redirects=True, timeout=self._timeout)
      match = re.search(r'{"guest_token":"(\d+)"}', res.text)
      if match:
        self.config.Guest_token = str(match.group(1))
      else:
        self.config.Guest_token = None
        exception = RefreshTokenException('Could not find the Guest token in HTML')
        exception.response = res
        raise exception

twint.token.Token = Token

query = "#kaag" #@param {"type": "string"}
since = "2021-01-01" #@param {"type": "string"}
until = "2022-01-08" #@param {"type": "string"}


c = twint.Config()
query = query
c.Search = query
c.Limit = 1000
c.Since, c.Until = since, until
c.Store_object =  True
c.User_full = True
c.Profile_full = True
c.Hide_output = True

try:
  twint.run.Search(c)
except Exception as e:
  response = e.response
  raise

[rangersmyth74]

Solution

The following patch works for me.

# This patch is WTFPL (http://www.wtfpl.net/txt/copying/) and no warranty.
diff --git a/twint/token.py b/twint/token.py
index ae66a24..2eedcee 100644
--- a/twint/token.py
+++ b/twint/token.py
@@ -65,5 +65,30 @@ class Token:
             logme.debug('Found guest token in HTML')
             self.config.Guest_token = str(match.group(1))
         else:
-            self.config.Guest_token = None
-            raise RefreshTokenException('Could not find the Guest token in HTML')
+            headers = {
+                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0',
+                'authority': 'api.twitter.com',
+                'content-length': '0',
+                'authorization': self.config.Bearer_token,
+                'x-twitter-client-language': 'en',
+                'x-csrf-token': res.cookies.get("ct0"),
+                'x-twitter-active-user': 'yes',
+                'content-type': 'application/x-www-form-urlencoded',
+                'accept': '*/*',
+                'sec-gpc': '1',
+                'origin': 'https://twitter.com',
+                'sec-fetch-site': 'same-site',
+                'sec-fetch-mode': 'cors',
+                'sec-fetch-dest': 'empty',
+                'referer': 'https://twitter.com/',
+                'accept-language': 'en-US',
+            }
+            self._session.headers.update(headers)
+            req = self._session.prepare_request(requests.Request('POST', 'https://api.twitter.com/1.1/guest/activate.json'))
+            res = self._session.send(req, allow_redirects=True, timeout=self._timeout)
+            match = re.search(r'{"guest_token":"(\d+)"}', res.text)
+            if match:
+                logme.debug('Found guest token in JSON')
+                self.config.Guest_token = str(match.group(1))
+            else:
+                self.config.Guest_token = None
+                raise RefreshTokenException('Could not find the Guest token in JSON')

I don't understand session of requests, so the code may be not good. I hope someone rewrite the patch better and create a pull request.

About the problem

In my environment, this problem has recently begun to occur. This doesn't happen every time, so if you are lucky, you don't get the error.

The cause is literally literally that twint could not find the Guest token in HTML. Actually, sometimes token isn't included in HTML recently.

#!/usr/bin/env python3
# This program is WTFPL.
import requests

res = requests.get('https://twitter.com')
print(res.text.split('\n')[-1])

twint require the result of running the above code is })();</script><script nonce="VALUE">document.cookie = decodeURIComponent("gt=VALUE; Max-Age=VALUE; Domain=.twitter.com; Path=/; Secure");</script>. However, sometimes the result is only })();</script> and missing the Guest token.

About the solution

In my patch, twint get the Guest token from https://api.twitter.com/1.1/guest/activate.json if could not find the one. I referred to the code of gallery-dl.

These are the steps I took to install twint and get try and get it to work on 2 versions of Linux and 1 Deb. I have spent 9 hours trying my best, but no fix anywhere, none at all, not one.

Unfortunately, twint hasn't been updated for a long time. If the problem is new, updating will not solve the problem.

In this case, the problem is too new to find a solution. However, the problem may not happen in other environments, so I want reports.

Happy New Year to you minamotorin and thank you for explaining what was happening.

Happy New Year to everyone also!

I left this problem and worked on another project only coming back now to have another go at installing.

As a complete noob, I understand that the token.py file is the one to edit. I am just not sure what to do next.

Q: Is this the code I have to put into the token.py file?

#!/usr/bin/env python3

This program is WTFPL.

import requests

res = requests.get('https://twitter.com')
print(res.text.split('\n')[-1])

Q: Or, do I copy and paste the code long code above into the token.py?

I will report back with my findings.

This post will help many I believe and hope.

Thanks in advance.

Ranger.