Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TypeError: the JSON object must be str, not 'bytes' in import_logs.py #286

Open
1000i100 opened this issue Nov 7, 2020 · 4 comments · May be fixed by #302
Open

TypeError: the JSON object must be str, not 'bytes' in import_logs.py #286

1000i100 opened this issue Nov 7, 2020 · 4 comments · May be fixed by #302

Comments

@1000i100
Copy link

1000i100 commented Nov 7, 2020

When i run ./misc/log-analytics/import_logs.py following https://github.com/matomo-org/matomo-log-analytics#how-do-i-setup-nginx-to-directly-import-to-matomo-via-syslog
( and after update like describe here : #285 )

Here is what i get :

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "******/misc/log-analytics/import_logs.py", line 1849, in _run_bulk
    self._record_hits(hits)
  File ""******/misc/log-analytics/import_logs.py", line 1995, in _record_hits
    'requests': [self._get_hit_args(hit) for hit in hits]
  File ""******/misc/log-analytics/import_logs.py", line 1995, in <listcomp>
    'requests': [self._get_hit_args(hit) for hit in hits]
  File ""******/misc/log-analytics/import_logs.py", line 1891, in _get_hit_args
    site_id, main_url = resolver.resolve(hit)
  File ""******/misc/log-analytics/import_logs.py", line 1775, in resolve
    return self._resolve_by_host(hit)
  File ""******/misc/log-analytics/import_logs.py", line 1758, in _resolve_by_host
    site_id = self._resolve(hit)
  File ""******/misc/log-analytics/import_logs.py", line 1726, in _resolve
    res = self._get_site_id_from_hit_host(hit)
  File ""******/misc/log-analytics/import_logs.py", line 1681, in _get_site_id_from_hit_host
    url=hit.host,
  File ""******/misc/log-analytics/import_logs.py", line 1633, in call_api
    return self._call_wrapper(self._call_api, None, None, method, **kwargs)
  File ""******/misc/log-analytics/import_logs.py", line 1585, in _call_wrapper
    response = func(*args, **kwargs)
  File ""******/misc/log-analytics/import_logs.py", line 1574, in _call_api
    return json.loads(res)
  File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
    s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'

1365 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
[... Ctrl+C to stop it ...]
1365 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)

Logs import summary
-------------------

    0 requests imported successfully
    17 requests were downloads
    0 requests ignored:
        0 HTTP errors
        0 HTTP redirects
        0 invalid log lines
        0 filtered log lines
        0 requests did not match any known site
        0 requests did not match any --hostname
        0 requests done by bots, search engines...
        0 requests to static resources (css, js, images, ico, ttf...)
        0 requests to file downloads did not match any --download-extensions

Website import summary
----------------------

    0 requests imported to 0 sites
        0 sites already existed
        0 sites were created:

    0 distinct hostnames did not match any existing site:



Performance summary
-------------------

    Total time: 149 seconds
    Requests imported per second: 0.0 requests per second

Processing your log data
------------------------

    In order for your logs to be processed by Matomo, you may need to run the following command:
     ./console core:archive --force-all-websites --force-all-periods=315576000 --force-date-last-n=1000 --url='****'

And it's the same either the way i run it :

cat /var/tmp/nginx.tmp | ./misc/log-analytics/import_logs.py [...] -

./misc/log-analytics/import_logs.py [...] /var/tmp/nginx.tmp
@kstka
Copy link

kstka commented Dec 8, 2020

In my case moving to Python 3.8.5 (anaconda) solved this issue

@Me-okane
Copy link

Me-okane commented Dec 8, 2020

for Python3.5 there is a missing "decode"
Line 1573:
return json.loads(res)

should be:
return json.loads(res.decode("utf-8"))

@1000i100
Copy link
Author

1000i100 commented Dec 8, 2020

I've updated to python3.9 and edit ./misc/log-analytics/import_logs.py to specify in first line python3.9 and it fix it for me.

@digitalshow digitalshow linked a pull request Jan 19, 2021 that will close this issue
Open
@fwolfst
Copy link
Contributor

fwolfst commented Feb 22, 2021

for Python3.5 there is a missing "decode"
Line 1573:
return json.loads(res)

should be:
return json.loads(res.decode("utf-8"))

Thanks, helped my python3.5 setup .

Probably one more at ll 2015:

                # check for invalid requests
                try:
                    response = json.loads(response.decode("utf-8"))
                except:
                    logging.info("bulk tracking returned invalid JSON")

At least that helped in my case.

@tups tups mentioned this issue Jun 25, 2021
Merged
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Compatibility for Python3.5... digitalshow/matomo-log-analytics
4 participants
@fwolfst @1000i100 @kstka @Me-okane