Service

[MartinKei]

Issue Description

Service stopped working but not running when facing Kraken API issues. Process is still "running" but doesn't do anything. (eg. executing triggers, sending Telegram notifs, etc.) Due to the process still running, systemd isn't able to restart it.

Expectation

As said above, I'd be happy with either of:

• Process recovers and retries later
• Process dies completely and stops so systemd can restart it

System

Raspberry Pi 3 B with Raspbian
Using systemd

Exception

Apr 24 11:17:10 raffer vigilant-cqrypto-snatch[483]: 2022-04-24 11:17:10 raffer vigilant_crypto_snatch[483] INFO Starting up with version 5.5.0 \u2026
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]: Traceback (most recent call last):
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 169, in _new_conn
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     conn = connection.create_connection(
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 73, in create_connection
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/usr/lib/python3.9/socket.py", line 953, in getaddrinfo
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]: socket.gaierror: [Errno -3] Temporary failure in name resolution
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]: During handling of the above exception, another exception occurred:
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]: Traceback (most recent call last):
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     httplib_response = self._make_request(
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 382, in _make_request
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     self._validate_conn(conn)
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1012, in _validate_conn
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     conn.connect()
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 353, in connect
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     conn = self._new_conn()
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 181, in _new_conn
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     raise NewConnectionError(
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]: urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x74957a78>: Failed to establish a new connection: [Errno -3] Temporar>
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]: During handling of the above exception, another exception occurred:
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]: Traceback (most recent call last):
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     resp = conn.urlopen(
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     retries = retries.increment(
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 574, in increment
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     raise MaxRetryError(_pool, url, error or ResponseError(cause))
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.kraken.com', port=443): Max retries exceeded with url: /0/private/Balance (Caused by NewCon>
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]: During handling of the above exception, another exception occurred:
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]: Traceback (most recent call last):
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/home/user/.local/lib/python3.9/site-packages/vigilant_crypto_snatch/marketplace/krakenex_adaptor.py", line 93, in get_balance
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     answer = self.handle.query_private("Balance")
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/usr/local/lib/python3.9/dist-packages/krakenex/api.py", line 193, in query_private
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     return self._query(urlpath, data, headers, timeout = timeout)
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/usr/local/lib/python3.9/dist-packages/krakenex/api.py", line 134, in _query
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     self.response = self.session.post(url, data = data, headers = headers,
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590, in post
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     return self.request('POST', url, data=data, json=json, **kwargs)
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     resp = self.send(prep, **send_kwargs)
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     r = adapter.send(request, **kwargs)
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     raise ConnectionError(e, request=request)
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.kraken.com', port=443): Max retries exceeded with url: /0/private/Balance (Caused by New>
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]: The above exception was the direct cause of the following exception:
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]: Traceback (most recent call last):
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/usr/local/bin/vigilant-crypto-snatch", line 8, in <module>
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     sys.exit(main())
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/home/user/.local/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     return self.main(*args, **kwargs)
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/home/user/.local/lib/python3.9/site-packages/click/core.py", line 1053, in main
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     rv = self.invoke(ctx)
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/home/user/.local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     return _process_result(sub_ctx.command.invoke(sub_ctx))
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/home/user/.local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     return ctx.invoke(self.callback, **ctx.params)
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/home/user/.local/lib/python3.9/site-packages/click/core.py", line 754, in invoke
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     return __callback(*args, **kwargs)
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/home/user/.local/lib/python3.9/site-packages/vigilant_crypto_snatch/cli.py", line 34, in watch
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     watch.main()
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/home/user/.local/lib/python3.9/site-packages/vigilant_crypto_snatch/commands/watch.py", line 31, in main
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     check_and_perform_widthdrawal(market)
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/home/user/.local/lib/python3.9/site-packages/vigilant_crypto_snatch/marketplace/interface.py", line 64, in check_and_perform_widthdrawal
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     balance = market.get_balance()
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:   File "/home/user/.local/lib/python3.9/site-packages/vigilant_crypto_snatch/marketplace/krakenex_adaptor.py", line 95, in get_balance
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]:     raise HttpRequestError("Connection error in Kraken Balance") from e
Apr 24 11:17:10 raffer vigilant-crypto-snatch[483]: vigilant_crypto_snatch.myrequests.HttpRequestError: Connection error in Kraken Balance

[martin-ueding]

I don't really see what we could do about that. The program catches this exception and ignores it. And they keep on coming?

[MartinKei]

@martin-ueding the issue isn't about the exception keep on coming, but that the application stopped checking triggers.

It just kept running without doing anything.

No "I am still here!" messages, no triggers execution, nothing.

My expectation is that the app either exits with an error, or continue on with it's inner lifecycle.

[martin-ueding]

Yes, that indeed sounds like a very strange bug. If that happens again, could you perhaps use sudo py-spy dump --pid PID with the process ID of the program to see where it is stuck?