Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Raiden crashes when Matrix server is changed #3080

Closed
rakanalh opened this issue Nov 27, 2018 · 6 comments
Closed

Raiden crashes when Matrix server is changed #3080

rakanalh opened this issue Nov 27, 2018 · 6 comments
Labels
Component / Transport Transport related issues

Comments

@rakanalh
Copy link
Contributor

I mistakingly ran Raiden with production config and explicity passed a flag to use transport01.

Command:

/home/rakan/.pyenv/versions/raiden/bin/raiden --keystore-path /home/rakan/.local/share/io.parity.ethereum/keys/DevelopmentChain/ --log-config raiden:DEBUG --api-address localhost:5502 --eth-rpc-endpoint http://localhost:8545 --accept-disclaimer --network-id 17 --environment-type production --matrix-server=https://transport01.raiden.network --no-sync-check --tokennetwork-registry-contract-address=0x7098b0393aDC815B42a046628AB5fFa378Eb7cA4 --secret-registry-contract-address=0x21a50f9D0c9B2709dfaD8da91344E90943FfD83E --endpoint-registry-contract-address=0xe0fb6C6119ee7777E7208A9fDF2B85Ae97BF3fdb

When i removed the --matrix-server flag, Raiden crashed with the following error:

2018-11-27 11:47:49 [info     ] Starting Raiden                [raiden.ui.runners] distribution=source python_implementation=CPython python_version=3.6.5 raiden=0.16.1.dev99+gfc6028e1 system=Linux 64bit_ELF 4.19.2-arch1-1-ARCH x86_64
The following accounts were found in your machine:

[  0] - 0xF6276982911a8E4Df05440382704bD2aA8dB2624
[  1] - 0x3f0Cb919344DA6536fAAC3fa6E39aF482670FbC6
[  2] - 0x9734BC77a2717aC357D97401c99E4c14b87Fa9d2
[  3] - 0xccAA48B8A965b31aAaFe5463E6BdF1ae5f249C76
[  4] - 0x00a329c0648769A73afAc7F9381E08FB43dBEA72
[  5] - 0xbeca471569B63c1C3Cd1C91FcC3fc082F865e25B

Select one of them by index to continue: 0
Enter the password to unlock 0xF6276982911a8E4Df05440382704bD2aA8dB2624:
2018-11-27 11:47:51 [debug    ] JSONRPCClient created          [raiden.network.rpc.client] available_nonce=3 client=Parity-Ethereum//v2.1.6-stable-491f17f149-20181114/x86_64-linux-gnu/rustc1.30.1 node=f6276982
Raiden is running in production mode

You are connected to the '17' network and the DB path is: /home/rakan/.raiden/node_f6276982/netid_17/network_7098b039/v14_log.db
2018-11-27 11:47:51 [debug    ] Fetching available matrix servers [raiden.ui.app] available_servers=['https://raidentransport.exchangeunion.com', 'https://persephone.raidentransport.digitalvirtues.com', 'https://raidentransport.ki-decentralized.com', 'https://raidentransport.mycryptoapi.com']
2018-11-27 11:47:55 [debug    ] Matrix homeserver RTT times    [raiden.network.transport.matrix] rtt_times=[('https://persephone.raidentransport.digitalvirtues.com', 0.03507766666666667), ('https://raidentransport.ki-decentralized.com', 0.03543166666666667), ('https://raidentransport.mycryptoapi.com', 0.111642), ('https://raidentransport.exchangeunion.com', 0.12437866666666668)]
2018-11-27 11:47:55 [info     ] Automatically selecting matrix homeserver based on RTT [raiden.network.transport.matrix] homeserver=https://persephone.raidentransport.digitalvirtues.com rtt=0.03507766666666667
2018-11-27 11:47:55 [debug    ] Restoring from snapshot        [raiden.storage.wal] from_state_change_id=2000 to_state_change_id=latest
2018-11-27 11:47:55 [debug    ] Replaying state changes        [raiden.storage.wal] num_state_changes=84
2018-11-27 11:47:55 [debug    ] Restored state from WAL        [raiden.raiden_service] last_restored_block=2286 node=f6276982
2018-11-27 11:47:55 [debug    ] Alarm task first run           [raiden.tasks] known_block_number=2286 latest_block_hash=0x180ef5c74fb0bc75317608ce5cd0d4af918b4f2bb6c9aa2edc092622cb615300 latest_block_number=2337 latest_gas_limit=8000000
2018-11-27 11:47:55 [debug    ] Received new block             [raiden.tasks] known_block_number=2286 latest_block_gas_limit=8000000 latest_block_hash=0x180ef5c74fb0bc75317608ce5cd0d4af918b4f2bb6c9aa2edc092622cb615300 latest_block_number=2337 num_missed_blocks=50
2018-11-27 11:47:55 [debug    ] State change                   [raiden.raiden_service] node=f6276982 state_change={"block_number": "2332", "gas_limit": 8000000, "block_hash": "0xfe875d488a028f7a1c0c202d1fe6532f6934d787b0799c4a2d68560249e4a091", "_type": "raiden.transfer.state_change.Block", "_version": 0}
2018-11-27 11:47:55 [debug    ] Processing pending transactions [raiden.raiden_service] node=f6276982 num_pending_transactions=0
2018-11-27 11:47:56 [debug    ] Login                          [raiden.network.transport.matrix] homeserver=persephone.raidentransport.digitalvirtues.com server_url=https://persephone.raidentransport.digitalvirtues.com username=0xf6276982911a8e4df05440382704bd2aa8db2624
2018-11-27 11:47:56 [debug    ] Could not join discovery room  [raiden.network.transport.matrix] _exception=MatrixRequestError('404: {"errcode":"M_NOT_FOUND","error":"Room alias #raiden_17_discovery:persephone.raidentransport.digitalvirtues.com not found"}',) current_user=@0xf6276982911a8e4df05440382704bd2aa8db2624:persephone.raidentransport.digitalvirtues.com node=f6276982 room_alias_full=#raiden_17_discovery:persephone.raidentransport.digitalvirtues.com
Traceback (most recent call last):
  File "/home/rakan/.pyenv/versions/raiden/bin/raiden", line 11, in <module>
    load_entry_point('raiden', 'console_scripts', 'raiden')()
  File "/home/rakan/Brainbot/raiden/raiden/__main__.py", line 11, in main
    run(auto_envvar_prefix='RAIDEN')  # pylint: disable=no-value-for-parameter
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/click/core.py", line 1043, in invoke
    return Command.invoke(self, ctx)
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/rakan/Brainbot/raiden/raiden/ui/cli.py", line 425, in run
    app = runner.run()
  File "/home/rakan/Brainbot/raiden/raiden/ui/runners.py", line 247, in run
    return self._start_services()
  File "/home/rakan/Brainbot/raiden/raiden/ui/runners.py", line 96, in _start_services
    app_ = run_app(**self._options)
  File "/home/rakan/Brainbot/raiden/raiden/ui/app.py", line 418, in run_app
    raiden_app.start()
  File "/home/rakan/Brainbot/raiden/raiden/app.py", line 152, in start
    self.raiden.start()
  File "/home/rakan/Brainbot/raiden/raiden/raiden_service.py", line 380, in start
    self.transport.start(self, self.message_handler)
  File "/home/rakan/Brainbot/raiden/raiden/network/transport/matrix.py", line 349, in start
    self._join_discovery_room()
  File "/home/rakan/Brainbot/raiden/raiden/network/transport/matrix.py", line 574, in _join_discovery_room
    discovery_room = self._client.join_room(discovery_room_alias_full)
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/matrix_client/client.py", line 320, in join_room
    response = self.api.join_room(room_id_or_alias)
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/matrix_client/api.py", line 195, in join_room
    return self._send("POST", path)
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/raiden_libs/network/matrix/client.py", line 75, in _send
    return super()._send(method, path, *args, **kwargs)
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/matrix_client/api.py", line 691, in _send
    code=response.status_code, content=response.text
matrix_client.errors.MatrixRequestError: 500: {"errcode":"M_UNKNOWN","error":"Internal server error"}
--- Logging error ---
Exception ignored in: <bound method BaseFileLock.__del__ of <filelock.UnixFileLock object at 0x7fd27cb60d30>>
Traceback (most recent call last):
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/filelock.py", line 333, in __del__
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/filelock.py", line 320, in release
  File "/home/rakan/.pyenv/versions/3.6.5/lib/python3.6/logging/__init__.py", line 1306, in info
  File "/home/rakan/.pyenv/versions/3.6.5/lib/python3.6/logging/__init__.py", line 1442, in _log
  File "/home/rakan/.pyenv/versions/3.6.5/lib/python3.6/logging/__init__.py", line 1452, in handle
  File "/home/rakan/.pyenv/versions/3.6.5/lib/python3.6/logging/__init__.py", line 1514, in callHandlers
  File "/home/rakan/.pyenv/versions/3.6.5/lib/python3.6/logging/__init__.py", line 863, in handle
  File "/home/rakan/.pyenv/versions/3.6.5/lib/python3.6/logging/handlers.py", line 75, in emit
  File "/home/rakan/.pyenv/versions/3.6.5/lib/python3.6/logging/__init__.py", line 916, in handleError
  File "/home/rakan/.pyenv/versions/3.6.5/lib/python3.6/traceback.py", line 100, in print_exception
  File "/home/rakan/Brainbot/raiden/raiden/log_config.py", line 126, in tracebackexception_format
  File "/home/rakan/Brainbot/raiden/raiden/log_config.py", line 111, in processor_wrapper
ImportError: sys.meta_path is None, Python is likely shutting down

This is due to the fact that Matrix transport was looking to join a room for a specific channel and the room did not exist on the selected production server as the channel's room was created on transport01.

It would have been nice if Raiden displays a meaningful error message and stop.
@LefterisJP
Copy link
Contributor

This sounds very similar to what this gitter user experienced

@rakanalh rakanalh added the Component / Transport Transport related issues label Nov 27, 2018
@ulope
Copy link
Collaborator

ulope commented Nov 28, 2018

I received matrix log files from KI-Decentralized. There are exceptions for the relevant times trying to federate with the Exchange Union server.

I've created issues in raiden-transport to provide us better insight into the state of the servers:

Additionally we should improve the handling of such failures in the matrix transport in Raiden itself.

@ulope ulope added the P2 label Jan 14, 2019
@andrevmatos
Copy link
Contributor

Probably fixed by raiden-transport changes and #3180 on client side

@LefterisJP
Copy link
Contributor

Shouldn't this be easy to test? I would rather have a test to say that this is fixed or not.

@rakanalh rakanalh added Type / Bug and removed P2 labels Oct 28, 2019
@Dominik1999 Dominik1999 added this to the Alderaan milestone Nov 21, 2019
@rakanalh rakanalh changed the title Matrix crashes Raiden when server is changed Raiden crashes Matrix when server is changed Nov 28, 2019
@Dominik1999
Copy link
Contributor

  • just handle in exception and close this!

@Dominik1999 Dominik1999 removed this from the Alderaan milestone Dec 2, 2019
@Dominik1999 Dominik1999 changed the title Raiden crashes Matrix when server is changed Raiden crashes when Matrix server is changed Dec 6, 2019
@ulope
Copy link
Collaborator

ulope commented Jan 13, 2020 •
edited
Loading

We have changed a lot of the code this issue referenced. Closing.

@ulope ulope closed this as completed Jan 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Component / Transport Transport related issues
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ulope @rakanalh @andrevmatos @LefterisJP @Dominik1999