Add Electrum backend #1222

jp1ac4 · 2024-08-08T08:53:09Z

This is to add the Electrum backend as part of #56 (comment).

This requires the database to be running version 5 following #1180. The migration from a previous DB version must be done using bitcoind. Thereafter, the daemon can be run using an Electrum backend by replacing the [bitcoind_config] section in the daemon.toml config file with:

[electrum_config]
addr = "127.0.0.1:50001" # adjust IP:port as required

Remaining tasks:

Include ancestors and descendants when getting a transaction's MempoolEntry
Additional sanity checks, e.g. check Electrum version
Check if logic regarding ongoing rescan needs to be adjusted
Add Electrum backend to functional tests
Add Electrum backend to CI pipline functional tests
Add unit tests
Update README & other docs

src/bitcoin/electrum/mod.rs

src/bitcoin/electrum/wallet.rs

src/config.rs

darosior

Just been through the commits without reading the details. Thanks for neatly organizing the PR, it's a pleasure to read through.

Just a few drive-by comments, nothing major.

src/lib.rs

src/bitcoin/poller/looper.rs

Cargo.lock

src/bitcoin/electrum/mod.rs

src/bitcoin/mod.rs

src/config.rs

src/database/mod.rs

jp1ac4 · 2024-08-13T16:23:12Z

I've updated the mempool_entry logic to take account of ancestors and descendants.

jp1ac4 · 2024-08-21T16:41:30Z

I've made various fixes after running the functional tests locally:

Although we were handling our own RBF transactions correctly, we weren't detecting unconfirmed transactions that had dropped out of the mempool for some other reason. I now keep track of unconfirmed transactions in each sync in order to identify those in the graph that have expired and to exclude the corresponding coins and spend transactions.
When getting the mempool entry, I now ensure we have all previous txouts for descendants. It was possible before that a descendant was spending the output from a confirmed transaction, which we wouldn't have had in the graph.
I simplified the logic for finding the common ancestor in case of a reorg and ensure this common ancestor is no higher than the current tip. Before it was possible that the local chain had advanced and we would find a common ancestor later than the DB, which meant we would not update the DB with all changes.
I added a rollback_wallet_tip method to the Bitcoin interface to roll back the local chain's tip in case of a reorg. This helps when syncing the wallet to detect reorgs (see below). I considered adding this logic to the sync_wallet method, but for a few reasons it was simpler and more effective to be in its own method.
It was possible when syncing the wallet that a reorg had happened since getting the new tip, which would then cause the local chain to get out of sync with the DB. Now when syncing the wallet, I ensure that the updates don't involve a reorg, and if so, I don't update anything => the poller will detect this change and restart a bit later on.

I'm now preparing the updates to functional tests so that they use Electrum as a backend and will push these changes when ready.

Check if logic regarding ongoing rescan needs to be adjusted

As discussed in Discord, I don't think we need the rescan functionality when using the Electrum backend given that a full scan is performed at wallet creation (for both new & imported wallets).

The startrescan command will return an error:

{
  "error": {
    "code": -32603,
    "message": "Error while starting rescan: 'rescan is not required for electrum'"
  }
}

jp1ac4 · 2024-08-22T12:29:01Z

I've made a further fix to how unconfirmed transactions are handled. It was possible before, especially when running the functional tests, that two polls would be done within the same second and so setting the last seen to the current timestamp would not identify those unconfirmed transactions that had dropped from the mempool. I've now added a sync_count and use this instead as the last seen value, which is guaranteed to strictly increase for successive polls.

I also fixed an error in how the wallet's coins were returned in case a coin's spend txid had dropped from the mempool. It was previously dropping the whole coin whereas now I only drop the spend txid value and keep the coin.

I've updated the functional tests to work with Electrum (specifically electrs).

The next step is to add this electrs backend to the CI pipeline.

jp1ac4 · 2024-08-22T14:47:50Z

I've now added the Electrum backend to the CI pipeline. There's a failing test but I think it's just a transient error.

tests/test_misc.py

jp1ac4 · 2024-08-23T09:45:18Z

Unrelated to the Electrum changes, I added the test_spend.py tests to the CI as I don't think there was any reason for them not to be included.

The failing test is passing on my local machine and seems to be a transient error.

jp1ac4 · 2024-08-26T13:54:28Z

Unrelated to the Electrum changes, I added the test_spend.py tests to the CI as I don't think there was any reason for them not to be included.

This change will be added as part of #1235 instead.

src/lib.rs

jp1ac4 · 2024-08-27T13:42:10Z

Rebased on #1235.

This includes changes from darosior's comment in wizardsardine#1222 (comment).

This is copied from darosior's changes in wizardsardine#1222 (comment).

jp1ac4 · 2024-09-04T11:10:40Z

Rebased on master and applied @darosior's rescan logic.

darosior · 2024-09-04T11:34:27Z

@jp1ac4 to fix the functional tests:

diff --git a/tests/test_framework/electrs.py b/tests/test_framework/electrs.py
index 1dc12d3..ee8c6a5 100644
--- a/tests/test_framework/electrs.py
+++ b/tests/test_framework/electrs.py
@@ -19,6 +19,10 @@ class Electrs(BitcoinBackend):
         if rpcport is None:
             rpcport = reserve()
 
+        # Prometheus metrics can't be deactivated in Electrs. Configure the port so it doesn't
+        # conflict with other instances when running tests in parallel.
+        monitoring_port = reserve()
+
         self.electrs_dir = electrs_dir
         self.rpcport = rpcport
 
@@ -39,6 +43,7 @@ class Electrs(BitcoinBackend):
             "db_dir": electrs_dir,
             "network": "regtest",
             "electrum_rpc_addr": f"127.0.0.1:{self.rpcport}",
+            "monitoring_addr": f"127.0.0.1:{monitoring_port}",
         }
         self.conf_file = os.path.join(regtestdir, "electrs.toml")
         with open(self.conf_file, "w") as f:

h/t @pythcoiner for pointing to me it's what conflicted between instances.

This is copied from darosior's changes in wizardsardine#1222 (comment).

jp1ac4 · 2024-09-04T19:29:35Z

@jp1ac4 to fix the functional tests:

Applied these changes and another small fix to do with checking the backend type.

There's an issue with electrs returning an error when a channel it's sending on becomes disconnected (which happens in some tests when we stop lianad). To make the tests pass, we could restart the electrs backend in that case, but this wouldn't address the underlying issue.

pythcoiner · 2024-09-05T05:53:11Z

run only tests/test_chain.py::test_rescan_and_recovery it fail 20% of time on:

DEBUG    root:utils.py:253 Calling stop with params {}
ERROR    root:lianad.py:135 lianad : error when calling stop: '[Errno 2] No such file or directory'

Full logs:

~/rust/liana pr1222 *32 !2 ❯ BITCOIN_BACKEND_TYPE=electrs pytest -vvvv --log-cli-level=DEBUG tests/test_chain.py::test_rescan_and_recovery
=========================================================== test session starts ============================================================
platform linux -- Python 3.12.4, pytest-8.3.2, pluggy-1.5.0 -- /home/pyth/rust/liana/tests/venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/pyth/rust/liana
plugins: forked-1.6.0, timeout-2.3.1, xdist-1.31.0
collected 1 item                                                                                                                           

tests/test_chain.py::test_rescan_and_recovery 
-------------------------------------------------------------- live log setup --------------------------------------------------------------
DEBUG    root:utils.py:316 Starting 'bitcoind -datadir=/tmp/lianad-tests-3w2macqn/test_rescan_and_recovery_1/bitcoind -printtoconsole -server -debug=1 -debugexclude=libevent -debugexclude=tor'
DEBUG    root:utils.py:403 Waiting for ['Done loading'] in the logs
DEBUG    root:utils.py:427 Found 're.compile('Done loading')' in logs
INFO     root:bitcoind.py:94 Bitcoind started
DEBUG    root:utils.py:316 Starting 'electrs --conf /tmp/lianad-tests-3w2macqn/test_rescan_and_recovery_1/electrs/regtest/electrs.toml'
INFO     root:electrs.py:57 Electrs started
DEBUG    root:utils.py:316 Starting '/home/pyth/rust/liana/tests/test_framework/../../target/debug/lianad --conf /tmp/lianad-tests-3w2macqn/test_rescan_and_recovery_1/lianad/config.toml'
DEBUG    root:utils.py:403 Waiting for ['Database initialized and checked', 'JSONRPC server started.'] in the logs
DEBUG    root:utils.py:427 Found 're.compile('Database initialized and checked')' in logs
DEBUG    root:utils.py:427 Found 're.compile('JSONRPC server started.')' in logs
-------------------------------------------------------------- live log call ---------------------------------------------------------------
DEBUG    root:utils.py:253 Calling getnewaddress with params {}
DEBUG    root:utils.py:269 Received response for getnewaddress call: {'jsonrpc': '2.0', 'result': {'address': 'bcrt1q2700qx72qpm8a8r07srmx5v6280jt8tuauz8mjypnfhe4ehxx6hqq8gxnt', 'derivation_index': 0}, 'id': 0}
DEBUG    root:utils.py:253 Calling getinfo with params {}
DEBUG    root:utils.py:269 Received response for getinfo call: {'jsonrpc': '2.0', 'result': {'block_height': 0, 'descriptors': {'main': {'change_desc': 'wsh(or_d(pk([85d83c29]tpubD6NzVbkrYhZ4X71sRFatBukgznQTazD3toBTfthzLvmUVYw2z1jS6gzvJaeR7i8dndaxD7f9ZRWT4zsC56Nk8vXG8XA4LKPgrro8q574YGz/1/*),and_v(v:pkh([f0f444dc]tpubD6NzVbkrYhZ4YcSPGiAxVUqd6a7Q1MjEd6d6oYcTcCnj85gNPDVFkxaq1HcRVnzHu6umLSsRDM1qWzayhYsiHLwgsSCcc8dZQosGsDmYoXs/1/*),older(10))))#euzgfe24', 'multi_desc': 'wsh(or_d(pk([85d83c29]tpubD6NzVbkrYhZ4X71sRFatBukgznQTazD3toBTfthzLvmUVYw2z1jS6gzvJaeR7i8dndaxD7f9ZRWT4zsC56Nk8vXG8XA4LKPgrro8q574YGz/<0;1>/*),and_v(v:pkh([f0f444dc]tpubD6NzVbkrYhZ4YcSPGiAxVUqd6a7Q1MjEd6d6oYcTcCnj85gNPDVFkxaq1HcRVnzHu6umLSsRDM1qWzayhYsiHLwgsSCcc8dZQosGsDmYoXs/<0;1>/*),older(10))))#ecysgm34', 'receive_desc': 'wsh(or_d(pk([85d83c29]tpubD6NzVbkrYhZ4X71sRFatBukgznQTazD3toBTfthzLvmUVYw2z1jS6gzvJaeR7i8dndaxD7f9ZRWT4zsC56Nk8vXG8XA4LKPgrro8q574YGz/0/*),and_v(v:pkh([f0f444dc]tpubD6NzVbkrYhZ4YcSPGiAxVUqd6a7Q1MjEd6d6oYcTcCnj85gNPDVFkxaq1HcRVnzHu6umLSsRDM1qWzayhYsiHLwgsSCcc8dZQosGsDmYoXs/0/*),older(10))))#lku8043e'}}, 'network': 'regtest', 'rescan_progress': None, 'sync': 1.0, 'timestamp': 1725515069, 'version': '6.0.0-dev'}, 'id': 0}
DEBUG    root:utils.py:253 Calling getinfo with params {}
DEBUG    root:utils.py:269 Received response for getinfo call: {'jsonrpc': '2.0', 'result': {'block_height': 101, 'descriptors': {'main': {'change_desc': 'wsh(or_d(pk([85d83c29]tpubD6NzVbkrYhZ4X71sRFatBukgznQTazD3toBTfthzLvmUVYw2z1jS6gzvJaeR7i8dndaxD7f9ZRWT4zsC56Nk8vXG8XA4LKPgrro8q574YGz/1/*),and_v(v:pkh([f0f444dc]tpubD6NzVbkrYhZ4YcSPGiAxVUqd6a7Q1MjEd6d6oYcTcCnj85gNPDVFkxaq1HcRVnzHu6umLSsRDM1qWzayhYsiHLwgsSCcc8dZQosGsDmYoXs/1/*),older(10))))#euzgfe24', 'multi_desc': 'wsh(or_d(pk([85d83c29]tpubD6NzVbkrYhZ4X71sRFatBukgznQTazD3toBTfthzLvmUVYw2z1jS6gzvJaeR7i8dndaxD7f9ZRWT4zsC56Nk8vXG8XA4LKPgrro8q574YGz/<0;1>/*),and_v(v:pkh([f0f444dc]tpubD6NzVbkrYhZ4YcSPGiAxVUqd6a7Q1MjEd6d6oYcTcCnj85gNPDVFkxaq1HcRVnzHu6umLSsRDM1qWzayhYsiHLwgsSCcc8dZQosGsDmYoXs/<0;1>/*),older(10))))#ecysgm34', 'receive_desc': 'wsh(or_d(pk([85d83c29]tpubD6NzVbkrYhZ4X71sRFatBukgznQTazD3toBTfthzLvmUVYw2z1jS6gzvJaeR7i8dndaxD7f9ZRWT4zsC56Nk8vXG8XA4LKPgrro8q574YGz/0/*),and_v(v:pkh([f0f444dc]tpubD6NzVbkrYhZ4YcSPGiAxVUqd6a7Q1MjEd6d6oYcTcCnj85gNPDVFkxaq1HcRVnzHu6umLSsRDM1qWzayhYsiHLwgsSCcc8dZQosGsDmYoXs/0/*),older(10))))#lku8043e'}}, 'network': 'regtest', 'rescan_progress': None, 'sync': 1.0, 'timestamp': 1725515069, 'version': '6.0.0-dev'}, 'id': 0}
DEBUG    root:utils.py:253 Calling getinfo with params {}
DEBUG    root:utils.py:269 Received response for getinfo call: {'jsonrpc': '2.0', 'result': {'block_height': 102, 'descriptors': {'main': {'change_desc': 'wsh(or_d(pk([85d83c29]tpubD6NzVbkrYhZ4X71sRFatBukgznQTazD3toBTfthzLvmUVYw2z1jS6gzvJaeR7i8dndaxD7f9ZRWT4zsC56Nk8vXG8XA4LKPgrro8q574YGz/1/*),and_v(v:pkh([f0f444dc]tpubD6NzVbkrYhZ4YcSPGiAxVUqd6a7Q1MjEd6d6oYcTcCnj85gNPDVFkxaq1HcRVnzHu6umLSsRDM1qWzayhYsiHLwgsSCcc8dZQosGsDmYoXs/1/*),older(10))))#euzgfe24', 'multi_desc': 'wsh(or_d(pk([85d83c29]tpubD6NzVbkrYhZ4X71sRFatBukgznQTazD3toBTfthzLvmUVYw2z1jS6gzvJaeR7i8dndaxD7f9ZRWT4zsC56Nk8vXG8XA4LKPgrro8q574YGz/<0;1>/*),and_v(v:pkh([f0f444dc]tpubD6NzVbkrYhZ4YcSPGiAxVUqd6a7Q1MjEd6d6oYcTcCnj85gNPDVFkxaq1HcRVnzHu6umLSsRDM1qWzayhYsiHLwgsSCcc8dZQosGsDmYoXs/<0;1>/*),older(10))))#ecysgm34', 'receive_desc': 'wsh(or_d(pk([85d83c29]tpubD6NzVbkrYhZ4X71sRFatBukgznQTazD3toBTfthzLvmUVYw2z1jS6gzvJaeR7i8dndaxD7f9ZRWT4zsC56Nk8vXG8XA4LKPgrro8q574YGz/0/*),and_v(v:pkh([f0f444dc]tpubD6NzVbkrYhZ4YcSPGiAxVUqd6a7Q1MjEd6d6oYcTcCnj85gNPDVFkxaq1HcRVnzHu6umLSsRDM1qWzayhYsiHLwgsSCcc8dZQosGsDmYoXs/0/*),older(10))))#lku8043e'}}, 'network': 'regtest', 'rescan_progress': None, 'sync': 1.0, 'timestamp': 1725515069, 'version': '6.0.0-dev'}, 'id': 0}
DEBUG    root:utils.py:253 Calling listcoins with params {}
DEBUG    root:utils.py:269 Received response for listcoins call: {'jsonrpc': '2.0', 'result': {'coins': [{'address': 'bcrt1q2700qx72qpm8a8r07srmx5v6280jt8tuauz8mjypnfhe4ehxx6hqq8gxnt', 'amount': 50000000, 'block_height': 102, 'derivation_index': 0, 'is_change': False, 'is_immature': False, 'outpoint': '0ca339c609dc9d03a1fdd2d96cc5b132c5c46fcecc76f21eaf4107f78f88ddb0:1', 'spend_info': None}]}, 'id': 0}
DEBUG    root:utils.py:253 Calling stop with params {}
DEBUG    root:utils.py:269 Received response for stop call: {'jsonrpc': '2.0', 'result': {}, 'id': 0}
DEBUG    root:utils.py:403 Waiting for ['Stopping the liana daemon.'] in the logs
DEBUG    root:utils.py:427 Found 're.compile('Stopping the liana daemon.')' in logs
ERROR    root:lianad.py:135 lianad : error when calling stop: 'Command '['/home/pyth/rust/liana/tests/test_framework/../../target/debug/lianad', '--conf', '/tmp/lianad-tests-3w2macqn/test_rescan_and_recovery_1/lianad/config.toml']' timed out after 5 seconds'
DEBUG    root:utils.py:316 Starting '/home/pyth/rust/liana/tests/test_framework/../../target/debug/lianad --conf /tmp/lianad-tests-3w2macqn/test_rescan_and_recovery_1/lianad/config.toml'
DEBUG    root:utils.py:403 Waiting for ['Database initialized and checked', 'JSONRPC server started.'] in the logs
DEBUG    root:utils.py:427 Found 're.compile('Database initialized and checked')' in logs
FAILED                                                                                                                               [100%]
------------------------------------------------------------ live log teardown -------------------------------------------------------------
DEBUG    root:utils.py:253 Calling stop with params {}
ERROR    root:lianad.py:135 lianad : error when calling stop: '[Errno 2] No such file or directory'


================================================================= FAILURES =================================================================
_________________________________________________________ test_rescan_and_recovery _________________________________________________________

lianad = <test_framework.lianad.Lianad object at 0x7673e02d9a90>, bitcoind = <test_framework.bitcoind.Bitcoind object at 0x7673e0dab800>

    def test_rescan_and_recovery(lianad, bitcoind):
        """Test user recovery flow"""
        # Get initial_tip to use for rescan later
        initial_tip = bitcoind.rpc.getblockheader(bitcoind.rpc.getbestblockhash())
    
        # Start by getting a few coins
        destination = lianad.rpc.getnewaddress()["address"]
        txid = bitcoind.rpc.sendtoaddress(destination, 0.5)
        bitcoind.generate_block(1, wait_for_mempool=txid)
        wait_for(
            lambda: lianad.rpc.getinfo()["block_height"] == bitcoind.rpc.getblockcount()
        )
        assert len(lianad.rpc.listcoins()["coins"]) == 1
    
        # Advance the blocktime by >2h in median-time past for rescan
        added_time = 60 * 60 * 3
        bitcoind.rpc.setmocktime(initial_tip["time"] + added_time)
        bitcoind.generate_block(12)
    
        # Clear lianad state
>       lianad.restart_fresh(bitcoind)

tests/test_chain.py:355: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_framework/lianad.py:112: in restart_fresh
    self.start()
tests/test_framework/lianad.py:119: in start
    self.wait_for_logs(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_framework.lianad.Lianad object at 0x7673e02d9a90>, regexs = ['Database initialized and checked', 'JSONRPC server started.']
timeout = 120

    def wait_for_logs(self, regexs, timeout=TIMEOUT):
        """Look for `regexs` in the logs.
    
        We tail the stdout of the process and look for each regex in `regexs`,
        starting from last of the previous waited-for log entries (if any).  We
        fail if the timeout is exceeded or if the underlying process
        exits before all the `regexs` were found.
    
        If timeout is None, no time-out is applied.
        """
        logging.debug("Waiting for {} in the logs".format(regexs))
    
        exs = [re.compile(r) for r in regexs]
        start_time = time.time()
        pos = self.logsearch_start
    
        while True:
            if timeout is not None and time.time() > start_time + timeout:
                print("Time-out: can't find {} in logs".format(exs))
                for r in exs:
                    if self.is_in_log(r):
                        print("({} was previously in logs!)".format(r))
                raise TimeoutError('Unable to find "{}" in logs.'.format(exs))
    
            with self.logs_cond:
                if pos >= len(self.logs):
                    if not self.running:
>                       raise ValueError("Process died while waiting for logs")
E                       ValueError: Process died while waiting for logs

tests/test_framework/utils.py:420: ValueError
---------------------------------------------------------- Captured stdout setup -----------------------------------------------------------
Running tests in /tmp/lianad-tests-3w2macqn
------------------------------------------------------------ Captured log setup ------------------------------------------------------------
DEBUG    root:utils.py:316 Starting 'bitcoind -datadir=/tmp/lianad-tests-3w2macqn/test_rescan_and_recovery_1/bitcoind -printtoconsole -server -debug=1 -debugexclude=libevent -debugexclude=tor'
DEBUG    root:utils.py:403 Waiting for ['Done loading'] in the logs
DEBUG    root:utils.py:427 Found 're.compile('Done loading')' in logs
INFO     root:bitcoind.py:94 Bitcoind started
DEBUG    root:utils.py:316 Starting 'electrs --conf /tmp/lianad-tests-3w2macqn/test_rescan_and_recovery_1/electrs/regtest/electrs.toml'
INFO     root:electrs.py:57 Electrs started
DEBUG    root:utils.py:316 Starting '/home/pyth/rust/liana/tests/test_framework/../../target/debug/lianad --conf /tmp/lianad-tests-3w2macqn/test_rescan_and_recovery_1/lianad/config.toml'
DEBUG    root:utils.py:403 Waiting for ['Database initialized and checked', 'JSONRPC server started.'] in the logs
DEBUG    root:utils.py:427 Found 're.compile('Database initialized and checked')' in logs
DEBUG    root:utils.py:427 Found 're.compile('JSONRPC server started.')' in logs
------------------------------------------------------------ Captured log call -------------------------------------------------------------
DEBUG    root:utils.py:253 Calling getnewaddress with params {}
DEBUG    root:utils.py:269 Received response for getnewaddress call: {'jsonrpc': '2.0', 'result': {'address': 'bcrt1q2700qx72qpm8a8r07srmx5v6280jt8tuauz8mjypnfhe4ehxx6hqq8gxnt', 'derivation_index': 0}, 'id': 0}
DEBUG    root:utils.py:253 Calling getinfo with params {}
DEBUG    root:utils.py:269 Received response for getinfo call: {'jsonrpc': '2.0', 'result': {'block_height': 0, 'descriptors': {'main': {'change_desc': 'wsh(or_d(pk([85d83c29]tpubD6NzVbkrYhZ4X71sRFatBukgznQTazD3toBTfthzLvmUVYw2z1jS6gzvJaeR7i8dndaxD7f9ZRWT4zsC56Nk8vXG8XA4LKPgrro8q574YGz/1/*),and_v(v:pkh([f0f444dc]tpubD6NzVbkrYhZ4YcSPGiAxVUqd6a7Q1MjEd6d6oYcTcCnj85gNPDVFkxaq1HcRVnzHu6umLSsRDM1qWzayhYsiHLwgsSCcc8dZQosGsDmYoXs/1/*),older(10))))#euzgfe24', 'multi_desc': 'wsh(or_d(pk([85d83c29]tpubD6NzVbkrYhZ4X71sRFatBukgznQTazD3toBTfthzLvmUVYw2z1jS6gzvJaeR7i8dndaxD7f9ZRWT4zsC56Nk8vXG8XA4LKPgrro8q574YGz/<0;1>/*),and_v(v:pkh([f0f444dc]tpubD6NzVbkrYhZ4YcSPGiAxVUqd6a7Q1MjEd6d6oYcTcCnj85gNPDVFkxaq1HcRVnzHu6umLSsRDM1qWzayhYsiHLwgsSCcc8dZQosGsDmYoXs/<0;1>/*),older(10))))#ecysgm34', 'receive_desc': 'wsh(or_d(pk([85d83c29]tpubD6NzVbkrYhZ4X71sRFatBukgznQTazD3toBTfthzLvmUVYw2z1jS6gzvJaeR7i8dndaxD7f9ZRWT4zsC56Nk8vXG8XA4LKPgrro8q574YGz/0/*),and_v(v:pkh([f0f444dc]tpubD6NzVbkrYhZ4YcSPGiAxVUqd6a7Q1MjEd6d6oYcTcCnj85gNPDVFkxaq1HcRVnzHu6umLSsRDM1qWzayhYsiHLwgsSCcc8dZQosGsDmYoXs/0/*),older(10))))#lku8043e'}}, 'network': 'regtest', 'rescan_progress': None, 'sync': 1.0, 'timestamp': 1725515069, 'version': '6.0.0-dev'}, 'id': 0}
DEBUG    root:utils.py:253 Calling getinfo with params {}
DEBUG    root:utils.py:269 Received response for getinfo call: {'jsonrpc': '2.0', 'result': {'block_height': 101, 'descriptors': {'main': {'change_desc': 'wsh(or_d(pk([85d83c29]tpubD6NzVbkrYhZ4X71sRFatBukgznQTazD3toBTfthzLvmUVYw2z1jS6gzvJaeR7i8dndaxD7f9ZRWT4zsC56Nk8vXG8XA4LKPgrro8q574YGz/1/*),and_v(v:pkh([f0f444dc]tpubD6NzVbkrYhZ4YcSPGiAxVUqd6a7Q1MjEd6d6oYcTcCnj85gNPDVFkxaq1HcRVnzHu6umLSsRDM1qWzayhYsiHLwgsSCcc8dZQosGsDmYoXs/1/*),older(10))))#euzgfe24', 'multi_desc': 'wsh(or_d(pk([85d83c29]tpubD6NzVbkrYhZ4X71sRFatBukgznQTazD3toBTfthzLvmUVYw2z1jS6gzvJaeR7i8dndaxD7f9ZRWT4zsC56Nk8vXG8XA4LKPgrro8q574YGz/<0;1>/*),and_v(v:pkh([f0f444dc]tpubD6NzVbkrYhZ4YcSPGiAxVUqd6a7Q1MjEd6d6oYcTcCnj85gNPDVFkxaq1HcRVnzHu6umLSsRDM1qWzayhYsiHLwgsSCcc8dZQosGsDmYoXs/<0;1>/*),older(10))))#ecysgm34', 'receive_desc': 'wsh(or_d(pk([85d83c29]tpubD6NzVbkrYhZ4X71sRFatBukgznQTazD3toBTfthzLvmUVYw2z1jS6gzvJaeR7i8dndaxD7f9ZRWT4zsC56Nk8vXG8XA4LKPgrro8q574YGz/0/*),and_v(v:pkh([f0f444dc]tpubD6NzVbkrYhZ4YcSPGiAxVUqd6a7Q1MjEd6d6oYcTcCnj85gNPDVFkxaq1HcRVnzHu6umLSsRDM1qWzayhYsiHLwgsSCcc8dZQosGsDmYoXs/0/*),older(10))))#lku8043e'}}, 'network': 'regtest', 'rescan_progress': None, 'sync': 1.0, 'timestamp': 1725515069, 'version': '6.0.0-dev'}, 'id': 0}
DEBUG    root:utils.py:253 Calling getinfo with params {}
DEBUG    root:utils.py:269 Received response for getinfo call: {'jsonrpc': '2.0', 'result': {'block_height': 102, 'descriptors': {'main': {'change_desc': 'wsh(or_d(pk([85d83c29]tpubD6NzVbkrYhZ4X71sRFatBukgznQTazD3toBTfthzLvmUVYw2z1jS6gzvJaeR7i8dndaxD7f9ZRWT4zsC56Nk8vXG8XA4LKPgrro8q574YGz/1/*),and_v(v:pkh([f0f444dc]tpubD6NzVbkrYhZ4YcSPGiAxVUqd6a7Q1MjEd6d6oYcTcCnj85gNPDVFkxaq1HcRVnzHu6umLSsRDM1qWzayhYsiHLwgsSCcc8dZQosGsDmYoXs/1/*),older(10))))#euzgfe24', 'multi_desc': 'wsh(or_d(pk([85d83c29]tpubD6NzVbkrYhZ4X71sRFatBukgznQTazD3toBTfthzLvmUVYw2z1jS6gzvJaeR7i8dndaxD7f9ZRWT4zsC56Nk8vXG8XA4LKPgrro8q574YGz/<0;1>/*),and_v(v:pkh([f0f444dc]tpubD6NzVbkrYhZ4YcSPGiAxVUqd6a7Q1MjEd6d6oYcTcCnj85gNPDVFkxaq1HcRVnzHu6umLSsRDM1qWzayhYsiHLwgsSCcc8dZQosGsDmYoXs/<0;1>/*),older(10))))#ecysgm34', 'receive_desc': 'wsh(or_d(pk([85d83c29]tpubD6NzVbkrYhZ4X71sRFatBukgznQTazD3toBTfthzLvmUVYw2z1jS6gzvJaeR7i8dndaxD7f9ZRWT4zsC56Nk8vXG8XA4LKPgrro8q574YGz/0/*),and_v(v:pkh([f0f444dc]tpubD6NzVbkrYhZ4YcSPGiAxVUqd6a7Q1MjEd6d6oYcTcCnj85gNPDVFkxaq1HcRVnzHu6umLSsRDM1qWzayhYsiHLwgsSCcc8dZQosGsDmYoXs/0/*),older(10))))#lku8043e'}}, 'network': 'regtest', 'rescan_progress': None, 'sync': 1.0, 'timestamp': 1725515069, 'version': '6.0.0-dev'}, 'id': 0}
DEBUG    root:utils.py:253 Calling listcoins with params {}
DEBUG    root:utils.py:269 Received response for listcoins call: {'jsonrpc': '2.0', 'result': {'coins': [{'address': 'bcrt1q2700qx72qpm8a8r07srmx5v6280jt8tuauz8mjypnfhe4ehxx6hqq8gxnt', 'amount': 50000000, 'block_height': 102, 'derivation_index': 0, 'is_change': False, 'is_immature': False, 'outpoint': '0ca339c609dc9d03a1fdd2d96cc5b132c5c46fcecc76f21eaf4107f78f88ddb0:1', 'spend_info': None}]}, 'id': 0}
DEBUG    root:utils.py:253 Calling stop with params {}
DEBUG    root:utils.py:269 Received response for stop call: {'jsonrpc': '2.0', 'result': {}, 'id': 0}
DEBUG    root:utils.py:403 Waiting for ['Stopping the liana daemon.'] in the logs
DEBUG    root:utils.py:427 Found 're.compile('Stopping the liana daemon.')' in logs
ERROR    root:lianad.py:135 lianad : error when calling stop: 'Command '['/home/pyth/rust/liana/tests/test_framework/../../target/debug/lianad', '--conf', '/tmp/lianad-tests-3w2macqn/test_rescan_and_recovery_1/lianad/config.toml']' timed out after 5 seconds'
DEBUG    root:utils.py:316 Starting '/home/pyth/rust/liana/tests/test_framework/../../target/debug/lianad --conf /tmp/lianad-tests-3w2macqn/test_rescan_and_recovery_1/lianad/config.toml'
DEBUG    root:utils.py:403 Waiting for ['Database initialized and checked', 'JSONRPC server started.'] in the logs
DEBUG    root:utils.py:427 Found 're.compile('Database initialized and checked')' in logs
--------------------------------------------------------- Captured stdout teardown ---------------------------------------------------------
Test failed, leaving directory '/tmp/lianad-tests-3w2macqn/test_rescan_and_recovery_1' intact
Leaving base dir '/tmp/lianad-tests-3w2macqn' as it still contains ['test_rescan_and_recovery_1']
---------------------------------------------------------- Captured log teardown -----------------------------------------------------------
DEBUG    root:utils.py:253 Calling stop with params {}
ERROR    root:lianad.py:135 lianad : error when calling stop: '[Errno 2] No such file or directory'
============================================================= warnings summary =============================================================
tests/test_chain.py: 6845 warnings
  /home/pyth/rust/liana/tests/test_framework/utils.py:376: DeprecationWarning: notifyAll() is deprecated, use notify_all() instead
    self.logs_cond.notifyAll()

tests/test_chain.py::test_rescan_and_recovery
  /home/pyth/rust/liana/tests/venv/lib/python3.12/site-packages/_pytest/threadexception.py:82: PytestUnhandledThreadExceptionWarning: Exception in thread Thread-4 (tail)
  
  Traceback (most recent call last):
    File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
      self.run()
    File "/usr/lib/python3.12/threading.py", line 1010, in run
      self._target(*self._args, **self._kwargs)
    File "/home/pyth/rust/liana/tests/test_framework/utils.py", line 379, in tail
      self.proc.stderr.close()
      ^^^^^^^^^^^^^^^^^^^^^^
  AttributeError: 'NoneType' object has no attribute 'close'
  
    warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg))

tests/test_chain.py::test_rescan_and_recovery
  /home/pyth/rust/liana/tests/venv/lib/python3.12/site-packages/_pytest/threadexception.py:82: PytestUnhandledThreadExceptionWarning: Exception in thread Thread-1 (tail)
  
  Traceback (most recent call last):
    File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
      self.run()
    File "/usr/lib/python3.12/threading.py", line 1010, in run
      self._target(*self._args, **self._kwargs)
    File "/home/pyth/rust/liana/tests/test_framework/utils.py", line 379, in tail
      self.proc.stderr.close()
      ^^^^^^^^^^^^^^^^^^^^^^
  AttributeError: 'NoneType' object has no attribute 'close'
  
    warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================= short test summary info ==========================================================
FAILED tests/test_chain.py::test_rescan_and_recovery - ValueError: Process died while waiting for logs
===================================================== 1 failed, 6847 warnings in 8.90s =====================================================

jp1ac4 · 2024-09-05T12:51:06Z

Applied @pythcoiner's fix for the failing functional tests.

darosior

ACK 79ddfbb. Didn't go through everything in details once again, but it's certainly good enough to start testing along with the other features in master.

darosior · 2024-09-05T12:55:50Z

tests/test_framework/lianad.py

@@ -107,8 +106,9 @@ def restart_fresh(self, bitcoind):
        self.stop()
        dir_path = os.path.join(self.datadir, "regtest")
        shutil.rmtree(dir_path)
-        wallet_path = os.path.join(dir_path, "lianad_watchonly_wallet")
-        bitcoind.node_rpc.unloadwallet(wallet_path)
+        if BITCOIN_BACKEND_TYPE is BitcoinBackendType.Bitcoind:


nit: could have used self.backend rather than the constant.

I don't think that would work as self.backend is of type BitcoinBackend rather than BitcoinBackendType.

Another option would be to add a backend_type() method to the BitcoinBackend abstract class so we could use if self.backend.backend_type() is BitcoinBackendType.Bitcoind.

tests/test_misc.py

darosior · 2024-09-05T13:00:43Z

src/lib.rs

+) -> Result<Electrum, StartupError> {
+    let electrum_config = match config.bitcoin_backend.as_ref() {
+        Some(config::BitcoinBackend::Electrum(electrum_config)) => electrum_config,
+        _ => Err(StartupError::MissingElectrumConfig)?,


nit: it could be asserted since this function is only ever called when this condition is true.

src/bitcoin/mod.rs

tests/test_misc.py

src/bitcoin/electrum/mod.rs

tests/test_framework/electrs.py

This includes changes from darosior's comment in wizardsardine#1222 (comment).

Here, the min RBF feerate is 1 more than that of the transaction to be replaced. The feerate of the transaction to be replaced may vary slightly depending on the signature size and is calculated during the test. As such, the min feerate should be set according to this calculated value.

This is copied from darosior's changes in wizardsardine#1222 (comment).

Thanks to pythcoiner for providing this fix.

darosior · 2024-09-05T14:06:24Z

re-ACK 341f940

darosior · 2024-09-05T15:00:10Z

The reorg tests flakiness demonstrated by CI will be addressed in a followup. @jp1ac4 will open an issue for this.

819eb92 gui(settings): allow to change node type (Michael Mallan) 2381227 gui(settings): view & edit Electrum settings (Michael Mallan) b570039 gui(settings): rename Bitcoin Core to Node (Michael Mallan) db20ae4 gui(installer): reduce empty space height (Michael Mallan) 0993905 gui(installer): update wording to include Electrum (Michael Mallan) f40af57 gui(installer): split long string and run cargo fmt (Michael Mallan) 0f09be1 gui(installer): don't change values while waiting for ping result (Michael Mallan) c93aa88 gui(installer): add electrum node option (Michael Mallan) 341e446 gui(installer): allow for different node types (Michael Mallan) 83172c7 gui(installer): add general node definition (Michael Mallan) 046b54e gui(installer): define bitcoind from general node struct (Michael Mallan) c5d9d00 gui: move bitcoind to new node module (Michael Mallan) 4536eff gui(installer): extract logic for try ping bitcoind (Michael Mallan) ef44cf3 gui(installer): add module for node step (Michael Mallan) f74f071 gui: upgrade liana dependency (jp1ac4) Pull request description: This is for #1223. For now, it's possible to edit the node's settings but not to change node type. Remaining tasks: - [x] Revert Cargo.toml once #1222 is merged. - [x] Update wording as per #1223 (comment). ACKs for top commit: pythcoiner: ACK 819eb92 Tree-SHA512: 362a14d32c2e13ba286d252d9f8a1106d63e5c40198776653b0623b433435329663126307e17da017fdbbd8a8ad273b703cc3ba54fd13fa5a0afd7dd9179089a

jp1ac4 self-assigned this Aug 8, 2024

nondiremanuel added this to the v7 - Liana milestone Aug 8, 2024

jp1ac4 force-pushed the add-electrum-backend branch from 27367e1 to 0af1a78 Compare August 8, 2024 09:19

jp1ac4 mentioned this pull request Aug 8, 2024

[GUI] Add Electrum backend option to Installer & update Settings #1223

Closed

jp1ac4 commented Aug 8, 2024

View reviewed changes

src/bitcoin/electrum/mod.rs Outdated Show resolved Hide resolved

jp1ac4 commented Aug 8, 2024

View reviewed changes

src/bitcoin/electrum/wallet.rs Outdated Show resolved Hide resolved

jp1ac4 commented Aug 8, 2024

View reviewed changes

src/config.rs Outdated Show resolved Hide resolved

darosior reviewed Aug 8, 2024

View reviewed changes

jp1ac4 mentioned this pull request Aug 9, 2024

Upgrade bitcoin dependency to v0.32 #1224

Open

jp1ac4 force-pushed the add-electrum-backend branch 2 times, most recently from f27ed53 to 0ef4613 Compare August 9, 2024 14:19

pythcoiner mentioned this pull request Aug 10, 2024

Add remote Node in the same network #1225

Closed

jp1ac4 force-pushed the add-electrum-backend branch from 0ef4613 to ea66f79 Compare August 13, 2024 16:18

jp1ac4 force-pushed the add-electrum-backend branch 2 times, most recently from a3cea58 to bb0b981 Compare August 21, 2024 15:46

jp1ac4 force-pushed the add-electrum-backend branch from bb0b981 to dac26c8 Compare August 22, 2024 12:15

jp1ac4 force-pushed the add-electrum-backend branch 3 times, most recently from 47ad084 to 42710e4 Compare August 22, 2024 14:28

jp1ac4 commented Aug 22, 2024

View reviewed changes

tests/test_misc.py Show resolved Hide resolved

jp1ac4 commented Aug 26, 2024

View reviewed changes

src/lib.rs Show resolved Hide resolved

jp1ac4 force-pushed the add-electrum-backend branch from a9d1d88 to f87912f Compare August 27, 2024 13:39

jp1ac4 mentioned this pull request Aug 28, 2024

[GUI] Installer - Add Electrum node option #1241

Merged

2 tasks

bitcoin: expose MempoolEntryFees

c4c2424

jp1ac4 added a commit to jp1ac4/liana that referenced this pull request Sep 4, 2024

bitcoin: add electrum backend

9158d96

This includes changes from darosior's comment in wizardsardine#1222 (comment).

jp1ac4 added a commit to jp1ac4/liana that referenced this pull request Sep 4, 2024

func test: wait for block heights to match

f6c54f9

This is copied from darosior's changes in wizardsardine#1222 (comment).

jp1ac4 force-pushed the add-electrum-backend branch from 20c1880 to f6c54f9 Compare September 4, 2024 11:06

jp1ac4 added a commit to jp1ac4/liana that referenced this pull request Sep 4, 2024

func test: wait for block heights to match

6901f8e

This is copied from darosior's changes in wizardsardine#1222 (comment).

jp1ac4 force-pushed the add-electrum-backend branch from f6c54f9 to 6901f8e Compare September 4, 2024 19:25

darosior approved these changes Sep 5, 2024

View reviewed changes

jp1ac4 and others added 9 commits September 5, 2024 14:55

bitcoin: add electrum backend

c7ee862

This includes changes from darosior's comment in wizardsardine#1222 (comment).

func test: allow for different bitcoin backends

a85d488

func test: allow to run using electrs backend

371e31e

ci: add electrs to func tests

72c6314

lib: expose BDK's electrum client

177dfc4

doc: update with electrum info

ada89c5

func test: wait for block heights to match

b630d46

This is copied from darosior's changes in wizardsardine#1222 (comment).

func test: prevent disconnects when using mocktime

341f940

Thanks to pythcoiner for providing this fix.

jp1ac4 force-pushed the add-electrum-backend branch from 79ddfbb to 341f940 Compare September 5, 2024 13:56

darosior merged commit 6f73347 into wizardsardine:master Sep 5, 2024
30 of 31 checks passed

jp1ac4 mentioned this pull request Sep 5, 2024

electrum: fix flaky reorg tests #1270

Closed

jp1ac4 deleted the add-electrum-backend branch September 5, 2024 15:09

This was referenced Sep 13, 2024

[WIP] tests: do not call methods on None #1249

Draft

poller: check if coin is change when setting DB's next derivation index #1333

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Electrum backend #1222

Add Electrum backend #1222

jp1ac4 commented Aug 8, 2024 •

edited

Loading

darosior left a comment

jp1ac4 commented Aug 13, 2024

jp1ac4 commented Aug 21, 2024

jp1ac4 commented Aug 22, 2024

jp1ac4 commented Aug 22, 2024

jp1ac4 commented Aug 23, 2024

jp1ac4 commented Aug 26, 2024

jp1ac4 commented Aug 27, 2024

jp1ac4 commented Sep 4, 2024

darosior commented Sep 4, 2024

jp1ac4 commented Sep 4, 2024

pythcoiner commented Sep 5, 2024 •

edited by darosior

Loading

jp1ac4 commented Sep 5, 2024

darosior left a comment

darosior Sep 5, 2024

jp1ac4 Sep 13, 2024 •

edited

Loading

darosior Sep 5, 2024

darosior commented Sep 5, 2024

darosior commented Sep 5, 2024

Add Electrum backend #1222

Add Electrum backend #1222

Conversation

jp1ac4 commented Aug 8, 2024 • edited Loading

darosior left a comment

Choose a reason for hiding this comment

jp1ac4 commented Aug 13, 2024

jp1ac4 commented Aug 21, 2024

jp1ac4 commented Aug 22, 2024

jp1ac4 commented Aug 22, 2024

jp1ac4 commented Aug 23, 2024

jp1ac4 commented Aug 26, 2024

jp1ac4 commented Aug 27, 2024

jp1ac4 commented Sep 4, 2024

darosior commented Sep 4, 2024

jp1ac4 commented Sep 4, 2024

pythcoiner commented Sep 5, 2024 • edited by darosior Loading

jp1ac4 commented Sep 5, 2024

darosior left a comment

Choose a reason for hiding this comment

darosior Sep 5, 2024

Choose a reason for hiding this comment

jp1ac4 Sep 13, 2024 • edited Loading

Choose a reason for hiding this comment

darosior Sep 5, 2024

Choose a reason for hiding this comment

darosior commented Sep 5, 2024

darosior commented Sep 5, 2024

jp1ac4 commented Aug 8, 2024 •

edited

Loading

pythcoiner commented Sep 5, 2024 •

edited by darosior

Loading

jp1ac4 Sep 13, 2024 •

edited

Loading