Create QML test automation target (qml/test/test_bitcoin-qt) #215

johnny9 · 2023-01-13T01:59:33Z

Adds src/qml/test/test_bitcoin-qt as a new test target. This program is built using the QuickTest library. It can be used to validate our qml code by rederimg our components and providing simulated user input. It is set to only be built if QuickTest library is found, gui tests are enabled, and bitcoin-qt is configured for qml.

As a part of test setup, a test implementation of our imageprovider is added to the test target to avoid bringing in too many dependencies. Additionally, a fake AppMode component is added to mock setting the device layout mode.

More information about the QuickTest library can be found at https://doc.qt.io/qt-5.15/qtquicktest-index.html.

jarolrod

Concept ACK, a good start. Testing that we can progress through onboarding is a great first test case. Will look into failures.

hebasto · 2023-01-14T11:27:07Z

@johnny9

Thank you very much for working on tests!

Mind adding some words to the PR description?

hebasto

Concept ACK.

If there are no reasons against it, maybe rename s/tst_onboarding.qml/test_onboarding.qml/?

src/qml/test/tst_onboarding.qml

src/Makefile.qmltest.include

src/qml/test/onboardingtests.h

src/qml/test/org/bitcoincore/qt/AppMode.qml

johnny9 · 2023-01-14T11:57:18Z

tst_*.qml is actually the pattern the library uses to find test cases in a directory.

johnny9 · 2023-01-14T12:08:16Z

Still have some linking issues with Makefile.qmltest.include it seems as CI isn't happy. I likely linked too make things into the binary.

hebasto · 2023-01-14T12:12:50Z

Still have some linking issues with Makefile.qmltest.include it seems as CI isn't happy. I likely linked too make things into the binary.

Looking into them...

src/qml/bitcoin_qml.qrc

hebasto · 2023-01-14T13:31:05Z

Static builds, which all builds with depends are, require importing of plugins like this:

--- a/src/qml/test/onboardingtests.cpp
+++ b/src/qml/test/onboardingtests.cpp
@@ -1,11 +1,32 @@
 // Copyright (c) 2023 The Bitcoin Core developers
 // Distributed under the MIT software license, see the accompanying
 // file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+#include <qml/test/onboardingtests.h>
+
+#if defined(HAVE_CONFIG_H)
+#include <config/bitcoin-config.h>
+#endif
+
 #include <QtQuickTest>
 #include <QQmlEngine>
 #include <QQmlContext>
 #include <qml/test/testimageprovider.h>
-#include <qml/test/onboardingtests.h>
+
+#if defined(QT_STATICPLUGIN)
+#include <QtPlugin>
+Q_IMPORT_PLUGIN(QtQuick2Plugin)
+Q_IMPORT_PLUGIN(QTestQmlModule)
+#if defined(QT_QPA_PLATFORM_XCB)
+Q_IMPORT_PLUGIN(QXcbIntegrationPlugin);
+#elif defined(QT_QPA_PLATFORM_WINDOWS)
+Q_IMPORT_PLUGIN(QWindowsIntegrationPlugin);
+#elif defined(QT_QPA_PLATFORM_COCOA)
+Q_IMPORT_PLUGIN(QCocoaIntegrationPlugin);
+#elif defined(QT_QPA_PLATFORM_ANDROID)
+Q_IMPORT_PLUGIN(QAndroidPlatformIntegrationPlugin)
+#endif
+#endif
 
 Setup::Setup()
 {

And add to build-aux/m4/bitcoin_qt.m4 a few lines QT_LIBS="$QT_LIBS -L$qt_plugin_path/../qml/QtTest" and _BITCOIN_QT_CHECK_STATIC_PLUGIN([QTestQmlModule], [-lqmltestplugin]) (did not test yet).

hebasto · 2023-01-14T13:34:00Z

Maybe, label this PR as a draft for now?

build-aux/m4/bitcoin_qt.m4

hebasto · 2023-01-14T14:27:28Z

FWIW, I'm testing this PR on Ubuntu 22.04 with depends:

$ make -C depends NO_WALLET=1 NO_ZMQ=1 NO_UPNP=1 NO_NATPMP=1 NO_QR=1
$ ./autogen.sh
$ ./configure --disable-fuzz-binary --disable-bench CONFIG_SITE=$PWD/depends/x86_64-pc-linux-gnu/share/config.site
$ make clean
$ make

hebasto · 2023-01-14T14:52:58Z

From https://doc.qt.io/qt-5/qtquicktest-index.html:

Note: There is no binary compatibility guarantee for the Qt Quick Test module. This means that an application that uses Qt Quick Test is only guaranteed to work with the Qt version it was developed against. However, source compatibility is guaranteed.

Should we skip Quick test when building with depends?

johnny9 · 2023-01-14T16:52:00Z

Thank you for the review Hebasto. Very helpful. Marked as a draft and will unmark after addressing the issues you've mentioned and am more confident it can get through CI.

hebasto · 2023-01-15T14:32:50Z

@johnny9

Here is a quick and dirty branch which is built successfully in all CI tasks. However, running qml/test/test_bitcoin-qt in the CI environment needs more investigation regarding which value of the QT_QPA_PLATFORM variable should be used.

Feel free to use my branch at your discretion.

johnny9 · 2023-01-15T14:40:02Z

@johnny9

Here is a quick and dirty branch which is built successfully in all CI tasks. However, running qml/test/test_bitcoin-qt in the CI environment needs more investigation regarding which value of the QT_QPA_PLATFORM variable should be used.

Feel free to use my branch at your discretion.

Thank you. I'll take a look at it and merge it with my branch. I made an attempt last night but struggled with the QT STATIC PLUGIN checks. I'll be investigating how to get this supported in cirrus afterwards. I have some ideas but it'll depend on what kind of platform support they have for a buffer to render into. I'd like to use this to validate the qml in our PRs and maybe even generate screenshots since that eats up a lot of review time.

hebasto · 2023-01-15T14:43:53Z

I made an attempt last night but struggled with the QT STATIC PLUGIN checks.

Still thinking it doesn't worth. See #215 (comment).

hebasto · 2023-01-16T13:25:46Z

The qml/pages/onboarding/OnboardingWizard.qml is still to be added to QML_RES_QML.

hebasto · 2023-01-20T10:12:39Z

Might the introducing of src/qml/pages/onboarding/OnboardingWizard.qml be split up into its own PR?

Co-authored-by: Hennadii Stepanov <[email protected]>

johnny9 · 2023-09-01T00:57:42Z

Update from 75da2a2 to ed8222b

Rebased with main

jarolrod

General Concept ACK

Test cases work and run

********* Start testing of onboarding *********
Config: Using QtTest library 5.15.10, Qt 5.15.10 (arm64-little_endian-lp64 shared (dynamic) release build; by Clang 14.0.0 (clang-1400.0.29.202) (Apple)), osx 13.5
PASS   : onboarding::initTestCase()
PASS   : onboarding::test_all_continue_buttons()
PASS   : onboarding::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted, 1042ms
********* Finished testing of onboarding *********

jarolrod · 2023-09-08T07:12:51Z

src/qml/test/testimageprovider.cpp

@@ -0,0 +1,110 @@
+// Copyright (c) 2023 The Bitcoin Core developers


There has to be a way for us to not have to copy over this into qml/test

jarolrod · 2023-09-08T07:19:54Z

src/qml/test/tst_onboarding.qml

+
+            var cover = findChild(onboardingWizard, "onboardingCover")
+            verify(cover, onboardingWizard.currentItem)
+            mouseClick(findChild(cover, "continueButton"))


What about abstracting a lot of the repetitive action of finding and clicking into a helper function?

helper function looking something like:

function clickContinueAndVerify(pageId, expectedIndex) { var page = findChild(onboardingWizard, pageId) verify(page, onboardingWizard.currentItem) mouseClick(findChild(page, "continueButton")) verify(onboardingWizard.currentIndex == expectedIndex) }

Then calling like:

clickContinueAndVerify("onboardingCover", 1) wait(200) clickContinueAndVerify("onboardingStrengthen", 2)

A lot of tests would involve a repetitive function, so good to think about this now to establish a style for how we write our tests

jarolrod · 2023-09-08T07:22:46Z

Need to cleanup the commit history :D

pablomartin4btc

Concept ACK

It would be good to have this soon.

tested ed8222b on Ubuntu 22.04.

********* Start testing of onboarding *********
Config: Using QtTest library 5.15.3, Qt 5.15.3 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 11.3.0), ubuntu 22.04
PASS   : onboarding::initTestCase()
PASS   : onboarding::test_all_continue_buttons()
PASS   : onboarding::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted, 1049ms
********* Finished testing of onboarding *********

Apart from @jarolrod's comments, since we are moving away from SwipeView I think we'd need to update this to PageStacks (#428).

At least in Ubuntu, I see the UI gets opened and pages are switching, not sure if this is on purpose and can be hidden (I haven't checked the code) as in current Qt this is not happening.

I got these errors before the successful results pasted above.

/src/qml/test/test_bitcoin-qt 
Warning: Ignoring XDG_SESSION_TYPE=wayland on Gnome. Use QT_QPA_PLATFORM=wayland to run on Wayland anyway.
file:///home/pablo/workspace/bitcoin-core-gui-qml/src/qml/controls/Theme.qml:13:5: QML Settings: Failed to initialize QSettings instance. Status code is: 1
file:///home/pablo/workspace/bitcoin-core-gui-qml/src/qml/controls/Theme.qml:13:5: QML Settings: The following application identifiers have not been set: QVector("organizationName", "organizationDomain")
file:///home/pablo/workspace/bitcoin-core-gui-qml/src/qml/components/StorageOptions.qml:33: ReferenceError: optionsModel is not defined
file:///home/pablo/workspace/bitcoin-core-gui-qml/src/qml/components/AboutOptions.qml:54: ReferenceError: nodeModel is not defined
file:///home/pablo/workspace/bitcoin-core-gui-qml/src/qml/components/DeveloperOptions.qml:20: ReferenceError: optionsModel is not defined
file:///home/pablo/workspace/bitcoin-core-gui-qml/src/qml/components/DeveloperOptions.qml:45: ReferenceError: optionsModel is not defined
file:///home/pablo/workspace/bitcoin-core-gui-qml/src/qml/components/DeveloperOptions.qml:16: ReferenceError: optionsModel is not defined
file:///home/pablo/workspace/bitcoin-core-gui-qml/src/qml/components/DeveloperOptions.qml:41: ReferenceError: optionsModel is not defined
file:///home/pablo/workspace/bitcoin-core-gui-qml/src/qml/components/StorageOptions.qml:27: ReferenceError: optionsModel is not defined
file:///home/pablo/workspace/bitcoin-core-gui-qml/src/qml/components/StorageOptions.qml:25: ReferenceError: chainModel is not defined
file:///home/pablo/workspace/bitcoin-core-gui-qml/src/qml/components/StorageOptions.qml:42: ReferenceError: chainModel is not defined
file:///home/pablo/workspace/bitcoin-core-gui-qml/src/qml/components/StorageOptions.qml:41: ReferenceError: optionsModel is not defined
file:///home/pablo/workspace/bitcoin-core-gui-qml/src/qml/components/StorageSettings.qml:19: ReferenceError: optionsModel is not defined
file:///home/pablo/workspace/bitcoin-core-gui-qml/src/qml/components/StorageSettings.qml:43: ReferenceError: optionsModel is not defined
file:///home/pablo/workspace/bitcoin-core-gui-qml/src/qml/components/ConnectionSettings.qml:17: ReferenceError: optionsModel is not defined
file:///home/pablo/workspace/bitcoin-core-gui-qml/src/qml/components/ConnectionSettings.qml:30: ReferenceError: optionsModel is not defined
file:///home/pablo/workspace/bitcoin-core-gui-qml/src/qml/components/ConnectionSettings.qml:43: ReferenceError: optionsModel is not defined
file:///home/pablo/workspace/bitcoin-core-gui-qml/src/qml/components/ConnectionSettings.qml:56: ReferenceError: optionsModel is not defined
file:///home/pablo/workspace/bitcoin-core-gui-qml/src/qml/pages/onboarding/OnboardingStorageLocation.qml:23: ReferenceError: chainModel is not defined
********* Start testing of onboarding *********
Config: Using QtTest library 5.15.3, Qt 5.15.3 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 11.3.0), ubuntu 22.04
PASS   : onboarding::initTestCase()
PASS   : onboarding::test_all_continue_buttons()
PASS   : onboarding::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted, 1049ms
********* Finished testing of onboarding *********

pablomartin4btc · 2024-11-17T02:55:13Z

During discussions with @jarolrod about reorganizing the project structure (components, pages, controls, etc.), we noted that tests will need to be updated once Qt 6 is introduced, along with CMake. Considering this, it might be better to treat this as a draft for now, until we decide whether to create a new PR from scratch.

jarolrod reviewed Jan 13, 2023

View reviewed changes

johnny9 force-pushed the qmltest branch from 082b543 to 27a615a Compare January 13, 2023 13:47

hebasto reviewed Jan 14, 2023

View reviewed changes

src/qml/test/tst_onboarding.qml Outdated Show resolved Hide resolved

src/Makefile.qmltest.include Outdated Show resolved Hide resolved

src/qml/test/onboardingtests.h Outdated Show resolved Hide resolved

src/qml/test/org/bitcoincore/qt/AppMode.qml Show resolved Hide resolved

hebasto reviewed Jan 14, 2023

View reviewed changes

src/qml/bitcoin_qml.qrc Show resolved Hide resolved

johnny9 force-pushed the qmltest branch from 1dc737c to 70b8d47 Compare January 14, 2023 12:50

hebasto reviewed Jan 14, 2023

View reviewed changes

build-aux/m4/bitcoin_qt.m4 Outdated Show resolved Hide resolved

johnny9 marked this pull request as draft January 14, 2023 16:50

johnny9 force-pushed the qmltest branch 3 times, most recently from 5cd0be3 to 38a8d16 Compare January 16, 2023 13:06

johnny9 force-pushed the qmltest branch 5 times, most recently from 0f0cfe3 to 75da2a2 Compare January 17, 2023 01:07

johnny9 marked this pull request as ready for review January 17, 2023 17:16

hebasto added the Needs rebase label Feb 6, 2023

Johnny Carlson and others added 20 commits August 31, 2023 20:49

qml: add makefile for qmltest target

59af85b

qml: add test harness for onboarding qml tests

3e229de

qml: add QtQuickTest library to qmltest target

6096e62

qml: add first qml testcase

325808c

qml: add imageprovider to test setup

6186017

qml: remove double linkage in Makefile.qmltest.include

0cf1f3d

qml/test: have OnboardingCover fill parent

c8b15a4

qml/test: add fake qt.bitcoincore.org module

a75059e

qml: extract OnboardingWizard from main.qml

8401964

qml/test: add test_all_continue_buttons

eb2eb8d

qmltest: configure to only compile with qml and tests enabled

2ef6479

qmltest: rename testimageprovider

332ed16

qml/test: add include guards to headers

3909a63

gitignore: add output from qmltest

dcfba81

qmltest: don't compile if qml isn't enabled

953ce9a

qmltest: rename onboardingtests to setup

ad9038f

qmltest: add/fix copyright comments

d41e3b3

qmltest: remove extra flags/link aguments from makefile

e309568

qmltest: fix build and don't build when static linking

9176f79

Co-authored-by: Hennadii Stepanov <[email protected]>

qmltest: remove from TESTS until able to run in ci

ed8222b

johnny9 force-pushed the qmltest branch from 75da2a2 to ed8222b Compare September 1, 2023 00:50

jarolrod removed the Needs rebase label Sep 1, 2023

jarolrod reviewed Sep 8, 2023

View reviewed changes

hebasto added the Needs rebase label Aug 12, 2024

pablomartin4btc reviewed Nov 7, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create QML test automation target (qml/test/test_bitcoin-qt) #215

Create QML test automation target (qml/test/test_bitcoin-qt) #215

johnny9 commented Jan 13, 2023 •

edited

Loading

jarolrod left a comment

hebasto commented Jan 14, 2023

hebasto left a comment

johnny9 commented Jan 14, 2023

johnny9 commented Jan 14, 2023

hebasto commented Jan 14, 2023

hebasto commented Jan 14, 2023

hebasto commented Jan 14, 2023

hebasto commented Jan 14, 2023

hebasto commented Jan 14, 2023

johnny9 commented Jan 14, 2023

hebasto commented Jan 15, 2023

johnny9 commented Jan 15, 2023

hebasto commented Jan 15, 2023

hebasto commented Jan 16, 2023

hebasto commented Jan 20, 2023

johnny9 commented Sep 1, 2023

jarolrod left a comment

jarolrod Sep 8, 2023

jarolrod Sep 8, 2023

jarolrod Sep 8, 2023

jarolrod commented Sep 8, 2023

pablomartin4btc left a comment

pablomartin4btc commented Nov 17, 2024

		@@ -0,0 +1,110 @@
		// Copyright (c) 2023 The Bitcoin Core developers

Create QML test automation target (qml/test/test_bitcoin-qt) #215

Are you sure you want to change the base?

Create QML test automation target (qml/test/test_bitcoin-qt) #215

Conversation

johnny9 commented Jan 13, 2023 • edited Loading

jarolrod left a comment

Choose a reason for hiding this comment

hebasto commented Jan 14, 2023

hebasto left a comment

Choose a reason for hiding this comment

johnny9 commented Jan 14, 2023

johnny9 commented Jan 14, 2023

hebasto commented Jan 14, 2023

hebasto commented Jan 14, 2023

hebasto commented Jan 14, 2023

hebasto commented Jan 14, 2023

hebasto commented Jan 14, 2023

johnny9 commented Jan 14, 2023

hebasto commented Jan 15, 2023

johnny9 commented Jan 15, 2023

hebasto commented Jan 15, 2023

hebasto commented Jan 16, 2023

hebasto commented Jan 20, 2023

johnny9 commented Sep 1, 2023

jarolrod left a comment

Choose a reason for hiding this comment

jarolrod Sep 8, 2023

Choose a reason for hiding this comment

jarolrod Sep 8, 2023

Choose a reason for hiding this comment

jarolrod Sep 8, 2023

Choose a reason for hiding this comment

jarolrod commented Sep 8, 2023

pablomartin4btc left a comment

Choose a reason for hiding this comment

pablomartin4btc commented Nov 17, 2024

johnny9 commented Jan 13, 2023 •

edited

Loading