OpenVPN Client unit test suite failures seen in Windows on ARM devices (Tested in Qualcomm X-Elite Laptops)

[quicmmarimut]

Hi Team,

• We are observing the below unit test suites are failing in Qualcomm Snapdragon based Windows on ARM devices (Tested in X-Elite based laptops).
• Can we do a code check-in with these failures observed in WOA devices ?
• Can you please share the pass criteria for the check-in ?

Failing Test Suites:

1. test_auth_token
2. test_crypto
3. test_pkt
4. test_provider
5. test_ssl
6. test_cryptoapi

Thanks & Regards,
Mallikarjunan

[schwabe]

Can you please be a bit more verbose what failures you are seeing? ARM64 works fine on other platforms and Windows unit tests also work fine. We have not test automation on windows/arm64 but it is suprising that almost all test suites fail for you. So it would be good if you can tell us what you are actually seeing.

[schwabe]

PS C:\repos\openvpn>  C:\Repos\openvpn\out\build\win-arm64-release\Release\test_auth_token.exe
[==========] auth-token tests: Running 9 test(s).
[ RUN      ] auth_token_basic_test
PS C:\repos\openvpn>  C:\Repos\openvpn\out\build\win-arm64-release\Release\test_buffer.exe
[==========] buffer: Running 17 test(s).
[ RUN      ] test_buffer_strprefix
[       OK ] test_buffer_strprefix
[ RUN      ] test_buffer_printf_catrunc
[       OK ] test_buffer_printf_catrunc
[ RUN      ] test_buffer_format_hex_ex
[       OK ] test_buffer_format_hex_ex
[ RUN      ] test_buffer_list_aggregate_separator_empty
[       OK ] test_buffer_list_aggregate_separator_empty
[ RUN      ] test_buffer_list_aggregate_separator_noop
[       OK ] test_buffer_list_aggregate_separator_noop
[ RUN      ] test_buffer_list_aggregate_separator_two
[       OK ] test_buffer_list_aggregate_separator_two
[ RUN      ] test_buffer_list_aggregate_separator_all
[       OK ] test_buffer_list_aggregate_separator_all
[ RUN      ] test_buffer_list_aggregate_separator_nosep
[       OK ] test_buffer_list_aggregate_separator_nosep
[ RUN      ] test_buffer_list_aggregate_separator_zerolen
[       OK ] test_buffer_list_aggregate_separator_zerolen
[ RUN      ] test_buffer_list_aggregate_separator_emptybuffers
[       OK ] test_buffer_list_aggregate_separator_emptybuffers
[ RUN      ] test_buffer_free_gc_one
[       OK ] test_buffer_free_gc_one
[ RUN      ] test_buffer_free_gc_two
[       OK ] test_buffer_free_gc_two
[ RUN      ] test_buffer_gc_realloc
[       OK ] test_buffer_gc_realloc
[ RUN      ] test_character_class
[       OK ] test_character_class
[ RUN      ] test_character_string_mod_buf
[       OK ] test_character_string_mod_buf
[ RUN      ] test_snprintf
[       OK ] test_snprintf
[ RUN      ] test_buffer_chomp
[       OK ] test_buffer_chomp
[==========] buffer: 17 test(s) run.
[  PASSED  ] 17 test(s).
PS C:\repos\openvpn>  C:\Repos\openvpn\out\build\win-arm64-release\Release\test_crypto.exe
[==========] crypto tests: Running 18 test(s).
[ RUN      ] crypto_pem_encode_decode_loopback
[       OK ] crypto_pem_encode_decode_loopback
[ RUN      ] crypto_translate_cipher_names
[       OK ] crypto_translate_cipher_names
[ RUN      ] crypto_test_tls_prf
PS C:\repos\openvpn>  C:\Repos\openvpn\out\build\win-arm64-release\Release\test_misc.exe
[==========] misc_tests: Running 6 test(s).
[ RUN      ] test_compat_lzo_string
[       OK ] test_compat_lzo_string
[ RUN      ] test_auth_fail_temp_no_flags
[       OK ] test_auth_fail_temp_no_flags
[ RUN      ] test_auth_fail_temp_flags
[       OK ] test_auth_fail_temp_flags
[ RUN      ] test_auth_fail_temp_flags_msg
[       OK ] test_auth_fail_temp_flags_msg
[ RUN      ] test_list
hash_init n_buckets=16384 mask=0x00003fff
[       OK ] test_list
[ RUN      ] test_atoi_variants
[       OK ] test_atoi_variants
[==========] misc_tests: 6 test(s) run.
[  PASSED  ] 6 test(s).
PS C:\repos\openvpn>  C:\Repos\openvpn\out\build\win-arm64-release\Release\test_ncp.exe
[==========] ncp_tests: Running 6 test(s).
[ RUN      ] test_check_ncp_ciphers_list
WARNING: cipher 'none' specified for --data-ciphers. This allows negotiation of NO encryption and tunnelled data WILL then be transmitted in clear text over the network! PLEASE DO RECONSIDER THIS SETTING!
WARNING: cipher 'none' specified for --data-ciphers. This allows negotiation of NO encryption and tunnelled data WILL then be transmitted in clear text over the network! PLEASE DO RECONSIDER THIS SETTING!
Unsupported cipher in --data-ciphers: BF-CBC
Unsupported optional cipher in --data-ciphers: vollbit
Unsupported optional cipher in --data-ciphers: kugelfisch
Unsupported optional cipher in --data-ciphers: grasshopper
Unsupported cipher in --data-ciphers: Vollbit
Unsupported cipher in --data-ciphers: Littlebit
Unsupported cipher in --data-ciphers: BF-CBC
Unsupported optional cipher in --data-ciphers: nixbit
Unsupported cipher algorithm 'AES-128-CCM'. It does not use CFB, OFB, CBC, or a supported AEAD mode
Unsupported optional cipher algorithm 'AES-128-CCM'. It does not use CFB, OFB, CBC, or a supported AEAD mode
Unsupported cipher in --data-ciphers: vollbit
Unsupported cipher in --data-ciphers: vollbit
Length of --data-ciphers is over the limit of 127 chars
[       OK ] test_check_ncp_ciphers_list
[ RUN      ] test_extract_client_ciphers
[       OK ] test_extract_client_ciphers
[ RUN      ] test_poor_man
[       OK ] test_poor_man
[ RUN      ] test_ncp_best
[       OK ] test_ncp_best
[ RUN      ] test_ncp_default
[       OK ] test_ncp_default
[ RUN      ] test_ncp_expand
[       OK ] test_ncp_expand
[==========] ncp_tests: 6 test(s) run.
[  PASSED  ] 6 test(s).
PS C:\repos\openvpn>  C:\Repos\openvpn\out\build\win-arm64-release\Release\test_pkt.exe
[==========] pkt tests: Running 10 test(s).
[ RUN      ] test_tls_decrypt_lite_none
[       OK ] test_tls_decrypt_lite_none
[ RUN      ] test_tls_decrypt_lite_auth
PS C:\repos\openvpn>  C:\Repos\openvpn\out\build\win-arm64-release\Release\test_provider.exe
[==========] xkey provider tests: Running 3 test(s).
[ RUN      ] xkey_provider_test_fetch
[       OK ] xkey_provider_test_fetch
[ RUN      ] xkey_provider_test_mgmt_sign_cb
PS C:\repos\openvpn>  C:\Repos\openvpn\out\build\win-arm64-release\Release\test_ssl.exe
[==========] ssl tests: Running 11 test(s).
[ RUN      ] crypto_pem_encode_certificate
PS C:\repos\openvpn>  C:\Repos\openvpn\out\build\win-arm64-release\Release\test_user_pass.exe
[==========] user_pass_tests: Running 7 test(s).
[ RUN      ] test_get_user_pass_defined
[       OK ] test_get_user_pass_defined
[ RUN      ] test_get_user_pass_needok
[       OK ] test_get_user_pass_needok
[ RUN      ] test_get_user_pass_inline_creds
[       OK ] test_get_user_pass_inline_creds
[ RUN      ] test_get_user_pass_authfile_stdin
[       OK ] test_get_user_pass_authfile_stdin
[ RUN      ] test_get_user_pass_authfile_file
[       OK ] test_get_user_pass_authfile_file
[ RUN      ] test_get_user_pass_dynamic_challenge
[       OK ] test_get_user_pass_dynamic_challenge
[ RUN      ] test_get_user_pass_static_challenge
[       OK ] test_get_user_pass_static_challenge
[==========] user_pass_tests: 7 test(s) run.
[  PASSED  ] 7 test(s).
PS C:\repos\openvpn>

On Win11 VM using UTM.app/qemu on a Macbook Pro with M1. It seems the tests seem to be failing but also not providing any indication of why.

[schwabe]

These crashes (basically all segfaults) only happen if you use the win-arm64-release preset. If you use the win-arm64-debug preset, everything works fine and as expected. Also all these crashes happen deep in the OpenSSL library like in ssl_cipher_collect_ciphers for the test_ssl test or in ossl_obj_obj2nid in the test_cryptoapi.

The openvpn binary shows with quick testing at least no sign of problems. It might be something that is different in the way that test binaries are built.

The VM that I have here is way too sluggish and I am not experienced enouugh in debugging these kind of issues on Windows. So it does not make sense that I continue debugging this.

[lstipakov]

I can confirm that we have some issues with the latest master and arm64. We're looking into it. The latest release (2.6.13) works fine.

[lstipakov]

Turns out it works if you just use OpenSSL binaries from the latest release, 2.6.13. While both master and 2.6.13 release use OpenSSL 3.4, there must be some difference in build process.

[quicmmarimut]

PS C:\repos\openvpn>  C:\Repos\openvpn\out\build\win-arm64-release\Release\test_auth_token.exe
[==========] auth-token tests: Running 9 test(s).
[ RUN      ] auth_token_basic_test
PS C:\repos\openvpn>  C:\Repos\openvpn\out\build\win-arm64-release\Release\test_buffer.exe
[==========] buffer: Running 17 test(s).
[ RUN      ] test_buffer_strprefix
[       OK ] test_buffer_strprefix
[ RUN      ] test_buffer_printf_catrunc
[       OK ] test_buffer_printf_catrunc
[ RUN      ] test_buffer_format_hex_ex
[       OK ] test_buffer_format_hex_ex
[ RUN      ] test_buffer_list_aggregate_separator_empty
[       OK ] test_buffer_list_aggregate_separator_empty
[ RUN      ] test_buffer_list_aggregate_separator_noop
[       OK ] test_buffer_list_aggregate_separator_noop
[ RUN      ] test_buffer_list_aggregate_separator_two
[       OK ] test_buffer_list_aggregate_separator_two
[ RUN      ] test_buffer_list_aggregate_separator_all
[       OK ] test_buffer_list_aggregate_separator_all
[ RUN      ] test_buffer_list_aggregate_separator_nosep
[       OK ] test_buffer_list_aggregate_separator_nosep
[ RUN      ] test_buffer_list_aggregate_separator_zerolen
[       OK ] test_buffer_list_aggregate_separator_zerolen
[ RUN      ] test_buffer_list_aggregate_separator_emptybuffers
[       OK ] test_buffer_list_aggregate_separator_emptybuffers
[ RUN      ] test_buffer_free_gc_one
[       OK ] test_buffer_free_gc_one
[ RUN      ] test_buffer_free_gc_two
[       OK ] test_buffer_free_gc_two
[ RUN      ] test_buffer_gc_realloc
[       OK ] test_buffer_gc_realloc
[ RUN      ] test_character_class
[       OK ] test_character_class
[ RUN      ] test_character_string_mod_buf
[       OK ] test_character_string_mod_buf
[ RUN      ] test_snprintf
[       OK ] test_snprintf
[ RUN      ] test_buffer_chomp
[       OK ] test_buffer_chomp
[==========] buffer: 17 test(s) run.
[  PASSED  ] 17 test(s).
PS C:\repos\openvpn>  C:\Repos\openvpn\out\build\win-arm64-release\Release\test_crypto.exe
[==========] crypto tests: Running 18 test(s).
[ RUN      ] crypto_pem_encode_decode_loopback
[       OK ] crypto_pem_encode_decode_loopback
[ RUN      ] crypto_translate_cipher_names
[       OK ] crypto_translate_cipher_names
[ RUN      ] crypto_test_tls_prf
PS C:\repos\openvpn>  C:\Repos\openvpn\out\build\win-arm64-release\Release\test_misc.exe
[==========] misc_tests: Running 6 test(s).
[ RUN      ] test_compat_lzo_string
[       OK ] test_compat_lzo_string
[ RUN      ] test_auth_fail_temp_no_flags
[       OK ] test_auth_fail_temp_no_flags
[ RUN      ] test_auth_fail_temp_flags
[       OK ] test_auth_fail_temp_flags
[ RUN      ] test_auth_fail_temp_flags_msg
[       OK ] test_auth_fail_temp_flags_msg
[ RUN      ] test_list
hash_init n_buckets=16384 mask=0x00003fff
[       OK ] test_list
[ RUN      ] test_atoi_variants
[       OK ] test_atoi_variants
[==========] misc_tests: 6 test(s) run.
[  PASSED  ] 6 test(s).
PS C:\repos\openvpn>  C:\Repos\openvpn\out\build\win-arm64-release\Release\test_ncp.exe
[==========] ncp_tests: Running 6 test(s).
[ RUN      ] test_check_ncp_ciphers_list
WARNING: cipher 'none' specified for --data-ciphers. This allows negotiation of NO encryption and tunnelled data WILL then be transmitted in clear text over the network! PLEASE DO RECONSIDER THIS SETTING!
WARNING: cipher 'none' specified for --data-ciphers. This allows negotiation of NO encryption and tunnelled data WILL then be transmitted in clear text over the network! PLEASE DO RECONSIDER THIS SETTING!
Unsupported cipher in --data-ciphers: BF-CBC
Unsupported optional cipher in --data-ciphers: vollbit
Unsupported optional cipher in --data-ciphers: kugelfisch
Unsupported optional cipher in --data-ciphers: grasshopper
Unsupported cipher in --data-ciphers: Vollbit
Unsupported cipher in --data-ciphers: Littlebit
Unsupported cipher in --data-ciphers: BF-CBC
Unsupported optional cipher in --data-ciphers: nixbit
Unsupported cipher algorithm 'AES-128-CCM'. It does not use CFB, OFB, CBC, or a supported AEAD mode
Unsupported optional cipher algorithm 'AES-128-CCM'. It does not use CFB, OFB, CBC, or a supported AEAD mode
Unsupported cipher in --data-ciphers: vollbit
Unsupported cipher in --data-ciphers: vollbit
Length of --data-ciphers is over the limit of 127 chars
[       OK ] test_check_ncp_ciphers_list
[ RUN      ] test_extract_client_ciphers
[       OK ] test_extract_client_ciphers
[ RUN      ] test_poor_man
[       OK ] test_poor_man
[ RUN      ] test_ncp_best
[       OK ] test_ncp_best
[ RUN      ] test_ncp_default
[       OK ] test_ncp_default
[ RUN      ] test_ncp_expand
[       OK ] test_ncp_expand
[==========] ncp_tests: 6 test(s) run.
[  PASSED  ] 6 test(s).
PS C:\repos\openvpn>  C:\Repos\openvpn\out\build\win-arm64-release\Release\test_pkt.exe
[==========] pkt tests: Running 10 test(s).
[ RUN      ] test_tls_decrypt_lite_none
[       OK ] test_tls_decrypt_lite_none
[ RUN      ] test_tls_decrypt_lite_auth
PS C:\repos\openvpn>  C:\Repos\openvpn\out\build\win-arm64-release\Release\test_provider.exe
[==========] xkey provider tests: Running 3 test(s).
[ RUN      ] xkey_provider_test_fetch
[       OK ] xkey_provider_test_fetch
[ RUN      ] xkey_provider_test_mgmt_sign_cb
PS C:\repos\openvpn>  C:\Repos\openvpn\out\build\win-arm64-release\Release\test_ssl.exe
[==========] ssl tests: Running 11 test(s).
[ RUN      ] crypto_pem_encode_certificate
PS C:\repos\openvpn>  C:\Repos\openvpn\out\build\win-arm64-release\Release\test_user_pass.exe
[==========] user_pass_tests: Running 7 test(s).
[ RUN      ] test_get_user_pass_defined
[       OK ] test_get_user_pass_defined
[ RUN      ] test_get_user_pass_needok
[       OK ] test_get_user_pass_needok
[ RUN      ] test_get_user_pass_inline_creds
[       OK ] test_get_user_pass_inline_creds
[ RUN      ] test_get_user_pass_authfile_stdin
[       OK ] test_get_user_pass_authfile_stdin
[ RUN      ] test_get_user_pass_authfile_file
[       OK ] test_get_user_pass_authfile_file
[ RUN      ] test_get_user_pass_dynamic_challenge
[       OK ] test_get_user_pass_dynamic_challenge
[ RUN      ] test_get_user_pass_static_challenge
[       OK ] test_get_user_pass_static_challenge
[==========] user_pass_tests: 7 test(s) run.
[  PASSED  ] 7 test(s).
PS C:\repos\openvpn>

On Win11 VM using UTM.app/qemu on a Macbook Pro with M1. It seems the tests seem to be failing but also not providing any indication of why.

Hi @schwabe & @lstipakov,

Thanks for your suggestions. We will again re-verify these unit test cases in Windows on ARM devices and attached more detailed logs for your analysis if we see any failures.

Thanks & Regards,
Mallikarjunan

[schwabe]

@quicmmarimut my test were on Mac M1/UTM and they did fail. So something is different with your Mac M1/UTM than mine.

In fact, your tests are also failing. You can see that some of the tests just stop in the middle and never print the [ PASSED ] line.

[quicmmarimut]

Thanks for the suggestion @schwabe !

Building for win-arm64-debug and testing using those binaries resolved the issues in Qualcomm Snapdragon XElite platform.

Results with debug version of (win-arm64-debug) OpenVPN:

Start 1: test_auth_token
1/11 Test #1: test_auth_token .................. Passed 0.12 sec
Start 2: test_buffer
2/11 Test #2: test_buffer ...................... Passed 0.04 sec
Start 3: test_crypto
3/11 Test #3: test_crypto ...................... Passed 0.05 sec
Start 4: test_misc
4/11 Test #4: test_misc ........................ Passed 0.05 sec
Start 5: test_ncp
5/11 Test #5: test_ncp ......................... Passed 0.05 sec
Start 6: test_packet_id
6/11 Test #6: test_packet_id ................... Passed 0.04 sec
Start 7: test_pkt
7/11 Test #7: test_pkt ......................... Passed 0.06 sec
Start 8: test_provider
8/11 Test #8: test_provider .................... Passed 0.06 sec
Start 9: test_ssl
9/11 Test #9: test_ssl ......................... Passed 0.20 sec
Start 10: test_user_pass
10/11 Test #10: test_user_pass ................... Passed 0.05 sec
Start 11: test_cryptoapi
11/11 Test #11: test_cryptoapi ................... Passed 0.14 sec

100% tests passed, 0 tests failed out of 11

Total Test time (real) = 0.89 sec

Thanks & Regards,
Mallikarjunan

[quicmmarimut]

Hi Team,

1. Is there any reason why we should use "win-arm64-debug" instead of "win-arm64-release" only in Qualcomm Snapdragon X-Elite based devices?
2. Is the CI / CT GitHub pipeline unit test case run is with "debug" builds or "release" builds?

Because we notice that the unit test execution is happening in intel i7 device with 100 percent pass rate even with "win-x64-release" builds. Please clarify.

Thanks & Regards,
Mallikarjunan

[lstipakov]

There is something with build tools we (and apparently you) use on development machines which produces broken OpenSSL binaries - we also see this problem with openvpn.exe. However, OpenSSL binaries produces by our release machinery are fine - and there we use the same OpenSSL version (3.4). You could, for example, replace your OpenSSL binaries with the ones from the latest release and the tests should work.

We are looking into it.

[lstipakov]

Related to openssl/openssl#26239 and microsoft/vcpkg#42828