Mocking using the mockall crate.

[Ben-PH]

As a spinoff from my efforts working on the bootstrap server, I explored mocking frameworks, and have come to point where I can share my learnings.

Current framework: An analogy of a company with multiple office-buildings

Architecture overview

A major aspect of our architecture is based on message passing, with what I describe as a "return enclosed" pattern. To draw an analogy:

• Imagine running a node as running a big company
• Each worker-module in our code-base could be represented as a separate building.
• For the most part, each of these buildings work independently.
• Some tasks, however, need to be completed by another building
• In this case, one building will need to send a "work-order" through the postal system to another building.
• If this work-order needs a response, then the envelope containing the work-order also contains a reply-paid envelope

So far so good? each "building" is actually a module: One could be the consensus-worker, another could be the bootstrap system, and so on. For the purpose of this analogy, let's consider just two building, building A and building B.

• Someone in building A would send an envelope in the post addressed to building B
• building B receives the envelope, and inside is a work-request, and a reply-paid envelope
• building B completes the work-order, and places the results in the envelope provided alongside the work request
• building B puts that response envelope in the post-box, and it makes its way back to building A

In normal operations, the bootstrap server will "use the postal services" to "send a letter" to the network-worker "building". This analogy in code form, for the purpose of fetching bootstrap peers, can be found here: https://github.com/massalabs/massa/blob/main/massa-network-exports/src/network_controller.rs#L165-L176

...and in the context of bootstrapping, is used here: https://github.com/massalabs/massa/blob/main/massa-bootstrap/src/server.rs#L825

But we just want hardcoded-results!

This is all well and good, but when testing, we don't actually care about knowing whether or not the network-worker building is doing it's job properly, so we want to hard-code what we would get from our "return envelope" contents (i.e. in the bootstrap code linked, we hard-code what network_command_sender.get_bootstrap_peers().await? returns)

...and so we mock building B. To illustrate-by-analogy: (be warned: this gets convoluted, but it's the best I can do. I'm not even 100% sure it's an accurate analogy...)

• Building A takes over the role of the post office
  • and anything sent to building B gets redirected straight back to building A instead
• Building A creates 2 teams:
  • The "mock-team": receives the work-orders that have been redirected back to Building A, and "pretends to be building B"
  • The "test-team": Sends the desired content of the returned envelope to the mock-team
• So now, in order to hard-code these "work-orders", you can send an envelope to the "test-team", and that envelope will make its way back to team in building A that sent the original work order.

And this illustrates how we hardcode what bootstrap peers are "in the network" when a bootstrap being tested. Bear in mind, I've tried my best to simplify things in the analogy.

A simplified approach

• Set up a "check-team" inside building A
• All work-orders get hand-delivered to the "check-team" instead of the post-office
• The "check-team" validates the work orders
• The "check-team" immediately provides provides the results.

in short: Just like a normal function call, but the function result is hard-coded, and the context in which it's called is validated:

• Is it called the correct number of times?
• Is it called with the correct arguments?
• Are the functions called in the correct order?
• etc.

In code form:

The intercepted messages

Here we will use the bootstrap server as an example, specifically, the need to get bootstrap peers:

#[tokio::test]
#[serial]
async fn test_bootstrap_server() {

    // The sender is what the tested server thinks of as "the envelope that will go to the other building"
    // but it's just the receiver
    let (network_cmd_tx, mut network_cmd_rx) = mpsc::channel::<NetworkCommand>(5);

    /* snip */
    let bootstrap_manager = start_bootstrap_server(
        /* snip */
        NetworkCommandSender(network_cmd_tx),
        /* snip */
    )
    .await
    .unwrap()
    .unwrap();

    /* snip */

    // intercept peers being asked
    let wait_peers = async move || {
        // wait for bootstrap to ask network for peers, send them
        let response =
            match {
                // This is `wait_network_command`, just inlined
                let network_command_receiver: &mut Receiver<NetworkCommand> = &mut network_cmd_rx;
                let timeout = 20_000.into();
                let filter_map = |cmd| match cmd {
                            NetworkCommand::GetBootstrapPeers(resp) => Some(resp),
                            _ => None,
                        };
                let timer = sleep(timeout.into());
                tokio::pin!(timer);
                loop {
                    tokio::select! {
                        cmd = network_command_receiver.recv() => match cmd {
                            Some(orig_evt) => if let Some(res_evt) = filter_map(orig_evt) { return Some(res_evt); },
                            _ => panic!("network event channel died")
                        },
                        _ = &mut timer => return None
                    }
                }
            }
            .await
            {
                Some(resp) => resp,
                None => panic!("timeout waiting for get peers command"),
            };
        let sent_peers = get_peers();
        response.send(sent_peers.clone()).unwrap();
        sent_peers
    };

    /* snip */

    // wait for peers and graph
    let sent_peers = wait_peers().await;

    // the rest is asserts and cleanup
}

using mockall

(Note: We are not validating the functions call-arguments, which is one of the many features of mockall

#[tokio::test]
#[serial]
async fn test_bootstrap_server() {

    /* snipped */

    // Account for the cloning of the network-command-sender
    let mut mocked1 = MockNetworkCommandSender::new();
    let mut mocked2 = MockNetworkCommandSender::new();
    // Assert `get_bootstrap_peers` is called EXACTLY once, hardcode its results
    mocked2
        .expect_get_bootstrap_peers()
        .times(1)
        .returning(|| Ok(get_peers()));
        
    // Assert that clone is used ONCE, and move its results into the other mock
    mocked1.expect_clone().return_once(move || mocked2);
    
    // Run the server, providing a Mock that hard-codes the `network-command-sender`s behavior, and assert-tests
    // the pattern of its usage.
    let bootstrap_manager = start_bootstrap_server::<TcpStream, MockNetworkCommandSender>(
        /* snipped */
        mocked2,
        /* snipped */
    )
    .unwrap()
    .unwrap();

    // the rest is snipped, then cleanup + asserts

}

Important observations: If mockall was used when I started my bootstrap project

The vast majority of pain points working on the bootstrap system (and still to this day), is likely to me resolved by this approach.

• There is a lot of noise removed by the /* snip */s that makes this test a challenge to reason about.
• Non-mock code that is otherwise extremely simple and straight forward becomes much more difficult to develope by virtue of the requirements of mock-equivilents. I anticipate mockall removing this pain-point
• Removing the need to do the "message-dance" removes the need to spin up threads/runtimes specifically for a mock
• The expectation pattern is a powerful tool to analyse interface use:
• The ability to have certainty with the way its used
• The ability to fail tests if it's used incorrectly

In my follow-up comment, I'll hypothesise about benefits, drawbacks, and observations that might apply in the general case.

[Ben-PH]

WIP (suggestions on what to address/take note of are encouraged)

• Basics of using mockall - simple cases
• functions taking impl args if you want to be able to mock
• address complication: mockable methods need some generecism
• address drawback: possible compile time issues
• address drawback: multiple mocks that interact with each-other
• address benefit: Thorough insight into complex interface interactions
• raise possibility of being able to test much better scenarios
• would it be easier to mock scenarios to catch regressions?
• would there be benefits for our internal/external audits?
• Onboarding pros/cons
• ...