How to keep hierarchy when parametrizing test cases with Schemathesis/Hypothesis?

[ericrommel]

Hello Everyone,

I'm trying to accomplish at least the following structure after running my pytest using Schemathesis (Hypothesis):

# S1:

Schema Testing
└─ Regular Endpoints
       └─ GET /symbol
       └─ GET /enum
       └─ POST /etc
└─ Admin Endpoints
      └─ GET /metrics
      └─ GET /admin/users
      └─ POST /etc

However, what I'm having is:

# S2:
test_full_schema
└── test_admin_endpoints[GET /metrics]
└── test_regular_endpoints[GET /enum/{enum_id}]

In S2, only the last endpoint handled by Schemathesis will be "stored". How can I have all endpoints for each type (regular and admin) under their particular base function?

I'm using the following code:

# test_full_schema.py

import schemathesis
import allure

from utils.helpers.loader import load_openapi_spec

REGULAR_BASE_URL = "http://localhost:9151"
ADMIN_BASE_URL = "http://localhost:9051"

METHODS = [
    "GET",
    "POST",
]

REGULAR_ENDPOINTS = [
    "/symbol",
    "/enum/{enum_id}",
    "/{market}/scan",
    "/global/scan",
]

ADMIN_ENDPOINTS = [
    "/scan/admin/markets",
    "/scan/{market}/admin/scan",
    "/admin/users",
    "/metrics",
]

schema = schemathesis.openapi.from_path(load_openapi_spec())


@allure.epic("Schema Epic")
@allure.feature("Schema Feature")
class TestSchema:

    @allure.title("Schema Regular Title")
    @allure.suite("Regular Endpoints")
    @schema.include(path=REGULAR_ENDPOINTS, method=METHODS).parametrize()
    def test_regular_endpoints(self, case):
        # Allure grouping
        allure.dynamic.parent_suite("Schema Regular Tests")
        allure.dynamic.sub_suite(f"{case.method} endpoints")    
        allure.dynamic.story(f"{case.method} {case.path}")

        # Using this way, I can see each case from the hypothesis inside the endpoint tested in "Execution"
        with allure.step("Endpoint: " + str(case.formatted_path)
                        + "; Query: " + str(case.query)
                        ):
            case.call(base_url=ADMIN_BASE_URL)

    @allure.title("Schema AdminTitle")
    @allure.suite("Admin Endpoints")
    @schema.include(path=ADMIN_ENDPOINTS, method=METHODS).parametrize()
    def test_admin_endpoints(self, case):
       # Allure grouping
        allure.dynamic.parent_suite("Schema Admin Tests")
        allure.dynamic.sub_suite(f"{case.method} endpoints")
        allure.dynamic.story(f"{case.method} {case.path}")

        # Using this way, I can see each case from the hypothesis inside the endpoint tested in "Execution"
        with allure.step("Endpoint: " + str(case.formatted_path)
                         + "; Query: " + str(case.query)
                         ):
            case.call(base_url=ADMIN_BASE_URL)

From the image below, you can see it better. See that there are only two test cases under test_full_schema (the Python file name), which I know are from my two pytest functions (test_regular_endpoints and test_admin_endpoints). In the right sidebar, you can see each case generated by Schemathesis/Hypothesis for the particular endpoint (Admin > GET /metrics and Regular GET /enum/{enum_id}). I was expecting, from the code shared, 4 endpoints for each pytest function.

I'm not sure if I was clear, but if not, please don't hesitate to ask, and I'll be more than happy to follow up with you.