expose execute() in HttpClient and rename batch_requests() into batch_execute()

Author: BurnzZ

docs/advanced/additional-requests.rst

@@ -23,6 +23,7 @@ properly extract data for some websites.
     with today's websites which relies on a lot of page interactions to display
     its contents.
 
+.. _`httprequest-example`:
 
 HttpRequest
 ===========
@@ -263,21 +264,67 @@ additional requests asynchronously using ``asyncio.gather()``, ``asyncio.wait()`
 etc. This means that ``asyncio`` could be used anywhere inside the Page Object,
 including the ``to_item()`` method.
 
-In the previous section, we've explored how :class:`~.HttpRequest` are defined.
-Fortunately, the :meth:`~.HttpClient.request`, :meth:`~.HttpClient.get`, and
-:meth:`~.HttpClient.post` methods of :class:`~.HttpClient` already defines the
-:class:`~.HttpRequest` and executes it as well. The only time you'll need to create
-:class:`~.HttpRequest` manually is via the :meth:`~.HttpClient.batch_requests`
-method which is described in this section: :ref:`http-batch-request-example`.
-
+In the previous section, we've explored how :class:`~.HttpRequest` is defined.
 Let's see a few quick examples to see how to execute additional requests using
 the :class:`~.HttpClient`.
 
+Executing a HttpRequest instance
+--------------------------------
+
+.. code-block:: python
+
+    import attrs
+    import web_poet
+
+
+    @attrs.define
+    class ProductPage(web_poet.ItemWebPage):
+        http_client: web_poet.HttpClient
+
+        async def to_item(self):
+            item = {
+                "url": self.url,
+                "name": self.css("#main h3.name ::text").get(),
+                "product_id": self.css("#product ::attr(product-id)").get(),
+            }
+
+            # Simulates clicking on a button that says "View All Images"
+            request = web_poet.HttpRequest(f"https://api.example.com/v2/images?id={item['product_id']}")
+            response: web_poet.HttpResponse = await self.http_client.execute(request)
+
+            item["images"] = response.css(".product-images img::attr(src)").getall()
+            return item
+
+As the example suggests, we're performing an additional request that allows us
+to extract more images in a product page that might not otherwise be possible.
+This is because in order to do so, an additional button needs to be clicked
+which fetches the complete set of product images via AJAX.
+
+There are a few things to take note of this example:
+
+    * Recall from the :ref:`httprequest-example` tutorial section that the
+      default method is ``GET``.
+    * We're now using the ``async/await`` syntax inside the ``to_item()`` method.
+    * The response from the additional request is of type :class:`~.HttpResponse`.
+
+.. tip::
+
+    See the :ref:`http-batch-request-example` tutorial section to see how to
+    execute a group of :class:`~.HttpRequest` in batch.
+
+Fortunately, there are already some quick shortcuts on how to perform single
+additional requests using the :meth:`~.HttpClient.request`, :meth:`~.HttpClient.get`,
+and :meth:`~.HttpClient.post` methods of :class:`~.HttpClient`. These already
+define the :class:`~.HttpRequest` and executes it as well.
+
 .. _`httpclient-get-example`:
 
 A simple ``GET`` request
 ------------------------
 
+Let's use the example from the previous section and use the :meth:`~.HttpClient.get`
+method on it.
+
 .. code-block:: python
 
     import attrs
@@ -306,13 +353,8 @@ There are a few things to take note in this example:
 
     * A ``GET`` request can be done via :class:`~.HttpClient`'s
       :meth:`~.HttpClient.get` method.
-    * We're now using the ``async/await`` syntax inside the ``to_item()`` method.
-    * The response from the additional request is of type :class:`~.HttpResponse`.
-
-As the example suggests, we're performing an additional request that allows us
-to extract more images in a product page that might not otherwise be possible.
-This is because in order to do so, an additional button needs to be clicked
-which fetches the complete set of product images via AJAX.
+    * There was no need to instantiate a :class:`~.HttpRequest` since :meth:`~.HttpClient.get`
+      already handles it before executing the request.
 
 .. _`request-post-example`:
 
@@ -378,16 +420,17 @@ Batch requests
 --------------
 
 We can also choose to process requests by **batch** instead of sequentially or 
-one by one. The :meth:`~.HttpClient.batch_requests` method can be used for this
-which accepts an arbitrary number of :class:`~.HttpRequest` instances.
+one by one (e.g. using :meth:`~.HttpClient.execute`). The :meth:`~.HttpClient.batch_execute`
+method can be used for this which accepts an arbitrary number of :class:`~.HttpRequest`
+instances.
 
 Let's modify the example in the previous section to see how it can be done.
 
 The difference for this code example from the previous section is that we're
 increasing the pagination from only the **2nd page** into the **10th page**.
 Instead of calling a single :meth:`~.HttpClient.post` method, we're creating a
 list of :class:`~.HttpRequest` to be executed in batch using the
-:meth:`~.HttpClient.batch_requests` method.
+:meth:`~.HttpClient.batch_execute` method.
 
 .. code-block:: python
 
@@ -415,7 +458,7 @@ list of :class:`~.HttpRequest` to be executed in batch using the
                 self.create_request(item["product_id"], page_num=page_num)
                 for page_num in range(2, self.default_pagination_limit)
             ]
-            responses: List[web_poet.HttpResponse] = await self.http_client.batch_requests(*requests)
+            responses: List[web_poet.HttpResponse] = await self.http_client.batch_execute(*requests)
             related_product_ids = [
                 id_
                 for response in responses
@@ -452,12 +495,12 @@ The key takeaways for this example are:
       It only contains the HTTP Request information for now and isn't executed yet.
       This is useful for creating factory methods to help create requests without any
       download execution at all.
-    * :class:`~.HttpClient` has a :meth:`~.HttpClient.batch_requests` method that
+    * :class:`~.HttpClient` has a :meth:`~.HttpClient.batch_execute` method that
       can process a list of :class:`~.HttpRequest` instances asynchronously together.
 
 .. tip::
 
-    The :meth:`~.HttpClient.batch_requests` method can accept different varieties
+    The :meth:`~.HttpClient.batch_execute` method can accept different varieties
     of :class:`~.HttpRequest` that might not be related with one another. For
     example, it could be a mixture of ``GET`` and ``POST`` requests or even
     representing requests for various parts of the page altogether.
@@ -466,7 +509,7 @@ The key takeaways for this example are:
     of async execution which could be faster in certain cases `(assuming you're
     allowed to perform HTTP requests in parallel)`.
 
-    Nonetheless, you can still use the :meth:`~.HttpClient.batch_requests` method
+    Nonetheless, you can still use the :meth:`~.HttpClient.batch_execute` method
     to execute a single :class:`~.HttpRequest` instance.
 
 
@@ -566,7 +609,7 @@ For this example, let's improve the code snippet from the previous subsection na
             ]
 
             try:
-                responses: List[web_poet.HttpResponse] = await self.http_client.batch_requests(*requests)
+                responses: List[web_poet.HttpResponse] = await self.http_client.batch_execute(*requests)
             except web_poet.exceptions.HttpRequestError:
                 logger.warning(
                     f"Unable to request for more related products for product ID: {item['product_id']}"
@@ -605,17 +648,17 @@ For this example, let's improve the code snippet from the previous subsection na
         def parse_related_product_ids(response_page) -> List[str]:
             return response_page.css("#main .related-products ::attr(product-id)").getall()
 
-Handling exceptions using :meth:`~.HttpClient.batch_requests` remains largely the same.
+Handling exceptions using :meth:`~.HttpClient.batch_execute` remains largely the same.
 However, the main difference is that you might be wasting perfectly good responses just
 because a single request from the batch ruined it.
 
 An alternative approach would be salvaging good responses altogether. For example, you've
 sent out 10 :class:`~.HttpRequest` and only 1 of them had an exception during processing.
 You can still get the data from 9 of the :class:`~.HttpResponse` by passing the parameter
-``return_exceptions=True`` to :meth:`~.HttpClient.batch_requests`.
+``return_exceptions=True`` to :meth:`~.HttpClient.batch_execute`.
 
 This means that any exceptions raised during request execution are returned alongside any
-of the successful responses. The return type of :meth:`~.HttpClient.batch_requests` could
+of the successful responses. The return type of :meth:`~.HttpClient.batch_execute` could
 be a mixture of :class:`~.HttpResponse` and :class:`web_poet.exceptions.http.HttpRequestError`.
 
 Here's an example:
@@ -625,7 +668,7 @@ Here's an example:
     # Revised code snippet from the to_item() method
 
     responses: List[Union[web_poet.HttpResponse, web_poet.exceptions.HttpRequestError]] = (
-        await self.http_client.batch_requests(*requests, return_exceptions=True)
+        await self.http_client.batch_execute(*requests, return_exceptions=True)
     )
 
     related_product_ids = []

docs/advanced/meta.rst

@@ -111,12 +111,12 @@ Let's try an example wherein :class:`~.Meta` is able to control how
                 self.create_next_page_request(page_num)
                 for page_num in range(2, max_pages + 1)
             ]
-            responses = await http_client.batch_requests(*requests)
+            responses = await http_client.batch_execute(*requests)
             return [
                 url
                 for response in responses
                 for product_urls in self.parse_product_urls(response)
-                for url in product_urls:
+                for url in product_urls
             ]
 
         @staticmethod

tests/test_requests.py

@@ -98,15 +98,26 @@ async def test_http_client_keyword_enforcing(async_mock):
 
 
 @pytest.mark.asyncio
-async def test_http_client_batch_requests(async_mock):
+async def test_http_client_execute(async_mock):
+    client = HttpClient(async_mock)
+
+    request = HttpRequest("url-1")
+    response = await client.execute(request)
+
+    assert isinstance(response, HttpResponse)
+    assert response.url == "url-1"
+
+
+@pytest.mark.asyncio
+async def test_http_client_batch_execute(async_mock):
     client = HttpClient(async_mock)
 
     requests = [
         HttpRequest("url-1"),
         HttpRequest("url-get", method="GET"),
         HttpRequest("url-post", method="POST"),
     ]
-    responses = await client.batch_requests(*requests)
+    responses = await client.batch_execute(*requests)
 
     assert all([isinstance(response, HttpResponse) for response in responses])
 
@@ -120,20 +131,20 @@ async def stub_request_downloader(*args, **kwargs):
         async def err():
             raise ValueError("test exception")
         return await err()
-    client.request_downloader = stub_request_downloader
+    client._request_downloader = stub_request_downloader
 
     return client
 
 
 @pytest.mark.asyncio
-async def test_http_client_batch_requests_with_exception(client_that_errs):
+async def test_http_client_batch_execute_with_exception(client_that_errs):
 
     requests = [
         HttpRequest("url-1"),
         HttpRequest("url-get", method="GET"),
         HttpRequest("url-post", method="POST"),
     ]
-    responses = await client_that_errs.batch_requests(*requests, return_exceptions=True)
+    responses = await client_that_errs.batch_execute(*requests, return_exceptions=True)
 
     assert len(responses) == 3
     assert isinstance(responses[0], Exception)
@@ -142,9 +153,9 @@ async def test_http_client_batch_requests_with_exception(client_that_errs):
 
 
 @pytest.mark.asyncio
-async def test_http_client_batch_requests_with_exception_raised(client_that_errs):
+async def test_http_client_batch_execute_with_exception_raised(client_that_errs):
     requests = [
         HttpRequest("url-1"),
     ]
     with pytest.raises(ValueError):
-        await client_that_errs.batch_requests(*requests)
+        await client_that_errs.batch_execute(*requests)

web_poet/requests.py

@@ -80,7 +80,7 @@ class HttpClient:
     """
 
     def __init__(self, request_downloader: Callable = None):
-        self.request_downloader = request_downloader or _perform_request
+        self._request_downloader = request_downloader or _perform_request
 
     async def request(
         self,
@@ -104,7 +104,7 @@ async def request(
         headers = headers or {}
         body = body or b""
         req = HttpRequest(url=url, method=method, headers=headers, body=body)
-        return await self.request_downloader(req)
+        return await self.execute(req)
 
     async def get(self, url: str, *, headers: Optional[_Headers] = None) -> HttpResponse:
         """Similar to :meth:`~.HttpClient.request` but peforming a ``GET``
@@ -124,10 +124,19 @@ async def post(
         """
         return await self.request(url=url, method="POST", headers=headers, body=body)
 
-    async def batch_requests(
+    async def execute(self, request: HttpRequest) -> HttpResponse:
+        """Accepts a single instance of :class:`~.HttpRequest` and executes it
+        using the request implementation configured in the :class:`~.HttpClient`
+        instance.
+
+        This returns a single :class:`~.HttpResponse`.
+        """
+        return await self._request_downloader(request)
+
+    async def batch_execute(
         self, *requests: HttpRequest, return_exceptions: bool = False
     ) -> List[Union[HttpResponse, Exception]]:
-        """Similar to :meth:`~.HttpClient.request` but accepts a collection of
+        """Similar to :meth:`~.HttpClient.execute` but accepts a collection of
         :class:`~.HttpRequest` instances that would be batch executed.
 
         If any of the :class:`~.HttpRequest` raises an exception upon execution,
@@ -139,7 +148,7 @@ async def batch_requests(
         ``True`` to the ``return_exceptions`` parameter.
         """
 
-        coroutines = [self.request_downloader(r) for r in requests]
+        coroutines = [self._request_downloader(r) for r in requests]
         responses = await asyncio.gather(
             *coroutines, return_exceptions=return_exceptions
         )