implement task [STT-1218] Feeding service for TT API #353

minhpham212 · 2025-08-17T01:21:22Z

Implement Task Feature: STT-1218 - Feeding Service for TT Content API

Feeding Service: STTContentAPIService – Handles API communication and data fetching

Parser: ContentAPITTItemParser – Transforms TT API responses into a Superdesk-compatible format

Add unit tests for the following files:

1.	server/tests/stt_tt_content_api_test.py
2.	server/tests/stt_tt_parse_content_api_test.py

Copilot

Pull Request Overview

This PR implements a feeding service and parser for the TT Content API as part of task STT-1218. It enables fetching content from TT's Content API and transforming it into a Superdesk-compatible format.

Adds STTContentAPIService for API communication with configurable URL and API key authentication
Implements ContentAPITTItemParser to transform TT API responses into Superdesk format
Includes comprehensive unit tests for both components with fixture data validation

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
server/stt/io/feeding_services/stt_tt_content_api.py	Main feeding service implementing HTTP-based API communication with TT Content API
server/stt/io/feed_parsers/stt_tt_parse_content_api.py	Parser that transforms TT API responses into Superdesk-compatible item format
server/tests/stt_tt_content_api_test.py	Comprehensive unit tests for the feeding service with mocked API responses
server/tests/stt_tt_parse_content_api_test.py	Unit tests for the parser with fixture data validation
server/tests/fixtures/api/stt_tt_content_api.json	Test fixture containing sample TT API response data
server/stt/io/feeding_services/init.py	Logging configuration for feeding services module
server/stt/io/feed_parsers/init.py	Logging configuration for feed parsers module
server/settings.py	Module registration for the new feeding service and parser

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-08-25T14:07:41Z

server/stt/io/feeding_services/stt_tt_content_api.py

+        # logger.warning("Data: %s", data.get("hits"))
+        logger.warning("type: %s", type(data))


These commented out and debug logging statements should be removed before production deployment. The warning log for type information is not needed in production code.

Suggested change

# logger.warning("Data: %s", data.get("hits"))

logger.warning("type: %s", type(data))

petrjasek · 2025-08-25T14:08:55Z

it's referring to content api in a few places (in code and PR name) but it's not related, would be good to rename it to just tt api to avoid confusion. also there is some code duplicated from #352, would be good to merge that one first and then reuse it

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

minhpham212 · 2025-09-01T04:03:07Z

it's referring to content api in a few places (in code and PR name) but it's not related, would be good to rename it to just tt api to avoid confusion. also there is some code duplicated from #352, would be good to merge that one first and then reuse it

Because the API returns two different data structures, we need two separate services to parse them accordingly.

petrjasek · 2025-09-02T09:08:21Z

server/stt/io/feed_parsers/stt_tt_parse_content_api.py

+        elif vc.tzinfo is None:
+            processed["versioncreated"] = vc.replace(tzinfo=timezone.utc)
+
+        # 4) Expiry based on provider config (hours)


that's handled automatically, pls remove it from parser

petrjasek · 2025-09-02T09:11:27Z

server/stt/io/feed_parsers/stt_tt_parse_content_api.py

+            h = hashlib.sha1(blob.encode("utf-8")).hexdigest()
+            return f"urn:newsml:stt.fi:stt_tt_content_api:{h}"
+        except Exception:
+            return f"urn:newsml:stt.fi:stt_tt_content_api:{uuid.uuid4()}"


not sure that's a good idea in general, that would create a new item for each version

petrjasek · 2025-09-02T09:13:04Z

server/stt/io/feed_parsers/stt_tt_parse_content_api.py

+        )
+        if isinstance(uri, (str, int)):
+            s = str(uri)
+            return f"urn:newsml:stt.fi:stt_tt_content_api:{hashlib.sha1(s.encode('utf-8')).hexdigest()}"


I wonder why using hashlib and not using eg. 250815-usaryssmote6uv-06eb0f86 as suffix or converthing the whole url into uri?

minhpham212 · 2025-09-25T00:26:00Z

@petrjasek pls help me check issue

petrjasek

pls remove all those extra and duplicated tests, seems like lots of code is duplicated in those two test files, not sure if we need parser specific one if it checks the parsing in the other file. there is no value in testing something 2x, just more work to maintain it

petrjasek · 2025-09-25T06:34:47Z

server/stt/io/feed_parsers/stt_tt_parse_content_api.py

+    def _ensure_guid(self, item: Dict[str, Any]) -> str:
+        """Generate GUIDs with a TT-specific namespace while respecting existing URNs."""
+        base_guid = super()._ensure_guid(item)
+        tt_prefix = "urn:newsml:stt.fi:stt_tt_content_api:"


imo the prefix should be like urn:tt.fi: it's not really STT content is it?

server/stt/io/feeding_services/stt_tt_content_api.py

petrjasek · 2025-09-25T06:37:22Z

server/stt/io/feeding_services/stt_tt_content_api.py

+            page_url = urlunparse(parsed._replace(query=urlencode(qs, doseq=True)))
+
+            # Use base class HTTP retry infrastructure
+            response = self._get_with_retry(page_url, headers=headers, timeout=300)


timeout 300 is imo a bit too much, or is the api that slow?

I change timeout to 60

petrjasek · 2025-09-25T06:37:45Z

server/stt/io/feeding_services/stt_tt_content_api.py

+                logger.info("TT API: total from first page = %s", total)
+
+            batch = self._extract_tt_items_from_response(data)
+            logger.info(


that's rather for .debug

server/tests/stt_tt_content_api_test.py

server/tests/stt_tt_parse_content_api_test.py

server/stt/io/feeding_services/stt_tt_content_api.py

minhpham212 · 2025-09-27T05:35:05Z

@petrjasek pls help me review

server/stt/io/feed_parsers/stt_tt_parse_content_api.py

server/stt/io/feed_parsers/__init__.py

server/tests/stt_tt_parse_content_api_test.py

minhpham212 · 2025-09-30T15:05:42Z

@petrjasek pls check again

server/stt/io/feed_parsers/stt_tt_parse_content_api.py

server/tests/stt_tt_parse_content_api_test.py

minhpham212 · 2025-10-02T15:11:11Z

I updated this code , follow review, pls help me check

petrjasek · 2025-10-03T13:25:06Z

server/stt/io/feeding_services/stt_tt_content_api.py

+            ),
+        },
+        {
+            "id": "use_trs",


I still don't know why is this configurable?

Users can enable or disable this configuration.

but why? I don't see it as a requirement and it complicates things

server/tests/stt_tt_parse_content_api_test.py

server/tests/stt_parse_content_api_test.py

server/tests/stt_content_api_test.py

minhpham212 · 2025-10-06T02:03:12Z

@petrjasek, please check this configuration library issue
The conflict is caused by:
The user requested tzlocal==2.1
superdesk-core 3.2.0.dev0 depends on tzlocal>=5.

petrjasek · 2025-10-08T09:24:05Z

I've merged the async version in so the tests should be working now

…ontent-API

minhpham212 · 2025-10-09T10:08:54Z

@petrjasek pls help me check again.

petrjasek · 2025-10-09T10:38:37Z

I checked it, would still remove that use_trs config and use it always

minhpham212 · 2025-10-13T01:48:08Z

@petrjasek I remove use_trs

petrjasek · 2025-10-13T07:32:39Z

server/stt/io/feeding_services/stt_tt_content_api.py

+        still synchronous, so we offload it via asyncio.to_thread.
+        """
+        # Offload sync fetch to a worker thread to avoid blocking the event loop
+        json_items = await asyncio.to_thread(self._fetch_tt_data, provider, update)


not sure if that's useful here, would be probably better to just use https://docs.aiohttp.org/en/stable/ for fetching, the rest can run as usual

petrjasek · 2025-10-13T07:33:50Z

server/stt/io/feeding_services/stt_tt_content_api.py

+            qs = {**base_qs, "s": str(page_size), "fr": str(offset)}
+            if trs_value:
+                qs["trs"] = trs_value
+            page_url = urlunparse(parsed._replace(query=urlencode(qs, doseq=True)))


I wonder why doing this urlencode urlunparse?

@petrjasek Please help me check the CI/CD issue.

petrjasek · 2025-10-14T09:20:32Z

server/stt/io/feeding_services/stt_tt_content_api.py

+                                )
+                            else:
+                                # No more results
+                                return [it for it in items if isinstance(it, dict)]


not sure if it needs another iteration if you only add dicts to items

petrjasek · 2025-10-14T09:21:43Z

server/stt/io/feeding_services/stt_tt_content_api.py

+                                # No more results
+                                return [it for it in items if isinstance(it, dict)]
+                            break
+                    except Exception as ex:


this should only rerun on that aiohttp.ClientResponseError right?

petrjasek · 2025-10-14T09:22:14Z

server/stt/io/feeding_services/stt_tt_content_api.py

+                offset += page_size
+                if isinstance(total, int) and offset >= total:
+                    break
+        return [it for it in items if isinstance(it, dict)]


same issue like on line 330

minhpham212 changed the title ~~implement task [STT-1217] Feeding service for Superdesk content API~~ implement task [STT-1218] Feeding service for Superdesk content API Aug 21, 2025

petrjasek requested a review from Copilot August 25, 2025 14:06

Copilot AI reviewed Aug 25, 2025

View reviewed changes

petrjasek changed the title ~~implement task [STT-1218] Feeding service for Superdesk content API~~ implement task [STT-1218] Feeding service for TT API Aug 28, 2025

petrjasek reviewed Sep 2, 2025

View reviewed changes

minhpham212 force-pushed the feature/STT-1218-Feeding-service-for-TT-Content-API branch from 9b9e6a7 to 04f089e Compare September 24, 2025 23:55

minhpham212 force-pushed the feature/STT-1218-Feeding-service-for-TT-Content-API branch from 04f089e to d624d4c Compare September 25, 2025 06:23

petrjasek reviewed Sep 25, 2025

View reviewed changes

server/stt/io/feeding_services/stt_tt_content_api.py Show resolved Hide resolved

petrjasek reviewed Sep 26, 2025

View reviewed changes

server/stt/io/feeding_services/stt_tt_content_api.py Outdated Show resolved Hide resolved

petrjasek reviewed Sep 29, 2025

View reviewed changes

petrjasek reviewed Oct 1, 2025

View reviewed changes

server/stt/io/feed_parsers/stt_tt_parse_content_api.py Outdated Show resolved Hide resolved

server/tests/stt_tt_parse_content_api_test.py Outdated Show resolved Hide resolved

minhpham212 force-pushed the feature/STT-1218-Feeding-service-for-TT-Content-API branch 2 times, most recently from 97187c0 to 4cda417 Compare October 3, 2025 03:50

petrjasek reviewed Oct 3, 2025

View reviewed changes

minhpham212 added 7 commits October 6, 2025 08:41

implement task [STT-1217] Feeding service for Superdesk content API

b011f17

fix issue test case

98a8595

fix issue format test cicd

abd69ee

improve feature STT-1218 inherit from STT-1217

d54e3c0

refactor code . fix code follow reviewer

d272b1c

add param for get latest update api

cb20e0d

update code remove _coerce_bool

5d14e77

minhpham212 added 3 commits October 6, 2025 08:43

fix code follow feedback, remove unused test case function

a7d58c5

remove unused code, follow feedback remove _ensure_guid

72707e8

Remove unnecessary code based on feedback

4aad873

minhpham212 force-pushed the feature/STT-1218-Feeding-service-for-TT-Content-API branch from d6d3bcd to 4aad873 Compare October 6, 2025 01:44

Merge branch 'develop' into feature/STT-1218-Feeding-service-for-TT-C…

24470f2

…ontent-API

minhpham212 force-pushed the feature/STT-1218-Feeding-service-for-TT-Content-API branch 2 times, most recently from ea2c88b to 4d67f9d Compare October 8, 2025 16:39

fix code correct

68a0ba7

minhpham212 force-pushed the feature/STT-1218-Feeding-service-for-TT-Content-API branch from 4d67f9d to 68a0ba7 Compare October 9, 2025 06:03

minhpham212 force-pushed the feature/STT-1218-Feeding-service-for-TT-Content-API branch 2 times, most recently from ca8f67d to b54accd Compare October 10, 2025 17:43

fix remove use_trs field

e06a417

minhpham212 force-pushed the feature/STT-1218-Feeding-service-for-TT-Content-API branch from b54accd to e06a417 Compare October 11, 2025 14:21

petrjasek reviewed Oct 13, 2025

View reviewed changes

fix api call to aiohttp and remove urlencode

adf0bfd

petrjasek reviewed Oct 14, 2025

View reviewed changes

fix correct follow review

6ab29f7

petrjasek approved these changes Oct 14, 2025

View reviewed changes

petrjasek merged commit 2ad2b94 into superdesk:develop Oct 14, 2025
12 checks passed

		# logger.warning("Data: %s", data.get("hits"))
		logger.warning("type: %s", type(data))

implement task [STT-1218] Feeding service for TT API #353

implement task [STT-1218] Feeding service for TT API #353

Uh oh!

Conversation

minhpham212 commented Aug 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Implement Task Feature: STT-1218 - Feeding Service for TT Content API

Add unit tests for the following files:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Aug 25, 2025

Choose a reason for hiding this comment

Uh oh!

petrjasek commented Aug 25, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

minhpham212 commented Sep 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

minhpham212 commented Sep 25, 2025

Uh oh!

petrjasek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

minhpham212 commented Sep 27, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

minhpham212 commented Sep 30, 2025

Uh oh!

Uh oh!

Uh oh!

minhpham212 commented Oct 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

minhpham212 commented Oct 6, 2025

Uh oh!

petrjasek commented Oct 8, 2025

minhpham212 commented Aug 17, 2025 •

edited

Loading

minhpham212 Oct 14, 2025 •

edited

Loading