Documentation of Core System Logic & Data Ingestion

[priorwave]

Apologies if I’m overlooking something, but as far as I can tell, there’s currently no documentation explaining how the system actually works. Example of things i think we need to know in order to understand the output of the system without it feeling like a black box:

•	What is the underlying algorithm or decision logic?
•	What types of data are ingested? From which sources? How much?
•	How are decisions like buy/sell/hold determined for each investor type or swarm?
•	How is recency handled—for example, how is old news weighted relative to new information?

I’m don't have the time to to read through all the source code. That said, I did review the endpoints, and it seems the only external API being called is the Financial Dataset API. When hitting the /news endpoint, it appears that only the headlines are ingested and used in decision-making, while the full article content (via URL) is not fetched or analysed.

This could be a significant issue. In many cases, articles returned for a given ticker mention the company only in passing, and the headline/title might not even be substantively related to that ticker. Without deeper content analysis, this can lead to context poisoning and poor investment decisions.
[happy to make this a seperate issue]

Suggested improvements:
• Clear documentation of the model logic and data ingestion pipeline
• Confirmation of whether full article text is or isn’t used
• If only headlines are ingested, consider fetching and evaluating full articles for better context and reduced noise

Let me know if I’ve misunderstood anything — happy to help clarify.

Example of the context poisoning with the /news end point. This ariticle is retrieved for ticker GOOG however the article is is about arm and the headline is about arm...

  {
    "ticker": "GOOG",
    "title": "Down 16%, Should You Buy the Dip on Arm Holdings?",
    "author": "Harsh Chauhan",
    "source": "The Motley Fool",
    "date": "2025-07-19T09:20:00Z",
    "url": "https://www.fool.com/investing/2025/07/19/down-19-should-you-buy-the-dip-on-arm-holdings/?source=iedfolrf0000001",
    "image_url": "https://g.foolcdn.com/editorial/images/825434/clock-time-to-buy.jpg",
    "sentiment": "neutral"
  },

[virattt]

We can certainly do more on how agents are reading news articles.

Given my personal time constraints, I've added initial implementations, but have made the code open source so folks can propose and add changes.

[priorwave]

Thanks, if I have time I can try and work on some features. But my main point is that it's really hard to figure out the high level logic without see diving into the code. I think if the logic were documented more people would contribute to the repo.

[priorwave]

Hey @virattt ! I was reviewing the response from the '/news' endpoint and considered a possible enhancement. While we could add logic in the repo to fetch the article URL, extract the text, and process it for better accuracy, I think this is better handled upstream at the data provider level (https://api.financialdatasets.ai/news/). What do you think, is it something on your roadmap for the API (i believe you own the api?)

All the best, Chris

[nccapo]

Hey, i have so many questions and it's so confusing to understand whole project without deep diving inside it.

All the best, Nik