Skip to content
BranchesTags

Repository files navigation

Market Intelligence Dashboard

Market Intelligence Dashboard

An experimental terminal-based dashboard for monitoring market news in real-time.

Note: This is an early-stage personal project. Features may be unstable and evolving.

This is NOT a web dashboard. This is NOT a generic RSS reader.

This is a terminal-first market observation tool with auto-refresh capabilities.

Features

  • Live Dashboard: Auto-refresh RSS feeds every 5 minutes with trading-desk inspired UI
  • Global Coverage: 20+ RSS sources from US, Europe, Asia, and emerging markets
  • AI Translation: Automatic EN→ZH translation with configurable backends (experimental)
  • Discord Integration: Push notifications to Discord channels
  • URL Validation: Automatic checking for expired/broken article links
  • Dual Modes: Live dashboard for monitoring, static terminal for browsing
  • Rich Content: Displays source, topic, title, description, and timestamps
  • Pluggable Storage: Optional persistence (Notion, Memory, etc.)
  • Lightweight: No orchestration, no microservices, just Node + TypeScript

Quick Start

Installation

npm install

Configuration

Create a .env file (see .env.example):

# Translation Settings
# Set to 'true' to skip translation (recommended for initial testing)
DISABLE_TRANSLATION=true

# OpenAI API Key (optional, only if using OpenAI translation)
OPENAI_API_KEY=your_openai_api_key

# Discord Integration (optional)
DISCORD_WEBHOOK_URL=your_discord_webhook_url

# Notion Storage (optional)
NOTION_TOKEN=your_notion_token
NOTION_DATABASE_ID=your_database_id

Minimum configuration to run:

DISABLE_TRANSLATION=true

All other settings are optional and can be added as needed.

Run the Dashboard

# Live Dashboard (auto-refresh every 5 minutes)
npm start
# or
npm run live

# Static Terminal (one-time fetch, interactive)
npm run terminal

# Send to Discord (for CI/CD)
npm run discord

# Legacy RSS script (deprecated, not recommended)
npm run rss

Recommended for first run: Use npm run terminal to verify everything works before trying the live dashboard.

User Interfaces

Live Dashboard (Market-Terminal Inspired)

┌────────────────────────────────────────────────────────┐
│     MARKET INTELLIGENCE DASHBOARD                      │
│   Fri Jan 10 2025 14:30:00 | Last refresh: 14:25:00   │
└────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────┐
│  1. [2h ago   ] CNBC         Markets                   │
│     Markets rally as inflation data beats estimates    │
│     The S&P 500 rose 1.2% after CPI data showed...    │
│     https://cnbc.com/...                               │
│                                                         │
│  2. [5m ago   ] Bloomberg    Energy                    │
│     Oil prices surge on supply concerns                │
│     Brent crude jumped 3% as OPEC+ extends cuts...    │
│     https://bloomberg.com/...                          │
└────────────────────────────────────────────────────────┘
 80 items | Next refresh in: 245s | r: refresh | q: quit

Features:

  • Auto-refresh every 5 minutes
  • Displays source, topic, title, description
  • Shows relative timestamps (e.g., "2h ago")
  • Countdown to next refresh
  • Smooth scrolling with vi-style navigation

Keyboard Controls:

Key Action
j / ↓ Scroll down
k / ↑ Scroll up
g Jump to top
G Jump to bottom
r Manual refresh
q / Ctrl-C Quit

Static Terminal (Interactive)

┌──────────────────────────────────────┐
│ Market News Feed                     │
│                                      │
│ > Oil prices rise...                 │
│   Fed officials signal...            │
│   China PMI weakens...               │
│                                      │
├──────────────────────────────────────┤
│ Details                              │
│                                      │
│ Reuters                              │
│ 中文: 原油价格上涨                     │
│ Description: Brent crude...          │
│ URL (Press 'o' to check & open)      │
└──────────────────────────────────────┘

Features:

  • One-time fetch with caching
  • Detailed view with URL validation
  • Interactive item selection
  • Duplicate detection

Keyboard Controls:

Key Action
j / ↓ Navigate down
k / ↑ Navigate up
enter Select item (updates details)
o Check URL validity and open in browser
q / Ctrl-C Quit

Architecture

See ARCHITECTURE.md for detailed documentation.

Folder Structure

src/
├── models/          # Type definitions (NewsItem, FeedSource)
├── feeds/           # RSS fetching and source configuration
│   ├── fetcher.ts   # Multi-format RSS/Atom parser
│   └── sources.ts   # Global source configuration
├── pipeline/        # Data normalization and validation
│   └── normalize.ts # HTML cleaning, URL validation, ID generation
├── ai/              # AI processing pipeline
│   ├── processor.ts # Main AI orchestration
│   └── translator.ts # EN→ZH translation
├── renderers/       # Pure formatting functions
│   └── formatters.ts # Terminal output formatting
├── terminal/        # TUI components
│   ├── ui.ts        # Static terminal UI
│   └── liveDashboard.ts # Live auto-refresh dashboard
├── storage/         # Pluggable storage backends
│   ├── memory.ts    # In-memory cache
│   └── notion.ts    # Notion database integration
├── utils/           # Utility functions
│   ├── logger.ts    # Silenceable logging system
│   ├── urlValidator.ts # URL checking and validation
│   ├── testFeeds.ts # RSS source testing tool
│   ├── testDiscord.ts # Discord webhook tester
│   └── sendToDiscord.ts # Discord notification sender
├── discord.ts       # Discord webhook client
├── terminal.ts      # Static terminal entry point
├── liveDashboard.ts # Live dashboard entry point
└── rss.ts           # Legacy RSS script (deprecated)

Data Flow

Live Dashboard:

┌─────────────────────────────────────────────────────┐
│                   Every 5 Minutes                    │
└─────────────────────────────────────────────────────┘
                         │
                         ↓
    RSS Feeds (23 sources) → Fetch → Normalize
                                        │
                                        ↓
                              AI Processing (Translation)
                                        │
                                        ↓
                              Memory Storage (Cache)
                                        │
                                        ↓
                         Live Terminal UI (Auto-update)

Static Terminal:

RSS Feeds → Fetch → Normalize → AI Processing → Memory Storage
                                                       │
                                                       ↓
                                              Terminal UI (One-time)
                                                       │
                                                       ↓
                                           Optional: Notion Storage

Discord Notifications:

RSS Feeds → Fetch → Normalize → AI Processing → Format → Discord Webhook

Design Philosophy

Terminal-First

The terminal interface is prioritized over web UIs. Storage is optional and pluggable.

Lightweight Infrastructure

This project intentionally avoids:

  • Container orchestration (Kubernetes, Docker Swarm)
  • Microservices architecture
  • Heavy databases (Redis, vector DBs)
  • Complex deployment pipelines

Instead, it focuses on:

  • Simple Node.js process
  • Local-first operation
  • Optional cloud integrations
  • Easy deployment

Observation Over Collection

This tool is designed for real-time observation rather than exhaustive archival.

Focus areas:

  • Market event monitoring
  • Pattern recognition over time
  • Multi-source correlation
  • Narrative trend tracking

Not focused on:

  • Complete historical archives
  • Advanced analytics dashboards
  • Automated trading signals

RSS Sources

20+ global sources across major markets:

US Sources

  • Bloomberg Markets
  • CNBC (multiple feeds covering Tech, Energy, Economy, Finance, Markets)
  • Wall Street Journal Markets
  • Yahoo Finance
  • Financial Times

European Sources

  • Euronews Business

Asian Sources

  • Nikkei Asia
  • NHK World Business
  • Xinhua Business
  • Economic Daily News (Taiwan)
  • Korea Herald Business
  • The Economic Times (India)

Other Regions

  • Financial Post (Canada)
  • El Financiero (Mexico)
  • BusinessTech (South Africa)
  • Investing.com (Global)

Configure sources in src/feeds/sources.ts. All sources are tested periodically but availability may vary.

Testing RSS Sources

# Test all configured feeds
npm run test:feeds

# Shows response times, success/error counts, and item counts

Discord Integration

Send market news updates to Discord channels.

Setup

  1. Create a webhook in your Discord server:

    • Go to Server Settings > Integrations > Webhooks
    • Click "New Webhook"
    • Copy the webhook URL

  2. Add to .env:

    DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/...

Usage

# Test Discord connection
npm run test:discord

# Send latest news to Discord (top 5 items)
npm run discord

Message Format:

  • Topic with emoji indicator (📈 Markets, ⚡ Energy, etc.)
  • Source and title (English + translated if enabled)
  • Description/summary (if available)
  • Article URL
  • Timestamp

Note: Rate limiting applies - messages are sent with 1 second intervals.

Utility Commands

# Test all RSS feeds
npm run test:feeds

# Test Discord webhook
npm run test:discord

# Send news to Discord
npm run discord

Troubleshooting

Dashboard shows "Fetching..." messages

The live dashboard may show console output if logger silencing fails. This is usually harmless but can be fixed by:

  • Restarting the dashboard
  • Ensuring you're using the latest code
  • Check for any error messages in the logs

Translation not working

Local model (transformers):

  • First run downloads ~100MB of model files
  • May fail on macOS due to ONNX Runtime compatibility
  • Solution: Set DISABLE_TRANSLATION=true or use OpenAI API

OpenAI API:

  • Check API key is valid and has credits
  • Verify .env file is properly loaded
  • Check for rate limiting errors

RSS feeds failing

Some feeds may be temporarily unavailable or rate-limited:

# Test specific feeds to identify issues
npm run test:feeds

Common causes:

  • Website maintenance or downtime
  • Rate limiting (especially on shared IPs)
  • Changed or discontinued RSS feeds
  • Network/firewall restrictions

Discord messages not sending

  • Verify webhook URL is correct and active
  • Check Discord server permissions
  • Test webhook separately: npm run test:discord
  • Check for rate limiting (max ~50 messages per minute)

"Command not found" errors

Ensure dependencies are installed:

npm install

TypeScript errors

# Check for compilation errors
npx tsc --noEmit

Translation Options

Note: Translation features are experimental and may have varying quality/reliability.

Option 1: Local Model (Experimental)

Uses @xenova/transformers for offline translation.

Limitations:

  • May have compatibility issues on macOS (ONNX Runtime errors)
  • Translation quality varies
  • First run downloads model files (~100MB)

Set in .env:

DISABLE_TRANSLATION=false

Option 2: OpenAI API

Better translation quality, more reliable.

Requirements: Valid OpenAI API key and credits.

Set in .env:

OPENAI_API_KEY=your_key_here
DISABLE_TRANSLATION=false

Option 3: Disable Translation (Recommended for Development)

Skip translation entirely.

Set in .env:

DISABLE_TRANSLATION=true

Known Limitations

  • Translation quality: Local models have varying accuracy; OpenAI API requires credits
  • RSS availability: Source feeds may be rate-limited or temporarily unavailable
  • URL validation: Some publishers use redirects/paywalls that may affect checking
  • macOS compatibility: Local translation models may have ONNX Runtime issues on macOS
  • No historical data: Currently focuses on real-time monitoring only
  • Manual source management: RSS source list requires manual updates when feeds change

Potential Future Enhancements

Note: These are ideas, not commitments. Contributions welcome.

AI Features

  • Translation (EN→ZH) - experimental
  • Article summarization
  • Sentiment classification
  • Narrative/theme extraction
  • Duplicate detection improvements

Dashboard Features

  • Auto-refresh dashboard
  • Real-time timestamps
  • Description display
  • Topic filtering UI
  • Search within results
  • Historical view
  • Customizable refresh intervals

Integrations

  • Discord webhooks
  • Notion storage
  • Telegram bot
  • Slack webhooks
  • Email digest
  • RSS output (filtered/processed feed)

Storage & Export

  • Memory cache
  • Notion database
  • SQLite persistence
  • JSON export
  • CSV export
  • Markdown reports

Tech Stack

Core

  • Node.js 20+ + TypeScript 5.x
  • rss-parser - Multi-format RSS/Atom parser
  • blessed - Terminal UI framework

AI & Translation

  • OpenAI API - EN→ZH translation (optional)
  • @xenova/transformers - Local ML models (experimental, optional)
  • opencc-js - Simplified/Traditional Chinese conversion

Storage & Integrations

  • @notionhq/client - Notion database integration (optional)
  • Discord Webhooks - Push notifications (optional)
  • Memory Storage - In-memory caching

Development

  • tsx - TypeScript execution
  • dotenv - Environment configuration

Development

Entry Points

  • src/liveDashboard.ts - Live dashboard (auto-refresh, primary)
  • src/terminal.ts - Static terminal (interactive browsing)
  • src/utils/sendToDiscord.ts - Discord notifications (for CI/CD)
  • src/rss.ts - Legacy script (deprecated)

CI/CD Integration

The project includes a GitHub Actions workflow for automated news delivery (experimental):

# .github/workflows/rss.yml
# Runs every 2 hours, sends latest news to Discord

To use in your own repository:

  1. Fork this repo
  2. Add secrets in GitHub: Settings > Secrets and variables > Actions
    • DISCORD_WEBHOOK_URL (required)
    • OPENAI_API_KEY (optional, for translation)
  3. Enable GitHub Actions
  4. News will be automatically posted to your Discord channel

Note: GitHub Actions has usage limits on free tier. Monitor your Actions usage.

Command Reference

Command Description
npm start Run live dashboard (auto-refresh)
npm run live Same as npm start
npm run terminal Run static terminal (interactive)
npm run discord Send latest news to Discord
npm run test:feeds Test all RSS sources
npm run test:discord Test Discord webhook
npm run rss Legacy script (deprecated)

Design Goals

This project aims to provide:

"A terminal-first market observation tool"

Rather than:

"Yet another news aggregator"

Focus: Real-time monitoring and pattern observation, not exhaustive collection.

Inspiration: Trading-desk terminals and market intelligence systems, adapted for personal use.

Disclaimer

This is an experimental personal project for educational and research purposes.

Important notes:

  • Not financial advice: This tool aggregates publicly available news. It does not provide financial, investment, or trading advice.
  • No guarantees: News sources may be unavailable, delayed, or inaccurate. RSS feeds can break or change without notice.
  • Third-party services: Relies on external RSS feeds, APIs, and services that are outside our control.
  • Use at your own risk: No warranty of any kind. Check source reliability independently.
  • Experimental features: Translation, AI features, and integrations are experimental and may not work reliably.

Trademarks: This project is not affiliated with, endorsed by, or sponsored by Bloomberg, Reuters, CNBC, or any news organizations mentioned. All trademarks belong to their respective owners.

Contributing

Contributions, issues, and feature requests are welcome! This is an open-source project and community improvements are encouraged.

Please note:

  • This is a personal/experimental project with no guaranteed maintenance schedule
  • Breaking changes may occur as the project evolves
  • Test your changes thoroughly before submitting PRs

License

ISC - See LICENSE file for details.

Summary: Free to use, modify, and distribute. No warranty provided.

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages