finviz-crawler

Why This Skill?

📰 Your own financial news database — most finance skills just wrap an API for one-shot queries. This skill runs continuously, building a local archive of every headline and article from Finviz. Query your history anytime — no API limits, no missing data.

🆓 No API key, no subscription — scrapes finviz.com directly using Crawl4AI + RSS. Bloomberg, Reuters, Yahoo Finance, CNBC articles extracted automatically. Zero cost.

🤖 Built for AI summarization — the query tool outputs clean text/JSON optimized for LLM digests. Pair with an OpenClaw cron job for automated morning briefings, evening wrap-ups, or weekly investment summaries.

💾 Auto-cleanup — configurable expiry automatically deletes old articles from both the database and disk. Set --expiry-days 30 to keep a month of history, or 0 to keep everything forever.

🔄 Daemon architecture — runs as a background service that starts/stops with OpenClaw. No manual intervention needed after setup. Works with systemd (Linux) and launchd (macOS).

Install

CODEBLOCK0

Works on macOS, Linux, and Windows. Installs Python packages (crawl4ai, feedparser), sets up Playwright browsers, creates data directories, and verifies everything.

Manual install

Usage

Run the crawler

Query articles

Manage tickers

Tickers are stored in the tickers table inside finviz.db alongside articles. The crawler reads this table each cycle to know which ticker pages to scrape.

Configuration

💬 Chat Commands (OpenClaw Agent)

When this skill is installed, the agent recognizes /finviz as a shortcut:

The agent runs these via the finviz_query.py CLI internally.

📱 PrivateApp Dashboard

A companion mobile dashboard is available in PrivateApp — a personal PWA dashboard for your home server.

The Finviz app provides:

• - Headlines browser with time-range filters (12h / 24h / Week)
• Ticker-specific news filtering
• LLM-powered summaries on demand

Install PrivateApp, and the Finviz dashboard is built-in — no extra setup needed.

Architecture

Crawler daemon (finviz_crawler.py):

• - Crawls finviz.com/news.ashx headlines every 5 minutes
• Fetches article content via Crawl4AI (Playwright) or RSS (paywalled sites)
• Bot/paywall detection rejects garbage content
• Per-domain rate limiting, user-agent rotation
• Deduplicates via SHA-256 title hash
• Auto-expires old articles (configurable)
• Clean shutdown on SIGTERM/SIGINT

Query tool (finviz_query.py):

• - Read-only SQLite queries (no HTTP, stdlib only)
• Filter by time window, export titles or full content
• Designed for LLM summarization pipelines

Run as a service (optional)

systemd (Linux)

launchd (macOS)

Data layout

Cron integration

Pair with an OpenClaw cron job for automated digests:
CODEBLOCK8