Building a Quant Data Pipeline in Python With FRED + Finnhub

I built an AI trading bot that needs to know what's happening in the economy, what stocks are doing right now, and how scared or greedy the market is feeling — all at the same time, all day, every day. That means pulling information from three completely different sources and combining it into one clear picture before every trade.

Getting data from one source is easy. Keeping three sources running reliably, knowing when data is stale, and handling failures gracefully — that's where most systems fall apart.

Three Sources, One Assembly Line

Think of it like running a restaurant kitchen. You need produce from a farm, meat from a butcher, and spices from a specialty supplier. Each one delivers on a different schedule, in different packaging, with different reliability. Your job is to make sure the chef always has fresh ingredients — and knows immediately if something showed up expired.

My three "suppliers" are:

Economic data from FRED — This is a massive government database with 800,000+ data points about the U.S. economy. I pull about 15-20 key indicators: interest rates, inflation, unemployment, GDP. This tells the bot whether the big-picture economy is healthy or deteriorating. It updates slowly — some numbers are monthly, some quarterly — but it provides critical context.

Stock market data from Finnhub — Real-time stock prices, company information, whether the market is even open right now. This changes by the second during trading hours. I get 60 data requests per minute on their free plan, which is plenty when the system is designed to ask only for what it needs, when it needs it.

Market sentiment from CNN's Fear & Greed Index — A single number from 0 to 100 that captures how fearful or greedy investors are feeling. When it drops below 25 (extreme fear), history says it's often a good time to buy. Above 75 (extreme greed), time to be cautious. It's a blunt tool, but its simplicity is a strength.

Each source has its own quirks. Economic data gets revised months after it's first released. Stock quotes can arrive stale for low-volume tickers. The Fear & Greed Index doesn't have an official way for programs to access it, so I have to scrape their webpage — which has broken twice when they redesigned their site.

The Smart Manager That Keeps Everything Running

The heart of the system is what I call a sentinel — think of it as a shift manager for the assembly line. It runs 24/7 and makes smart decisions about what information to fetch and when.

The sentinel knows that economic data only needs refreshing once a day (GDP doesn't change by the hour). It knows stock prices matter most when the market is open, so it checks those frequently during trading hours and barely at all overnight. It knows sentiment data updates once a day but checks more often when markets are volatile.

Every 10 seconds, the sentinel wakes up, looks at what's getting stale, batches its requests to stay within rate limits, grabs what it needs, and goes back to sleep. It's like a good manager who checks in constantly but only acts when something needs attention.

When things break — and they always break — the sentinel handles it:

• If the economic data source goes down, it serves the last known good data and flags it as stale.
• If stock data hits a rate limit, it backs off gradually and reduces request frequency.
• If the sentiment scraper fails because the website changed, it serves the cached value and alerts me. Both times this happened, it took about 20 minutes to fix. No bad data ever reached the trading bot.

This whole thing runs on a server that costs $5 a month. It's been running for months with 99.7% uptime. The only downtime was me making updates, not the system failing.

Why This Matters Beyond Trading

Every value that reaches the trading bot carries a label: what the number is, when it was fetched, when the source last updated it, and how confident the system is in its accuracy. The bot always knows exactly how fresh its information is. A 24-hour-old economic indicator is fine. A 24-hour-old stock price is not. The system makes this explicit.

The real lesson from building this: getting data from a source is about 5% of the work. The other 95% is making it reliable, keeping it fresh, handling failures gracefully, and making sure bad data never silently poisons your decisions.

I've applied this same pattern across my DTC fashion brand — for competitor price tracking, SEO monitoring, and dynamically pricing 564 products. The specific data changes. The architecture doesn't.

The AI model that makes trading decisions gets all the attention. But the assembly line feeding it clean, fresh, trustworthy data is what actually makes it work.

Thinking About AI for Your Business?

If this resonated, let's have a conversation. I do free 30-minute discovery calls where we look at your operations and identify where AI could actually move the needle.

Book a Discovery Call