Architecture

Data flow

On this page8

What happens after you call data.get(): where the data comes from, when it updates and how it is stored on your machine. This page walks through the flow and the fastest ways to load data.

How data reaches you

The first read of a dataset downloads it from the cloud into a local cache. Later reads use the cache and download again only when the cloud has a newer revision.

Data sources and update times

Category	Data source	Update frequency	Data range
Price (`us_price`)	NYSE, NASDAQ and NYSE American listings	Daily, after the U.S. close	2016 to present
ETF and fund prices (`us_fund_price`)	U.S.-listed ETFs and mutual funds	Daily, after the U.S. close	2016 to present
Financial statements (`us_income_statement`, `us_balance_sheet`, `us_cash_flow`)	SEC 10-Q / 10-K filings	After each filing; aligned to the filing date	2015 to present
Technical indicators	Calculated by FinLab	Computed on request	Depends on price data

U.S. prices and fundamentals update once a day after the U.S. market close.

Category	Data source	Update frequency	Data range
Price	TWSE/TPEx	Daily at 18:00 Taiwan time	2007 to present
Financial statements	MOPS (Market Observation Post System)	After each quarterly report	2000 to present
Monthly revenue	MOPS	After the 10th of each month	2000 to present
Institutional trading	TWSE	Daily at 18:00 Taiwan time	2010 to present
Technical indicators	Calculated by FinLab	Computed on request	Depends on price data

The daily update timeline (Taiwan time):

The local cache: download once, then read from disk

Where the cache lives

Downloaded datasets are cached in ~/finlab_db (in Google Colab, My Drive/Colab_tmp/finlab_db when Google Drive is mounted). To use another directory:

from finlab import data

data.set_storage(data.FileStorage('/path/to/custom/cache'))

How updated data reaches you

On each data.get() call, FinLab compares the local cache with the revision published in the cloud, then uses the cache, merges only the recent data, or downloads the full dataset again:

Published revisions come from a shared index fdata publishes, checked at most every five minutes. The result is kept in the default data directory (~/finlab_db), so a new process on the same machine within those five minutes reuses it without connecting again. When the index is unavailable, FinLab asks the signing service instead.

Updated or corrected data arrives within about five minutes; just call data.get again (finlab 2.2.5+).

The cache for free accounts

Free accounts receive different data from paid accounts (Taiwan history only up to the free cutoff), so their cache lives in a free subfolder of the default data directory, shared by every process on the machine and never read by a paid account.

Free accounts skip the revision index: each read reports the cache's time to the signing service. When no newer file exists, the local cache is used without a download and without using any daily download quota; otherwise it downloads again.

Incremental updates for large tables

Large full tables (such as broker_transactions and important_info_announcement) also publish a <dataset>.delta: a cache no older than its full_refresh_before downloads only the days that changed recently, replaces its rows on those days and verifies the whole-table hash; an older cache downloads the full table and then applies the delta.

A hash mismatch falls back to a full download, so inconsistent data is never returned.

Free memory or clear the cache

from finlab import data

# Release cached data from RAM (files on disk are kept)
data.free_memory()

# Delete the local cache and restore the default settings (every dataset must then be
# downloaded in full again; only needed when the cache is corrupted)
# data.clear()

Load data faster

Load once, reuse many times

Bad practice: calling data.get() for every backtest.

data.set_market('us')

# Loading the data again for every backtest (slow!)
for param in [20, 60, 120]:
    with data.us_universe(index='SP500'):
        close = data.get('price:adj_close')  # Redundant loading
    position = close > close.average(param)
    report = sim(position, resample='M')

Good practice: load once, then reuse.

from finlab import data
from finlab.backtest import sim

data.set_market('us')

# Load only once
with data.us_universe(index='SP500'):
    close = data.get('price:adj_close')

# Reuse it for every backtest
for param in [20, 60, 120]:
    position = close > close.average(param)
    report = sim(position, resample='M')

Keep only the dates you need

# Keep only data after 2020
close = data.get('price:adj_close')
close_recent = close[close.index >= '2020-01-01']

# Or use loc
close_recent = close.loc['2020-01-01':]

Keep only the stocks you need

# Keep only specific stocks
close = data.get('price:adj_close')
close_subset = close[['AAPL', 'MSFT', 'NVDA']]

# Or use filter conditions
market_cap = data.get('etl:us_market_value')
large_cap = market_cap > 100_000_000_000  # Market cap > $100 billion
close_large = close[large_cap]

Compute all stocks at once

Bad practice: loading and computing one stock at a time.

for stock in ['AAPL', 'MSFT', 'NVDA']:
    close = data.get('price:adj_close')[stock]
    ma20 = close.rolling(20).mean()  # Recomputed each time

Good practice:

close = data.get('price:adj_close')[['AAPL', 'MSFT', 'NVDA']]
ma20 = close.rolling(20).mean()  # Compute all stocks at once

Handle missing values and outliers

Missing values

from finlab import data

data.set_market('us')
close = data.get('price:adj_close')

# Check missing values
missing_ratio = close.isna().sum() / len(close)
print(f"Missing-value ratio:\n{missing_ratio[missing_ratio > 0.1]}")

# Method 1: Forward fill (recommended)
close_filled = close.ffill()

# Method 2: Remove stocks with missing values
close_clean = close.dropna(axis=1, how='any')

# Method 3: Remove dates with no data for any stock
close_clean = close.dropna(axis=0, how='all')

Outliers: implausible daily moves

Reverse-split artifacts and bad prints show up as implausible daily moves. Mask them before computing signals:

close = data.get('price:adj_close')

# Daily moves beyond +/-50% are almost always data artifacts
daily_return = close.pct_change()
outlier = daily_return.abs() > 0.5

# Remove the affected dates
close_filtered = close[~outlier]

Common datasets

from finlab import data

data.set_market('us')

# Prices
adj_close = data.get('price:adj_close')  # Adjusted close (accounting for dividends and stock splits)
raw_close = data.get('price:close')  # Raw close
volume = data.get('price:volume')  # Volume (shares)
market_value = data.get('etl:us_market_value')  # Market value (USD)

# Financial statements: quarterly statements are aligned to each SEC filing date
eps = data.get('us_income_statement:eps_diluted')  # Diluted earnings per share

# Return on equity
net_income = data.get('us_income_statement:net_income')
equity = data.get('us_balance_sheet:total_stockholders_equity')
roe = net_income / equity

# Operating margin
revenue = data.get('us_income_statement:revenue')
operating_margin = data.get('us_income_statement:operating_income') / revenue

# Revenue year-over-year growth
rev_yoy = revenue.pct_change(4)

Monthly revenue, institutional investor trading and margin trading data exist for the Taiwan market only. Find their dataset keys with data.search() under data.set_market('tw').

Save to CSV and load it back

close = data.get('price:adj_close')

# Save the entire DataFrame
close.to_csv('close_prices.csv')

# Save specific stocks
close[['AAPL', 'MSFT']].to_csv('selected_stocks.csv')

After loading it back, convert it to a FinlabDataFrame to keep using FinLab methods:

import pandas as pd
from finlab.dataframe import FinlabDataFrame

# Load CSV
close = pd.read_csv('close_prices.csv', index_col=0, parse_dates=True)

# Convert to FinlabDataFrame
close = FinlabDataFrame(close)

# Now you can use FinLab methods
ma20 = close.average(20)

Next steps

Download data Everything data.get() can do, and how to find a dataset's name.
finlab.data The API reference for downloading data and cache settings.
FinlabDataFrame The methods available on the tables you load.
Module architecture What happens once data reaches the backtest, analysis and order modules.

Was this page helpful?