finlab.data

finlab.data downloads every FinLab dataset: prices, fundamentals, filings and more. Each dataset is a table with one row per date and one column per stock. The usual workflow is to pick the market with data.set_market(), find dataset keys with data.search(), download them with data.get(), and narrow the stocks with data.us_universe() or data.universe().

Which function to use

First-time users should read Data Download. To turn downloaded data into stock-selection conditions, continue with FinlabDataFrame.

Download data

data.get() loads a dataset by its "table:field" name:

from finlab import data

data.set_market('us')

# Download adjusted closing prices
close = data.get('price:adj_close')
raw_close = data.get('price:close')  # unadjusted price for valuation

# Download quarterly revenue (aligned to SEC filing dates)
revenue = data.get('us_income_statement:revenue')

# Trailing-12-month P/E from diluted EPS
eps = data.get('us_income_statement:eps_diluted')
pe_ratio = raw_close / eps.rolling(4).sum()

Each row's date is the day that data was published and available to you, so quarterly fundamentals sit on their filing dates.

get(dataset, save_to_storage, force_download, lazy, progress, progress_callback, start, end, as_of, version) -> pd.DataFrame

Common U.S. datasets

Call data.set_market('us') first; generic price:* keys then resolve to us_price:*.

Ratios such as P/E, P/B and ROE are computed from these tables (for example P/E as unadjusted price:close divided by eps.rolling(4).sum()). The us_ratios:* and us_key_metrics:* tables hold only recent daily snapshots, so use them for current screening, not historical backtests. Browse everything in the data catalog.

English price keys in both markets

price:open, price:high, price:low, price:close, and price:volume work in both markets. After data.set_market("us") they return U.S. prices; under the default Taiwan market or data.set_market("tw") they return Taiwan daily prices and share the cache with the original Taiwan keys. The Taiwan aliases also work with data.gets(), lazy=True, and start/end date filters. volume means shares traded.

from finlab import data

data.set_market("us")
us_close = data.get("price:close")
data.set_market("tw")
tw_close = data.get("price:close")

Taiwan fields outside these five aliases keep their original keys.

Find tables and fields

data.search()

Searches the FinLab data catalog. The keyword matches dataset keys and, when the catalog provides them, each dataset's name, description, aliases and table columns. Matching ignores case and full-width/half-width differences. No dataset is downloaded.

from finlab import data

# U.S. fields about equity: dataset names, descriptions and columns are matched
data.search('equity', market='us')

# Price fields of U.S. ETFs and funds
data.search('close', market='us_fund')

# Which U.S. datasets are free and which need VIP
data.search('income', market='us', details=True)

# Every U.S. stock field
all_fields = data.search(market='us')
print(f"Total {len(all_fields)} fields")

# List Taiwan stock fields
data.search(market='tw')

Returns a pd.Series of dataset names (name="dataset"), or a pd.DataFrame when details=True. When results include VIP-only datasets, a hint to use details=True is printed. When requires_date_range is True, data.get() needs both start and end; those intraday keys stand for dataset families, not a full symbol list. Missing catalog permissions stay unknown; search does not grant access.

data.describe()

Returns a dataset's or a whole table's layout and description, without downloading it:

from finlab import data

data.describe('us_price:close', market='us')
data.describe('us_income_statement:revenue', market='us')
data.describe('us_income_statement', market='us')  # the whole table

Pass a table name such as us_income_statement instead of a full key to get the table's columns and description. This query shares the search catalog cache and downloads no dataset. layout describes the published dataset; source_layout is the upstream storage format only when explicitly documented. Missing metadata returns None. The update schedule does not establish observation frequency. index_end is the last index label and may be a future period date; source_created_time is the artifact creation time. Neither establishes row availability or latest_data_date.

columns lists fields published in the catalog. An unlisted key_date returns catalogued=False, key_date_available=None, and category_description. This does not establish whether it can be downloaded or whether its values are announcement dates. Other unknown names raise KeyError; an unavailable or empty catalog raises RuntimeError.

describe(dataset, market) -> dict[str, Any]

Select the market

data.set_market()

Sets the market that later data.get() calls and backtests use. The default is Taiwan.

from finlab import data

data.set_market('us')        # U.S. stocks
close = data.get('price:adj_close')

data.set_market('us_fund')   # U.S. ETFs and funds
fund_close = data.get('price:adj_close')

Limit the stock universe

data.us_universe() and data.universe() are with blocks: inside, data.get() returns only the matching stocks, and the previous selection is restored on exit.

data.us_universe()

Limits U.S. data to an index's constituents, sectors, industries, exchanges or security types.

from finlab import data

data.set_market('us')

# Get the full price history of the latest S&P 500 constituents
with data.us_universe(index='SP500'):
    close = data.get('price:close')

# Get Technology stocks among the latest NASDAQ 100 constituents
with data.us_universe(index='NDX', sector='Technology'):
    close = data.get('price:close')

# Get a clean U.S. common-stock universe
with data.us_universe(security_type='common_stock'):
    close = data.get('price:close')

# Can also use data.universe() with set_market
with data.universe(index='SP500', exchange='NASDAQ'):
    close = data.get('price:close')

Supported index values:

index can be combined with sector, industry, exchange, and security_type; the result is the intersection of all filters. U.S. sector names: Technology, Healthcare, Financial Services, Consumer Cyclical, Industrials, Communication Services, Consumer Defensive, Energy, Basic Materials, Real Estate, Utilities.

us_universe(sector, industry, exchange, security_type, index)

from finlab import data
with data.us_universe(sector='Technology', exchange='NASDAQ'):
    close = data.get('us_price:close')

with data.us_universe(security_type='common_stock'):
    close = data.get('us_price:close')

with data.us_universe(index='SP500'):
    close = data.get('us_price:close')

with data.us_universe(index='NDX', sector='Technology'):
    close = data.get('us_price:close')

with data.us_universe():  # Temporarily clear any existing universe filter.
    close = data.get('us_price:close')
    constituents = data.get('us_index_constituents:sp500')
in_index = constituents.reindex(close.index, method='ffill').eq(True)
score = close.pct_change(20)
position = score.where(in_index).is_largest(3)

Historical index backtests

index='SP500' or 'NDX' selects columns for the entire history using the last row of the constituent table. Backtesting these columns introduces survivorship and look-ahead bias: later additions appear before they joined, while removed stocks disappear from the entire history. Index changes can therefore change earlier results when the same strategy is rerun. The index parameter of data.universe(), data.set_universe() and data.set_us_universe() works the same way.

For historical selection, read unfiltered U.S. prices, apply the constituent record dated on or before each price date to scores, then rank stocks:

from finlab import data

with data.us_universe():  # Temporarily clear the existing universe; restore it on exit.
    close = data.get('us_price:close')
    constituents = data.get('us_index_constituents:sp500')
in_index = constituents.reindex(close.index, method='ffill').eq(True)
score = close.pct_change(20)
position = score.where(in_index).is_largest(3)

For NASDAQ 100, use us_index_constituents:nasdaq100. Dates before the first record and price symbols absent from the table are False; use each date's mask after that day's close. This example matches symbols exactly, so spelling differences, renamed stocks or missing price history can still exclude constituents. The mask is rebuilt from the records currently stored: source delays, gaps and later corrections still affect results. It is not a complete snapshot of what was known at the time.

data.universe()

Limits which securities data.get() returns, and restores the previous selection on exit. It dispatches on the active market: with data.set_market('us') it filters U.S. stocks like data.us_universe(); under the default Taiwan market it filters Taiwan listings.

from finlab import data

data.set_market('us')

# NASDAQ-listed stocks only
with data.universe(exchange='NASDAQ'):
    close = data.get('price:adj_close')

# Two sectors at once
with data.universe(sector=['Technology', 'Healthcare']):
    close = data.get('price:adj_close')

# A fixed list of tickers
with data.universe(whitelist=['AAPL', 'MSFT', 'NVDA']):
    close = data.get('price:adj_close')

Under the Taiwan market, exchange selects TWSE or TPEx listings and sector takes Taiwan industry names, which are written in Chinese in the Taiwan data.

exchange and sector filter the whole history with today's listing and classification: a stock that later moved from TPEx to TWSE counts as TWSE, not TPEx, before its move, and categories created later (such as those added in July 2023) apply to earlier dates too. For date-by-date listing eligibility in backtests, use universe.membership(); see Historical Taiwan listing eligibility.

data.reset_universe()

Disables the universe filter. data.set_universe(), data.universe(), and data.us_universe() without any filter do the same and drop no identifiers.

from finlab import data

data.set_market('us')
data.set_universe(sector='Technology')
filtered = data.get('price:adj_close')
data.reset_universe()
unfiltered = data.get('price:adj_close')

The reset applies to subsequent data.get() and data.gets() calls, including cached reads. It preserves cached datasets, the active market, date limits, and already returned DataFrames. Inside data.universe() or data.us_universe(), a reset lasts until another selection or context exit; exiting still restores the selection from before that context. Repeated resets are safe.

Historical Taiwan listing eligibility

Taiwan market only. universe.membership() returns a boolean FinlabDataFrame on the Taiwan close-price dates and columns. It supports TSE, OTC, and TSE_OTC without changing existing universe contexts or ranking methods.

from finlab import data
from finlab.data import universe

data.set_market('tw')
# Fetch scores without an existing snapshot universe removing securities.
close = data.get("price:close")
score = close.pct_change(20)
eligible = universe.membership("TSE_OTC")
position = score[eligible].is_largest(20)

Apply eligibility before every cross-sectional ranking. The mask cannot restore securities previously removed from score data. Membership reads bypass the snapshot universe without changing it and respect the active date window.

Intervals use current company records, historical delisted-company records, and exchange termination lists. Listing dates are inclusive; termination dates are exclusive. Multiple listing periods and exchange transfers are supported. Suspensions and missing trades do not terminate listing eligibility; apply trading conditions separately. Indicator warm-up data is not erased.

Unverified historical periods remain False with a warning. Inspect eligible.attrs["membership_missing_history"] for incomplete company histories. Securities without company listing records, such as ETFs, are not included merely because they have price columns; this is not a complete ETF universe.

This reconstructs effective-date eligibility from currently available records. It does not provide universe versions or as-known-at-the-time snapshots. Historical corrections can still change results, so it cannot guarantee reproduction of old backtests.

Compute technical indicators

data.indicator()

Computes a TA-Lib or pandas-ta indicator for every stock of the active market from its open, high, low, close and volume. Install TA-Lib or pandas-ta first.

from finlab import data

data.set_market('us')

# 14-day RSI of every U.S. stock
rsi = data.indicator('RSI', timeperiod=14)

# MACD has three outputs
macd, macd_signal, macd_hist = data.indicator('MACD')

# Weekly stochastic oscillator
k, d = data.indicator('STOCH', resample='W')

Returns one FinlabDataFrame (dates by symbols) per indicator output; a tuple when the indicator has several outputs (for example STOCH returns k, d).

Control download range and cache

Limit the date range

from finlab import data

# Only download data from 2020-2023 (saves memory)
data.truncate_start = '2020-01-01'
data.truncate_end = '2023-12-31'

# All subsequent data.get() calls will use this range
data.set_market('us')
close = data.get('price:adj_close')

Force cloud or local data

from finlab import data

# Force cloud download (re-download every time)
data.force_cloud_download = True

# Or use only the local cache (offline environment); never both at once
# data.use_local_data_only = True

data.set_storage()

Sets where downloaded data is kept locally so repeated data.get() calls do not download it again. FinLab provides two storages: CacheStorage keeps data in memory, and FileStorage (the default) writes it to files.

import pandas as pd

from finlab import data

# Use a custom directory
data.set_storage(data.FileStorage('/path/to/custom/cache'))

data.set_market('us')
close = data.get('price:adj_close')

# The file can be read directly with pandas
close_from_disk = pd.read_feather('/path/to/custom/cache/us_price#adj_close.feather')

# Check what is stored locally
data._storage.diagnose()

data.CacheStorage()

Keeps downloaded data in memory only, for the current Python session.

from finlab import data

data.set_storage(data.CacheStorage())

data.FileStorage()

Writes downloaded data as Feather files, one file per dataset, with : in the dataset key replaced by # (for example us_price#adj_close.feather).

from finlab import data

data.set_storage(data.FileStorage('/path/to/custom/cache', use_cache=False))

Estimate and record download usage

Paid plans can use data.plan_download(key) for a quota-free ordinary whole-dataset check. Each full/recent/delta candidate has a row with the planned route (cache/recent/delta/full), needs_update (null when unknown), size_bytes and planned_bytes. data.download_usage() is a with block that collects a usage row for each download inside it. For a list:

import pandas as pd
from finlab import data

keys = ['us_price:close', 'us_price:volume']
plan = pd.concat([data.plan_download(key) for key in keys], ignore_index=True)
plan.to_csv('download-plan.csv', index=False)
with data.download_usage() as rows:
    for key in keys:
        data.get(key)
receipts = pd.DataFrame(rows)
receipts.to_csv('download-usage.csv', index=False)
receipts.to_json('download-usage.json', orient='records')

Preflight never signs a download URL. Older servers reject it without falling back to a chargeable download probe. This is a snapshot: hash/merge fallback may add a full download and concurrent publication may change sizes. Unknown values stay null, never an inferred zero or guaranteed usage ceiling. Planning excludes intraday, retained versions and start/end optimisations.

For event='provider', billed_bytes is the provider's per-item charge. Signing already counts even if transfer or merge fails. quota is the shared daily account total in MiB, not a per-item bill. Fallback rows explain the reason; cache rows report local reads. Attempts begin with unknown bytes: lost/retried signing replies and older providers may leave them unknown. A ledger with unknown provider amounts is not a complete bill. Rows survive an exception; export pd.DataFrame(rows) in finally. Collection adds no requests and includes ordinary get/gets workers in this scope, excluding other processes.

Handle download errors

Mistyped keys: when data.get() cannot find a dataset it raises DatasetNotFoundError (a RuntimeError subclass) listing the closest valid keys. If the key is a registered equivalent legacy name, it emits a UserWarning and fetches the correct key instead.

Download failures: data.get() raises SignedDownloadError (a DataError subclass). The message names the dataset, the HTTP status or kind of connection problem, and what to do; it never contains the download URL. To branch on the cause, read e.kind ("http", "timeout", "connection", "ssl", "redirects", "browser", "invalid_reply", "other") and e.status (the HTTP status, None when no response arrived) instead of matching the message. For compatibility it also keeps the requests type raised before (such as requests.HTTPError or requests.Timeout) and is a requests.RequestException and OSError; new code should use except DataError.

Daily download quota: free and VIP accounts receive one warning per session when a download response reports daily usage at or above 80%. When the quota is exhausted, data.get() / data.gets() raise QuotaExceededError, a subclass of both DataError and RuntimeError. The message includes the limit and the daily reset time, 08:00 UTC+8 (00:00 UTC). In batch backtests, catch this exception before general backtest errors to distinguish exhausted quota from a failed strategy.

Check your account and strategies

Check the account role

from finlab import data

# Query user role (requires at least one data.get() call first)
data.set_market('us')
close = data.get('price:adj_close')

role = data.get_role()  # 'free', 'vip', ...
print(f"Current role: {role}")

if data.is_vip():
    print("VIP member: full data access enabled")

get_role() -> str | None

is_vip() -> bool

data.get_strategies()

Returns the data of the strategies you uploaded to FinLab, the same numbers shown on your strategy dashboard, so several strategies can be combined in your own code.

from finlab import data

strategies = data.get_strategies()
for name, info in strategies.items():
    print(name, info['stats']['daily_sharpe'])

Returns a dict keyed by strategy name (an empty dict when authentication fails). Each value contains, among others:

Common questions

from finlab import data

# List all U.S. stock fields
all_fields = data.search(market='us')
for field in all_fields[:10]:
    print(field)

from finlab import data

data.set_market('us')

# Method 1: Limit time range
data.truncate_start = '2020-01-01'

# Method 2: Use cache (second call will be fast)
close = data.get('price:adj_close')  # First call downloads
close = data.get('price:adj_close')  # Second call reads the local cache

# Method 3: Limit the stock universe
with data.us_universe(index='SP500'):
    close = data.get('price:adj_close')  # Only S&P 500 columns

import finlab

# Check if logged in
try:
    token, token_type = finlab.get_token()
    print(f"Logged in ({token_type})")
except Exception:
    print("Not logged in, please run finlab.login()")

from finlab import data

# ETFs and funds are a separate market
data.set_market('us_fund')
etf_close = data.get('price:adj_close')[['SPY', 'QQQ']]

# Search U.S. ETF/fund fields
fund_fields = data.search(market='us_fund')

from finlab import data

data.set_market('us')
close = data.get('price:adj_close')

# Check missing values
print(f"Missing value ratio: {close.isna().sum().sum() / close.size:.2%}")

# Fill missing values
close_filled = close.ffill()  # Forward fill

# Or drop stocks with too many missing values
close_clean = close.dropna(
    axis=1, thresh=int(len(close) * 0.8)
)  # Keep stocks with 80%+ data

from finlab import data

# Method 1: Limit time range
data.truncate_start = '2020-01-01'

# Method 2: Process in batches
data.set_market('us')
close = data.get('price:adj_close')
all_stocks = close.columns
for batch in [all_stocks[i : i + 100] for i in range(0, len(all_stocks), 100)]:
    batch_close = close[batch]
    # Process 100 stocks at a time...

# Method 3: Only download the fields you need
# Avoid calling data.get() for too many tables at once

from finlab import data

with data.universe():  # Temporarily clear any existing universe filter.
    close = data.get('price:\u6536\u76e4\u50f9')
    underlying = data.get('single_stock_futures_and_equity_options_underlying')
    volume = data.get('futures_price:\u6210\u4ea4\u91cf')
underlying = underlying[
    underlying['\u662f\u5426\u70ba\u80a1\u7968\u671f\u8ca8\u6a19\u7684'] == 'Y'
]
contracts = (
    underlying[
        '\u80a1\u7968\u671f\u8ca8\u3001\u9078\u64c7\u6b0a\u5546\u54c1\u4ee3\u78bc'
    ].astype(str)
    + 'F\u4e00\u822c'
)
traded = volume.reindex(columns=contracts).gt(0)
traded.columns = underlying['symbol'].astype(str)
listed = traded.T.groupby(level=0).any().T.cummax()
has_futures = listed.reindex(close.index, method='ffill', fill_value=False)
score = close.pct_change(20)
position = score.where(has_futures).is_largest(10)

Next steps