Download data

Every FinLab dataset downloads with data.get('table:column'), for all stocks and 10+ years of history at once. This page shows how to download, how to find a dataset's name, how to keep only the stocks you need, and how to fix download errors.

New to FinLab? Start with the Quickstart. For full function signatures, see the finlab.data API reference.

Download a dataset

One data.get() call downloads over 10 years of history for thousands of stocks:

from finlab import data

data.set_market('us')
close = data.get('price:adj_close')

You get back a table:

• Each row is a market trading date.
• Each column is a ticker, so a stock-selection rule can compare whole tables at once.

The type is FinlabDataFrame, an extended Pandas DataFrame: everything you know from Pandas works, plus extra tools for stock selection (see Compute with the data). New to Pandas? Read the Pandas 10-minute guide first.

from finlab import data

data.set_market('tw')
close = data.get('price:close')

What free and VIP accounts get

• Free users can download most datasets for backtesting, but recent data is not included. Free Taiwan historical data ends on 2023-12-31 (last trading day: 2023-12-29).
• VIP receives the latest published data: daily prices normally cover the previous trading day, and other datasets follow their own update schedules. Upgrade to VIP for the latest data.
• U.S. data and open reference tables have their own coverage, shown in the data catalog.
• VIP only: the Taiwan convertible-bond datasets cb_price, cb_published_info, and cb_converted_status. After the trial ends, downloading them requires a subscription; programs using them receive a permission error.

To see whether a dataset needs VIP and how far free data goes, use data.search(..., details=True), described in the next section.

Find a dataset's name

Three ways to find the key you need:

• Browse the data catalog for available datasets and their key names.
• Install the FinLab Skill in your AI coding assistant and ask "What revenue-related datasets are available?" to get the correct data.get() parameters and usage examples.
• Search in code with data.search(keyword=None, market=None), which returns names in table:column format:

from finlab import data

# U.S. market keyword search
close_us = data.search('close', market='us')

# U.S. revenue and earnings fields
revenue_us = data.search('revenue', market='us')

# List all U.S. market fields
all_us = data.search(market='us')

# Search all markets
price_all = data.search('price', market='all')

Check permissions and free cutoff dates

With details=True, each dataset comes with extra columns:

from finlab import data

data.search('cb', details=True)

Without details=True, data.search() still returns dataset keys.

Limit downloads to an index or sector

Wrap data.get() in a data.us_universe() block to filter by index, sector, industry, exchange, or security type. For example, Technology stocks in the S&P 500:

from finlab import data

data.set_market('us')
with data.us_universe(index='SP500', sector='Technology'):
    close = data.get('price:adj_close')

Every data.get() call inside the with block returns only the selected columns; the previous universe is restored when the block exits.

U.S. sectors: Technology, Healthcare, Financial Services, Consumer Cyclical, Industrials, Communication Services, Consumer Defensive, Energy, Basic Materials, Real Estate, Utilities. The values come from data.get('us_company_profile') (columns sector, industry, exchange, security_type).

Match one sector exactly

sector and industry use regular expressions, so a pattern matches every name that contains it. Anchor the pattern to match one name exactly, for example sector='^Energy$'.

When a filter matches nothing

A filter that matches no securities (for example a misspelled sector) raises a ValueError and leaves the previous universe unchanged. Use data.reset_universe() to remove a universe set with data.set_universe().

from finlab import data

data.set_market('tw')
# Listed and OTC stocks, excluding the ETF category
with data.universe(market='TSE_OTC', exclude_category=['ETF']):
    close = data.get('price:close')

Log in on a server or scheduled job

Log in before downloading. Jupyter/Colab, local Python, and VSCode can all use the browser-based login flow:

import finlab

finlab.login()

For headless or automated environments, log in with environment variables instead, which skips the browser prompt:

python -m finlab token --env

set FINLAB_REFRESH_TOKEN=<YOUR_REFRESH_TOKEN>
set FINLAB_SESSION_ID=<YOUR_SESSION_ID>
set FINLAB_API_KEY=<YOUR_API_KEY>

export FINLAB_REFRESH_TOKEN=<YOUR_REFRESH_TOKEN>
export FINLAB_SESSION_ID=<YOUR_SESSION_ID>
export FINLAB_API_KEY=<YOUR_API_KEY>

Compute with the data

data.get() returns a FinlabDataFrame, which extends Pandas DataFrame with all standard Pandas operations plus built-in accessors for stock selection (.cs, .sector, .weight).

Returns, moving averages and conditions

from finlab import data

data.set_market('us')
close = data.get('price:adj_close')
rev = data.get('us_income_statement:revenue')  # quarterly

# Arithmetic
ret = close / close.shift() - 1  # daily returns
rev_yoy = rev / rev.shift(4) - 1  # quarterly revenue YoY growth

# Rolling calculations
ma20 = close.average(20)  # 20-day moving average (FinLab extension)
std20 = close.rolling(20).std()  # 20-day volatility

# Comparison -> boolean DataFrame
cond = close > ma20  # price above MA
golden_cross = ma20 > close.average(60)  # golden cross

# Logical combination
position = cond & golden_cross  # both conditions
position = cond | golden_cross  # either condition

Pick the largest or smallest names

eps = data.get('us_income_statement:eps_diluted')
raw_close = data.get('price:close')  # unadjusted price for valuation
pe = raw_close / eps.rolling(4).sum()  # trailing-12-month P/E
pe = pe[pe > 0]  # drop loss-making companies

position = pe.is_smallest(10)  # lowest 10 PE stocks
position = pe.is_largest(10)  # highest 10 PE stocks
ranked = pe.rank(axis=1, pct=True)  # per-date percentile rank

Rank and standardize across stocks each day (.cs)

df.cs (cross-sectional) operates across all stocks on each date. Useful for factor ranking and standardization:

net_income = data.get('us_income_statement:net_income')
equity = data.get('us_balance_sheet:total_stockholders_equity')
roe = net_income / equity  # quarterly ROE

roe.cs.rank()  # per-date percentile rank
roe.cs.zscore()  # per-date standardization (mean 0, std 1)
roe.cs.winsorize(0.05, 0.95)  # per-date percentile winsorization
roe.cs.bucket(5)  # per-date quintile bucketing
roe.cs.demean()  # per-date de-mean

# Select top 20% ROE stocks
position = roe.cs.rank() > 0.8

Compare within a sector (.sector)

df.sector computes within the same sector, useful for sector-neutral analysis. Pass a symbol-to-sector mapping for U.S. stocks; without a mapping, .sector uses the Taiwan industry classification:

profile = data.get('us_company_profile')
sectors = dict(zip(profile['symbol'], profile['sector']))

# Transform: same methods as .cs, but within sector
roe.sector(sectors).rank()  # within-sector rank
roe.sector(sectors).zscore()  # within-sector standardization (about the same as neutralization)

# Aggregate: sector-level statistics
roe.sector(sectors).mean()  # sector average ROE
roe.sector(sectors).std()  # sector ROE standard deviation
roe.sector(sectors).median()  # sector median

# Select top 30% ROE within each sector
position = roe.sector(sectors).rank() > 0.7

Set each holding's weight (.weight)

df.weight builds refined weights from selection results, and the steps chain:

position = roe.cs.rank() > 0.8  # select stocks first

# Chain weight construction
weights = (
    position.weight.inverse_volatility(window=60, price=close)  # inverse-volatility weighting
    .rebalance('ME')  # keep rebalance dates only, so turnover is per rebalance
    .weight.limit_turnover(max_turnover=0.3)  # cap turnover at 30%
)

Pass price=close for U.S. stocks: without it, inverse_volatility reads Taiwan adjusted prices. weight.cap_industry() caps weights per Taiwan industry category.

See the finlab.dataframe reference for all methods.

Download many datasets

data.gets() downloads several datasets in parallel, which is faster, and returns regular FinlabDataFrame objects. When you are done, data.free_memory() releases memory:

import os
from finlab import data

os.environ['FINLAB_MAX_DOWNLOAD_WORKERS'] = '1'
data.set_market('us')
close, volume = data.gets('price:adj_close', 'price:volume')

del close, volume  # Remove your references when the frames are no longer needed.
data.free_memory()

Download threads: FINLAB_MAX_DOWNLOAD_WORKERS limits Python download threads for data.get() and data.gets(). It defaults to 8 and must be a positive integer. Set it to 1 to download sequentially in the calling thread when your environment limits threads. The setting is read at download time; Pyodide keeps its existing download behavior.

Releasing memory: data.free_memory() releases data and category caches in RAM, keeping FileStorage files, timestamps and current settings. Later reads can reload from disk. Keep in mind:

• CacheStorage has no disk backup and must download again after release.
• DataFrames you still reference remain alive, so del them first.
• Python may not immediately return memory to the operating system.

Show or report download progress

data.get() and data.gets() accept progress='display' | 'event' | 'silent'. Agents and workers should use event or silent mode to avoid creating Jupyter HTML display objects.

from finlab import data

events = []
data.set_market("us")
close, volume = data.gets(
    "price:adj_close",
    "price:volume",
    progress="event",
    progress_callback=events.append,
)

The event schema is {phase, dataset?, completed?, total?, cache_hit?, download_bytes?, elapsed_ms, timestamp}.

Update every dataset daily

When you refresh datasets every day, requires_date_range splits them in two:

• False: call data.get(key) directly, subject to authentication, data permissions, and download quota.
• True: a date-partitioned dataset; data.get() requires both start and end.

from finlab import data

catalog = data.search('', market='tw', details=True)  # Use market='us' for U.S. data
whole_tables = catalog.loc[~catalog['requires_date_range'], 'dataset']

# Update ordinary datasets using your existing FileStorage configuration
for key in whole_tables:
    data.get(key)

# Request date partitions separately, choosing a symbol and published dates
ticks = data.get('tw_tick:2330', start='2026-06-01', end='2026-06-01')

tw_tick:2330, tw_minute:2330, and tw_futures_minute:TX_202610 are examples of three date-partitioned families, not a complete symbol directory. Replace the stock or contract and dates as needed.

The flag describes required date arguments, separately from data frequency: intraday_trading contains daily data, while tw_price_60min is a downloadable wide table of 60-minute bars. Both have False; do not exclude datasets by name or frequency.

Fix common errors

DatasetNotFoundError: the dataset does not exist

data.get() raises finlab.exceptions.DatasetNotFoundError (a RuntimeError subclass), and the message lists the closest valid keys:

from finlab import data
from finlab.exceptions import DatasetNotFoundError

data.set_market('us')
key = 'price:adj_closing'  # Wrong: incorrect field name
try:
    close = data.get(key)
except DatasetNotFoundError as e:
    print(e)  # the message lists the closest valid keys, such as us_price:adj_close

Common causes:

• The name is misspelled (the correct name is adj_close, not adj_closing).
• The dataset does not exist in the database.
• Login has not been completed or the login state is invalid, preventing data access.

Search for the correct name with data.search('close', market='us'), or browse the data catalog. If the name is right and the download still fails, check that you have run finlab.login() successfully, that the network connection works, and that you have permission to access the dataset.

The download succeeds but the table is empty

close = data.get('price:adj_close')
print(close)
# Empty DataFrame
# Columns: []
# Index: []

Common causes:

• data.truncate_start / data.truncate_end exclude every date of the dataset.
• A free user's data is outside the available range.
• The data source temporarily has no data.

A universe filter that matches nothing (for example a misspelled sector) does not return empty data: it raises a ValueError. List the valid sector names, then check what you downloaded:

from finlab import data

data.set_market('us')

# A misspelled sector raises instead of returning empty data
try:
    with data.us_universe(sector='Technolgy'):
        close = data.get('price:adj_close')
except ValueError as e:
    print(e)
    profile = data.get('us_company_profile')
    print("Available sectors:", sorted(profile['sector'].dropna().unique()))

with data.us_universe(sector='Technology'):
    close = data.get('price:adj_close')

if close.empty:
    print("Empty: truncate_start / truncate_end exclude every date, or the dataset has no data")
elif close.shape[0] < 10:
    print(f"Warning: too few data rows (only {close.shape[0]})")
else:
    print(f"Data is normal: {close.shape[0]} trading days, {close.shape[1]} stocks")

Network timeout

The download hangs for a long time and finally raises a timeout or connection error:

close = data.get('price:adj_close')
# finlab.exceptions.SignedDownloadReadTimeout: Download of us_price:adj_close failed (connection timed out). Check your network, proxy or firewall, then retry data.get().

Common causes: an unstable network connection, a very large download (for example the full market over 10+ years), or a server that is temporarily unresponsive.

A timeout is a DataError whose kind is "timeout". Wait a few seconds and retry once; if it fails again, check your network and try later:

import time
from finlab import data
from finlab.exceptions import DataError

data.set_market('us')
try:
    close = data.get('price:adj_close')
except DataError as e:
    if getattr(e, "kind", None) != "timeout":
        raise
    time.sleep(5)  # Wait 5 seconds, then retry
    close = data.get('price:adj_close')

finlab automatically caches downloaded data; the second run reads from the cache.

Login or permission failure

You cannot log in, cached credentials have expired, or a download reports a permission error. Common causes:

• Browser login has not completed, or local credentials expired.
• The headless environment variables FINLAB_REFRESH_TOKEN, FINLAB_SESSION_ID, and FINLAB_API_KEY are incomplete.
• The account lacks permission (a free user accessing VIP-only data).

Log in again, then check your membership level:

import finlab
from finlab import data

finlab.login()  # Complete the browser login flow

data.set_market('us')
close = data.get('price:adj_close')  # get_role() needs at least one data.get() call
print(f"Data access is working, latest date: {close.index[-1]}")

if not data.is_vip():
    print(f"Current role: {data.get_role()}; free accounts only receive older data")

If it still fails, check that the browser login completed, that the account is a valid VIP member, that the three environment variables above are set in headless environments, and that the network works. Upgrade to VIP for the latest data; you can check account and data permissions in your account settings in FinLab Studio. For more troubleshooting, see the FAQ.

Next steps