finlab.market

A market object tells the backtest engine three things: where to get open, close, volume and other price data, which index to compare performance against (for example the S&P 500 for US stocks), and the market's rules (open and close times and timezone, price limits, transaction costs, special stock categories). FinLab has built-in markets for US stocks, US ETFs and funds, Taiwan stocks and the Taiwan Emerging Stock Board. To backtest anything else, such as crypto, futures or your own CSV, subclass Market.

Through market objects, the backtesting engine can:

1. Retrieve the correct price data
2. Calculate correct returns
3. Display performance comparison against the benchmark
4. Support time-based decisions for live trading

Which market to use

data.set_market() selects the market for both data.get() and backtest.sim(), so after data.set_market('us') you rarely need to pass market= yourself.

Use a built-in market

from finlab import data, backtest
from finlab.markets.us import USFundMarket, USMarket

# US stock backtest: set_market('us') selects USMarket automatically
data.set_market('us')
close = data.get('price:adj_close')[['AAPL', 'MSFT', 'NVDA']]
position = close > close.average(50)
report = backtest.sim(position, resample='W')
# Equivalent to
report = backtest.sim(position, resample='W', market=USMarket())

# US ETF backtest
data.set_market('us_fund')
etf_close = data.get('price:adj_close')
etf_position = etf_close[['SPY', 'QQQ']].pct_change(126).is_largest(1)
report = backtest.sim(etf_position, resample='M', market=USFundMarket())

US stocks use USMarket and us_price; US ETFs and funds use USFundMarket and us_fund_price, so after data.set_market('us_fund'), data.get('price:adj_close') reads us_fund_price:adj_close. Taiwan stocks are the package default (data.set_market('tw'), TWMarket); everything above works the same way there.

Built-in markets compared

When an emerging-board stock moves to TWSE or TPEx, ROTCMarket exits it on its first listed day at the listed price, not at its last emerging-board price.

Benchmark against total return

The US, Japan, Hong Kong, Korea and UK markets (and their ETF markets) benchmark against price indices from world_index:adj_close (^GSPC, ^N225, ^HSI, ^KS11, ^FTSE), which exclude dividends, while strategies use dividend-adjusted prices. Unlike Taiwan, whose benchmark is a total return index, an overseas backtest's benchmark line is therefore low and its alpha high, by roughly the market's dividend yield.

To benchmark against total return, subclass the market and return a representative ETF's adjusted price from get_benchmark():

from finlab import data
from finlab.markets.us import USMarket

class USMarketWithDividends(USMarket):
    @staticmethod
    def get_benchmark():
        return data.get('us_fund_price:adj_close')['SPY']

Build a custom market

The simplest custom market implements only two methods: get_name() returns the market name and get_price() returns a price DataFrame. Every other method is optional; override it when you need it.

Cryptocurrency

import pandas as pd
from finlab import backtest
from finlab.market import Market

class CryptoMarket(Market):
    @staticmethod
    def get_name():
        return 'crypto'

    def get_price(self, trade_at_price='close', adj=True):
        df = pd.read_csv('crypto_close.csv', index_col=0, parse_dates=True)
        return df

    @staticmethod
    def get_benchmark():
        df = pd.read_csv('crypto_close.csv', index_col=0, parse_dates=True)
        return df['BTC']  # Use BTC as benchmark

# Usage
report = backtest.sim(position, market=CryptoMarket())

Futures (with leverage)

import pandas as pd
from finlab import backtest
from finlab.market import Market

class FuturesMarket(Market):
    def __init__(self, leverage=10):
        self.leverage = leverage
        self.prices = pd.read_csv('futures_price.csv', index_col=0, parse_dates=True)

    @staticmethod
    def get_name():
        return 'futures'

    def get_price(self, trade_at_price='close', adj=True):
        price = self.prices[trade_at_price]
        daily_return = price.pct_change() * self.leverage
        leveraged_price = (1 + daily_return).fillna(1).cumprod() * 100
        return leveraged_price

# Use 10x leverage
report = backtest.sim(position, market=FuturesMarket(leverage=10))

When you need a custom market

The Market class

Market()

The base class for markets: inherit from finlab.market.Market and override the methods below to backtest a new market. USMarket, USFundMarket, TWMarket and the other built-in markets are subclasses.

from finlab.market import Market

class MyMarket(Market):
    _price_table = 'us_price'
    _adj_prefix = 'us_price:adj_'

A subclass that sets the class variables _price_table (for example 'us_price') and _adj_prefix (for example 'us_price:adj_') inherits a working get_price() that reads prices from finlab.data. Markets with a different schema override get_price().

Market.get_price()

Returns the prices or volume the backtest uses.

import pandas as pd
from finlab.market import Market

class CsvMarket(Market):
    def get_price(self, trade_at_price='close', adj=True):
        return pd.read_csv(f'{trade_at_price}.csv', index_col=0, parse_dates=True)

Returns a pd.DataFrame indexed by date with one column per symbol.

Implement at least 'open', 'close', 'high' and 'low': they are the trading prices and feed get_trading_price(). Implementing 'volume' is strongly recommended: liquidity analysis, finlab.ml.qlib features and PortfolioSyncManager need it. Supporting custom Series/DataFrame input lets users backtest with their own prices.

Market.get_trading_price()

Returns the prices used to fill trades, built on get_price().

from finlab.markets.us import USMarket

fill_price = USMarket().get_trading_price('open_close_avg')

Market.get_benchmark()

Returns the benchmark price series (date index) that reports compare the strategy against.

from finlab import data
from finlab.market import Market

class MyMarket(Market):
    @staticmethod
    def get_benchmark():
        return data.get('world_index:adj_close')['^GSPC']

The base class returns an empty Series, which hides the benchmark. USMarket returns data.get('world_index:adj_close')['^GSPC'], the S&P 500 price index, which excludes dividends.

Market.get_market_value()

Returns market values indexed by date with one column per symbol.

from finlab import data
from finlab.market import Market

class MyMarket(Market):
    @staticmethod
    def get_market_value():
        return data.get('etl:us_market_value')

The base class returns an empty DataFrame.

Market.get_asset_id_to_name()

Returns a mapping from symbol to display name, used to label holdings in reports.

from finlab.markets.us import USMarket

names = USMarket.get_asset_id_to_name()  # for example {'AAPL': 'Apple Inc.', ...}

USMarket maps each symbol to its company name. The base class returns {}.

Market.get_industry()

Returns a mapping from symbol to industry.

from finlab.markets.us import USMarket

industry = USMarket.get_industry()  # reads the industry column of us_company_profile

USMarket reads the industry column of us_company_profile. The base class returns {}.

Market.get_delisting_notices()

Returns delisting notices announced by the exchange: one row per notice, indexed by symbol, with columns announced (announcement date) and delisted (delisting date).

from finlab.markets.us import USMarket

notices = USMarket.get_delisting_notices()

A symbol can have several rows. The backtest sells a holding on the first trading day after the announcement and stops buying it. Symbols without a notice exit at their last quote when their prices end early. The base class returns an empty table.

Market rules and their defaults

Common questions

def get_price(self, trade_at_price='close', adj=True):
    if adj:
        # Return adjusted prices accounting for dividends and splits
        return pd.read_csv('adj_close.csv', index_col=0, parse_dates=True)
    else:
        # Return raw closing prices
        return pd.read_csv('raw_close.csv', index_col=0, parse_dates=True)

@staticmethod
def get_benchmark():
    return pd.Series(
        dtype=float, index=pd.DatetimeIndex([])
    )  # Do not display benchmark

import pandas as pd
from finlab import data
from finlab.market import Market

class StocksAndEtfsMarket(Market):
    def get_price(self, trade_at_price='close', adj=True):
        stocks = data.get('us_price:adj_close')[['AAPL', 'MSFT']]
        etfs = data.get('us_fund_price:adj_close')[['SPY', 'QQQ']]
        return pd.concat([stocks, etfs], axis=1)

from finlab.markets.us import USMarket

# A broker that charges 0.05% commission per trade; U.S. stocks have no transaction tax
report = backtest.sim(
    position,
    resample='W',
    market=USMarket(),
    fee_ratio=0.0005,
    tax_ratio=0,
)

Next steps

Source code: USMarket, TWMarket.