Microstructure

A capital market microstructure data simulation/generation framework

Work in progress...

Why data simulation?

One might wonder why it would be useful to generate simulated financial data instead of using the real thing: historical data. There are two main reasons for this.

First, high-quality financial data is extremely expensive. This is especially true of level 2 data, but even high-frequency level 1 data is often unaffordable. The only free, public historical data that is available is daily or hourly bar data, which is too low-frequency and too simplistic to be useful to professional traders (in practice there are also very few, if any, inefficiencies at this level).

Second, when using algorithmic or machine learning approaches, it is only too easy to overfit historical data or incur a "model selection bias" on this data. In fact, historical data and backtesting should not be used as a research tool, methodologically speaking (see Marcos Lopez de Prado's book "Advances in Financial Machine Learning", for example). Doing so is the easiest way to ensure that you develop a strategy that only works on paper, or a strategy that only works for a particular market regime. Instead, historical data and backtesting tools should be used merely as "sanity checks" to confirm that the resulting strategy makes sense, before going live with it.

A "best practice" methodology for developing trading strategies might go something like this:

• First generate a hypothesis based on domain expertise. This includes a mechanism of "why" such a strategy would be profitable.
• Generate simulated data that reproduces the necessary underlying mechanism, while randomizing everything else (within reasonable market parameters)
• Develop your strategy and iterate on this simulated data only, until the desired behaviour and level of success is reached.
• Backtest on historical data to confirm that the strategy is sensible. If it is, proceed (cautiously).
• If it isn't, either your hypothesis is wrong (i.e. the market doesn't behave the way you think it does) or the simulated mechanism doesn't correspond to your hypothesis.
• The worst thing you can do at this stage is to tweak a few parameters and try again on the historical data, forcefully trying to make it work. (unless the tweak is a literal bug fix -- but then why did it work well on the simulated data?)
• Note: unless you're an experienced trader, the vast majority of your hypotheses will be wrong. But it's better to know you're wrong than to think you're right (when you're not) and bet money on it...

Simulated data is also a great way to develop new algorithms (i.e. not a specific strategy per se, but an underlying machine learning approach, for example) and test whether they're able to successfully learn/optimize on this simulation data. If they're not, then it's unlikely that they will generalize to real market data.

For custom data generation:

DataGeneration.Asset generates historical quote and trade data for an individual asset. It is currently being modelled as a lognormal random walk. To instantiate it, pass a parameter dictionary with the following values:

• 'trend': the mean trend of this asset.
• 'volatility': the standard deviation of the random walk's innovation.
• 'base_price': the starting price level for this asset.
• 'mean_bid_ask': the average bid-ask spread of the asset.
• 'stdev_bid_ask': the standard deviation of the changes in the bid-ask spread.
• 'trade_freq': the trade frequency stochastic process, an object instance from the library stochastic. See RVtrading.py for an example.
• 'trade_skew': a stochastic process, as in trade_freq, that represents the ratio of buy trades vs sell trades. At 0.5, there is an equal distribution of buys and sells on the asset. At 1, trades only occur on the ask side (active buys). At 0, trades only occur on the bid side (active sells).
• 'dark_exchange': a boolean flag that enables a dark exchange. When enabled, a fraction of fills occur in-between the bid and the ask rather than at the lit (public) bid/ask. Optional, defaults to False.
• 'dark_freq': the fraction of trades that occur on the dark exchange vs. the lit/public books, a value between 0 and 1. Only used when 'dark_exchange' is True. Optional, defaults to 0.
• 'dark_mid_only': a boolean flag. If True, the dark exchange is a mid-book only venue and all dark fills occur exactly midway between the bid and the ask. If False, dark fills can occur at any price between the bid and the ask. Only used when 'dark_exchange' is True. Optional, defaults to True.
• 'regime_change_freq': the probability, at each time step, of a sudden regime change — a disjoint jump of the random walk's hyperparameters ('trend' and 'volatility') to brand new values. At 0 (the default), the hyperparameters stay fixed for the whole series. Optional.
• 'regime_trend_mean': when a regime change occurs, the new 'trend' is drawn from a normal distribution with this mean. Only used when 'regime_change_freq' > 0. Optional, defaults to 0.
• 'regime_trend_stdev': the standard deviation of the normal distribution used to draw the new 'trend' on a regime change. Only used when 'regime_change_freq' > 0. Optional, defaults to 0.
• 'regime_vol_mean': when a regime change occurs, the new 'volatility' is drawn from the absolute value of a normal distribution with this mean. Only used when 'regime_change_freq' > 0. Optional, defaults to the original 'volatility'.
• 'regime_vol_stdev': the standard deviation of the normal distribution used to draw the new 'volatility' on a regime change. Only used when 'regime_change_freq' > 0. Optional, defaults to 0.

DataGeneration.Cointegration is used to build relationships between assets of the provided list. In particular, this relationship is one of cointegration (not correlation). This means that the price levels of these assets will tend to revert to a fixed mean distance between each other. This is currently modelled as an Ornstein-Uhlenbeck process.

To instantiate a Cointegration object, pass a list of Asset objects and a parameter dictionary that contains the following values:

• 'theta': the strength of mean reversion. A value of 0 means no cointegration. A strength of 1 means the assets will follow each other to perfection.
• 'volatility': the strength of the innovations that temporarily drive the prices apart. This is the strength of the "pushing" force, while theta represents the strength of the "pulling" force.
• 'premia': for each asset, the constant mean difference w.r.t. the price level of the first asset (value at index 0 is therefore unused).
• 'shock_freq': for each asset, how frequent are the shocks, a value between 0 and 1. At 0, shocks are disabled. At 1, shocks happen at every time step (not recommended).
• 'shock_duration': for each asset, the strength of persistence of shocks, a value between 0 and 1. At 0, shocks immediately revert to the original premium value (i.e. they disappear). At 1, the shocks are permanent. This value is similar in functionality to the theta in the cointegration relationship.
• 'shock_mean': for each asset, the average jump in premium when a shock occurs.
• 'shock_stdev': for each asset, the standard deviation to use when selecting the random jump value for a shock.
• 'trend_freq': for each asset, the probability, at each time step, of a temporary trend starting (while none is already active for that asset). A temporary trend is a finite window during which the equilibrium premium drifts steadily in one direction; once the window ends, the asset simply cointegrates around the new equilibrium level (the directional movement is temporary, the resulting level shift persists). At 0, temporary trends are disabled. Optional, defaults to 0.
• 'trend_duration': for each asset, the average number of time steps a temporary trend lasts. The actual duration of each trend is drawn from an exponential distribution with this mean. Only used when 'trend_freq' > 0. Optional, defaults to 0.
• 'trend_slope_mean': for each asset, the average per-step drift applied to the equilibrium premium while a temporary trend is active. The slope of each trend is drawn from a normal distribution with this mean. Only used when 'trend_freq' > 0. Optional, defaults to 0.
• 'trend_slope_stdev': for each asset, the standard deviation of the normal distribution used to draw a temporary trend's per-step slope. Only used when 'trend_freq' > 0. Optional, defaults to 0.

Once an Asset or a Cointegration object (obj) has been instantiated, you can use it to generate a number of data points as follows:

data_points = obj.generate(N)

For the case of DataGeneration.Cointegration, this will generate N data points of the following format:

data_points = (bid_data, ask_data, trade_data)

  bid_data shape: (D, N) where D is the number of assets, and N is the number of timesteps (bid values) per asset.

  ask_data shape: (D, N) where D is the number of assets, and N is the number of timesteps (ask values) per asset.

  trade_data shape: (D, N) where D is the number of assets, and N is the number of timesteps (trade prices) per asset.
  Each time step where a trade did not occur will instead contain a *None* value.

For the case of DataGeneration.Asset, the returned data does not contain a D dimension. They are only flat vectors of prices.

Examples:

See the file RVtrading.py for a few examples on generating assets and cointegration data.

Run the main.py file to generate data and visualize it.

You can also run the example trading strategy under strategies/rv_trader1.py

TODO/Desired features:

• [x] More realistic/complex Cointegration mechanism that can simulate temporary shocks
• [x] trade_skew and trade_freq shouldn't be static parameters? They are stochastic processes?
• [x] Support for regime changes and temporary trends when simulating individual