crepes is a Python package for conformal prediction that implements conformal classifiers, regressors, and predictive systems on top of any standard classifier and regressor, turning the original predictions into well-calibrated p-values and cumulative distribution functions, or prediction sets and intervals with coverage guarantees.

The crepes package implements standard and Mondrian conformal classifiers as well as standard, normalized and Mondrian conformal regressors and predictive systems. While the package allows you to use your own functions to compute difficulty estimates, non-conformity scores and Mondrian categories, there is also a separate module, called crepes.extras, which provides some standard options for these. For testing the underlying assumption of exchangeability, you may use the classes in the module crepes.martingales.

Installation

From PyPI

pip install crepes

From conda-forge

conda install conda-forge::crepes

Documentation

For the complete documentation, see crepes.readthedocs.io.

Quickstart

Let us illustrate how we may use crepes to generate and apply conformal classifiers with a dataset from www.openml.org, which we first split into a training and a test set using train_test_split from sklearn, and then further split the training set into a proper training set and a calibration set:

from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split

dataset = fetch_openml(name="qsar-biodeg", parser="auto")

X = dataset.data.values.astype(float)
y = dataset.target.values

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)

X_prop_train, X_cal, y_prop_train, y_cal = train_test_split(X_train, y_train,
                                                            test_size=0.25)

We now "wrap" a random forest classifier, fit it to the proper training set, and fit a standard conformal classifier through the calibrate method:

from crepes import WrapClassifier
from sklearn.ensemble import RandomForestClassifier

rf = WrapClassifier(RandomForestClassifier(n_jobs=-1))

rf.fit(X_prop_train, y_prop_train)

rf.calibrate(X_cal, y_cal)

We may now produce p-values for the test set (an array with as many columns as there are classes):

rf.predict_p(X_test)

array([[0.00427104, 0.74842304],
       [0.07874355, 0.2950549 ],
       [0.50529983, 0.01557963],
       ...,
       [0.8413356 , 0.00201167],
       [0.84402215, 0.00654927],
       [0.29601955, 0.07766093]])

We can also get prediction sets, represented by binary vectors indicating presence (1) or absence (0) of the class labels that correspond to the columns, here at the 90% confidence level:

rf.predict_set(X_test, confidence=0.9)

array([[0, 1],
       [0, 1],
       [1, 0],
       ...,
       [1, 0],
       [1, 0],
       [1, 0]])

Since we have access to the true class labels, we can evaluate the conformal classifier (here using all available metrics which is the default), at the 99% confidence level:

rf.evaluate(X_test, y_test, confidence=0.99)

{'error': 0.007575757575757569,
 'avg_c': 1.6325757575757576,
 'one_c': 0.36742424242424243,
 'empty': 0.0,
 'ks_test': 0.0033578466103315894,
 'time_fit': 1.9073486328125e-06,
 'time_evaluate': 0.04798746109008789}

To control the error level across different groups of objects of interest, we may use so-called Mondrian conformal classifiers. A Mondrian conformal classifier is formed by providing a function or a MondrianCategorizer (defined in crepes.extras) as an additional argument, named mc, for the calibrate method.

For illustration, we will use the predicted labels of the underlying model to form the categories. Note that the prediction sets are generated for the test objects using the same categorization (under the hood).

rf_mond = WrapClassifier(rf.learner)

rf_mond.calibrate(X_cal, y_cal, mc=rf_mond.predict)

rf_mond.predict_set(X_test)

array([[0, 1],
       [1, 1],
       [1, 0],
       ...,
       [1, 0],
       [1, 0],
       [1, 1]])

The class-conditional conformal classifier is a special type of Mondrian conformal classifier, for which the categories are formed by the true labels; we can generate one by setting class_cond=True in the call to calibrate

rf_classcond = WrapClassifier(rf.learner)

rf_classcond.calibrate(X_cal, y_cal, class_cond=True)

rf_classcond.evaluate(X_test, y_test, confidence=0.99)

{'error': 0.0018939393939394478,
 'avg_c': 1.740530303030303,
 'one_c': 0.25946969696969696,
 'empty': 0.0,
 'ks_test': 0.11458837583733483,
 'time_fit': 7.152557373046875e-07,
 'time_evaluate': 0.06147575378417969}

When employing an inductive conformal predictor, the predicted p-values (and consequently the errors made) for a test set are not independent. Semi-online conformal predictors can however make them independent by updating the calibration set immediately after each prediction (assuming that the true label is then available). We can turn the conformal classifiers into semi-online conformal classifiers by enabling online calibration, i.e., setting online=True when calling the above methods, while also providing the true labels, e.g.,

rf_classcond.predict_p(X_test, y_test, online=True)

array([[8.13837566e-05, 8.86436603e-01],
       [6.60518590e-02, 4.02350293e-01],
       [4.28646783e-01, 4.29930890e-02],
       ...,
       [7.05118942e-01, 9.45056960e-03],
       [7.27003479e-01, 1.27347189e-02],
       [1.76403756e-01, 1.21434924e-01]])

Similarly, we can evaluate the conformal classifier while using online calibration:

rf_classcond.evaluate(X_test, y_test, confidence=0.99, online=True)

{'error': 0.007575757575757569,
 'avg_c': 1.6117424242424243,
 'one_c': 0.38825757575757575,
 'empty': 0.0,
 'ks_test': 0.14097384777782784,
 'time_fit': 1.9073486328125e-06,
 'time_evaluate': 0.05298352241516113}

Let us also illustrate how crepes can be used to generate conformal regressors and predictive systems. Again, we import a dataset from www.openml.org, which we split into a training and a test set and then further split the training set into a proper training set and a calibration set:

from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split

dataset = fetch_openml(name="house_sales", version=3, parser="auto")

X = dataset.data.values.astype(float)
y = dataset.target.values.astype(float)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)
X_prop_train, X_cal, y_prop_train, y_cal = train_test_split(X_train, y_train,
                                                            test_size=0.25)

Let us now "wrap" a RandomForestRegressor from sklearn using the class WrapRegressor from crepes and fit it (in the usual way) to the proper training set:

from sklearn.ensemble import RandomForestRegressor
from crepes import WrapRegressor

rf = WrapRegressor(RandomForestRegressor())
rf.fit(X_prop_train, y_prop_train)

We may now fit a conformal regressor using the calibration set through the calibrate method:

rf.calibrate(X_cal, y_cal)

The conformal regressor can now produce prediction intervals for the test set, here using a confidence level of 99%:

rf.predict_int(X_test, confidence=0.99)

array([[1938866.06, 3146372.54],
       [ 225335.1 , 1432841.58],
       [-403305.49,  804200.99],
       ...,
       [ 443742.33, 1651248.81],
       [-343684.48,  863822.  ],
       [-153629.93, 1053876.55]])

The output is a NumPy array with a row for each test instance, and where the two columns specify the lower and upper bound of each prediction interval.

We may request that the intervals are cut to exclude impossible values, in this case below 0, and if we also rely on the default confidence level (0.95), the output intervals will be a bit tighter:

rf.predict_int(X_test, y_min=0)

array([[2302049.84, 2783188.76],
       [ 588518.88, 1069657.8 ],
       [      0.  ,  441017.21],
       ...,
       [ 806926.11, 1288065.03],
       [  19499.3 ,  500638.22],
       [ 209553.85,  690692.77]])

The above intervals are not normalized, i.e., they are all of the same size (at least before they are cut). We coul

Crepes

Install / Use

README

Installation

Documentation

Quickstart