This project is maintained by REFRAME
We introduce a powerful technique to make classifiers more reliable and versatile. Background Check equips classifiers with the ability to assess the difference of unlabelled test data from the training data. In particular, Background Check gives classifiers the capability to (i) perform cautious classification with a reject option; (ii) identify outliers; and (iii) better assess the confidence in their predictions. We derive the method from first principles and consider four particular relationships between background and foreground distributions.
We have created an appendix with supplementary material containing the mathematical proofs, algorithms and extended explanations for figures and tables (see this pdf file).
We have created a series of tutorials in Python and Jupyter Notbook format that introduces the motivation of Background Check and how to apply in different situations.
The code for Background Check and for the experiments submitted to ICDM-2016 has been created by Miquel Perelló Nieto (@perellonieto) and Telmo de Menezes e Silva Filho (@tmfilho).
If you want to cite this work, please use the following citation format
M. Perello-Nieto, T. M. Silva Filho, M. Kull and P. Flach. "Background Check:
A general technique to build more reliable and versatile classifiers"
International Conference on Data Mining (ICDM). 2016. IEEE. to appear
or the following bibtex entry:
@inproceedings{perello2017,
title={Background Check: A general technique to build more reliable and
versatile classifiers},
keywords={Training data, Estimation, Standards, Reliability,
Data models, Probability, Conferences, Multiclass classification,
Outlier detection, Confident classification},
author={Miquel Perello-Nieto and
Telmo Menezes {Silva Filho} and
Meelis Kull and
Peter Flach},
year={2017},
month={3},
doi={10.1109/ICDM.2016.0150},
isbn={9781509054749},
language={English},
series={Proceedings of the IEEE International Conference on Data Mining (ICDM)},
publisher={Institute of Electrical and Electronics Engineers (IEEE)},
pages={1143-1148},
booktitle={2016 IEEE 16th International Conference on Data Mining (ICDM 2016)},
address={United States},
}
If you are having problems executing the experiments or the tutorials, do not hesitate to open an issue or contact us.