Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

Perturbed Model Validation: A New Framework to Validate Model Relevance

Jie Zhang 1, 2 Earl T Barr 1 Benjamin Guedj 2, 1, 3, 4 Mark Harman 1, 5 John Shawe-Taylor 1, 2, 4 
3 MODAL - MOdel for Data Analysis and Learning
LPP - Laboratoire Paul Painlevé - UMR 8524, Université de Lille, Sciences et Technologies, Inria Lille - Nord Europe, METRICS - Evaluation des technologies de santé et des pratiques médicales - ULR 2694, Polytech Lille - École polytechnique universitaire de Lille
Abstract : This paper introduces Perturbed Model Validation (PMV), a new technique to validate model relevance and detect overfitting or underfitting. PMV operates by injecting noise to the training data, re-training the model against the perturbed data, then using the training accuracy decrease rate to assess model relevance. A larger decrease rate indicates better concept-hypothesis fit. We realise PMV by perturbing labels to inject noise, and evaluate PMV on four real-world datasets (breast cancer, adult, connect-4, and MNIST) and nine synthetic datasets in the classification setting. The results reveal that PMV selects models more precisely and in a more stable way than cross-validation, and effectively detects both overfitting and underfitting.
Document type :
Preprints, Working Papers, ...
Complete list of metadata

Cited literature [9 references]  Display  Hide  Download
Contributor : Benjamin Guedj Connect in order to contact the contributor
Submitted on : Monday, May 27, 2019 - 4:49:47 PM
Last modification on : Thursday, March 24, 2022 - 3:12:59 AM


Files produced by the author(s)


  • HAL Id : hal-02139208, version 2



Jie Zhang, Earl T Barr, Benjamin Guedj, Mark Harman, John Shawe-Taylor. Perturbed Model Validation: A New Framework to Validate Model Relevance. 2019. ⟨hal-02139208v2⟩



Record views


Files downloads