Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

Perturbed Model Validation: A New Framework to Validate Model Relevance

Jie Zhang 1, 2 Earl Barr 1 Benjamin Guedj 2, 1, 3, 4 Mark Harman 1, 5 John Shawe-Taylor 1, 2, 4
3 MODAL - MOdel for Data Analysis and Learning
LPP - Laboratoire Paul Painlevé - UMR 8524, Université de Lille, Sciences et Technologies, Inria Lille - Nord Europe, METRICS - Evaluation des technologies de santé et des pratiques médicales - ULR 2694, Polytech Lille - École polytechnique universitaire de Lille
Abstract : This paper introduces Perturbed Model Validation (PMV), a new technique to validate model relevance and detect overfitting or underfitting. PMV operates by injecting noise to the training data, re-training the model against the perturbed data, then using the training accuracy decrease rate to assess model relevance. A larger decrease rate indicates better concept-hypothesis fit. We realise PMV by perturbing labels to inject noise, and evaluate PMV on four real-world datasets (breast cancer, adult, connect-4, and MNIST) and nine synthetic datasets in the classification setting. The results reveal that PMV selects models more precisely and in a more stable way than cross-validation, and effectively detects both overfitting and underfitting.
Document type :
Preprints, Working Papers, ...
Complete list of metadata

Cited literature [9 references]  Display  Hide  Download

https://hal.inria.fr/hal-02139208
Contributor : Benjamin Guedj <>
Submitted on : Monday, May 27, 2019 - 4:49:47 PM
Last modification on : Friday, February 5, 2021 - 3:29:23 AM

File

main.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02139208, version 2

Collections

Citation

Jie Zhang, Earl Barr, Benjamin Guedj, Mark Harman, John Shawe-Taylor. Perturbed Model Validation: A New Framework to Validate Model Relevance. 2019. ⟨hal-02139208v2⟩

Share

Metrics

Record views

134

Files downloads

911