Skip to Main content Skip to Navigation
Conference papers

Feature-Based Cluster Validation for High-Dimensional Data

Randa Kassab 1 Jean-Charles Lamirel
1 CORTEX - Neuromimetic intelligence
INRIA Lorraine, LORIA - Laboratoire Lorrain de Recherche en Informatique et ses Applications
Abstract : Cluster validation is commonly used to determine the optimal number of clusters in a data set. Despite the success of distance-based validity indexes, their efficacy decreases rapidly when dealing with high-dimensional data. The present paper introduces a feature-based cluster validation criterion which can cope with said situation. In contrast to distance-based methods, our criterion evaluates similarity in terms of shared relevant features between data. The idea is based on the identification of the ``core'' features which are correlated within the description of each of the discovered clusters. The individual quality of each cluster is then evaluated through the frequency of the core features with respect to that of the non-core features within the cluster, while the between-cluster isolation is measured by means of the overlap coefficient between clusters, considering only the core features within the clusters. The overall clustering quality is measured by a weighted combination of the within and between cluster correlation coefficients, which enables choosing an appropriate number of clusters according to the purpose of clustering. Furthermore, our validation can prune out unreliable clusters which have no correlated features and thus no specific description of their content. Extensive experiments on the Reuters-21578 collection are conducted to show the effectiveness of our validation criterion.
Complete list of metadatas

https://hal.inria.fr/inria-00332310
Contributor : Randa Kassab <>
Submitted on : Monday, October 20, 2008 - 3:55:21 PM
Last modification on : Friday, May 10, 2019 - 1:54:02 PM

Identifiers

  • HAL Id : inria-00332310, version 1

Collections

Citation

Randa Kassab, Jean-Charles Lamirel. Feature-Based Cluster Validation for High-Dimensional Data. Artificial Intelligence and Applications - AIA 2008, Feb 2008, Innsbruck, Austria, Austria. pp.232-239. ⟨inria-00332310⟩

Share

Metrics

Record views

404