Feature-Based Cluster Validation for High-Dimensional Data

Randa Kassab 1 Jean-Charles Lamirel
1 CORTEX - Neuromimetic intelligence
INRIA Lorraine, LORIA - Laboratoire Lorrain de Recherche en Informatique et ses Applications
Abstract : Cluster validation is commonly used to determine the optimal number of clusters in a data set. Despite the success of distance-based validity indexes, their efficacy decreases rapidly when dealing with high-dimensional data. The present paper introduces a feature-based cluster validation criterion which can cope with said situation. In contrast to distance-based methods, our criterion evaluates similarity in terms of shared relevant features between data. The idea is based on the identification of the ``core'' features which are correlated within the description of each of the discovered clusters. The individual quality of each cluster is then evaluated through the frequency of the core features with respect to that of the non-core features within the cluster, while the between-cluster isolation is measured by means of the overlap coefficient between clusters, considering only the core features within the clusters. The overall clustering quality is measured by a weighted combination of the within and between cluster correlation coefficients, which enables choosing an appropriate number of clusters according to the purpose of clustering. Furthermore, our validation can prune out unreliable clusters which have no correlated features and thus no specific description of their content. Extensive experiments on the Reuters-21578 collection are conducted to show the effectiveness of our validation criterion.
Type de document :
Communication dans un congrès
Artificial Intelligence and Applications - AIA 2008, Feb 2008, Innsbruck, Austria, Austria. pp.232-239, 2008
Liste complète des métadonnées

https://hal.inria.fr/inria-00332310
Contributeur : Randa Kassab <>
Soumis le : lundi 20 octobre 2008 - 15:55:21
Dernière modification le : jeudi 11 janvier 2018 - 06:19:48

Identifiants

  • HAL Id : inria-00332310, version 1

Collections

Citation

Randa Kassab, Jean-Charles Lamirel. Feature-Based Cluster Validation for High-Dimensional Data. Artificial Intelligence and Applications - AIA 2008, Feb 2008, Innsbruck, Austria, Austria. pp.232-239, 2008. 〈inria-00332310〉

Partager

Métriques

Consultations de la notice

278