Feature-Based Cluster Validation for High-Dimensional Data - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2008

Feature-Based Cluster Validation for High-Dimensional Data

Randa Kassab
Jean-Charles Lamirel

Abstract

Cluster validation is commonly used to determine the optimal number of clusters in a data set. Despite the success of distance-based validity indexes, their efficacy decreases rapidly when dealing with high-dimensional data. The present paper introduces a feature-based cluster validation criterion which can cope with said situation. In contrast to distance-based methods, our criterion evaluates similarity in terms of shared relevant features between data. The idea is based on the identification of the ``core'' features which are correlated within the description of each of the discovered clusters. The individual quality of each cluster is then evaluated through the frequency of the core features with respect to that of the non-core features within the cluster, while the between-cluster isolation is measured by means of the overlap coefficient between clusters, considering only the core features within the clusters. The overall clustering quality is measured by a weighted combination of the within and between cluster correlation coefficients, which enables choosing an appropriate number of clusters according to the purpose of clustering. Furthermore, our validation can prune out unreliable clusters which have no correlated features and thus no specific description of their content. Extensive experiments on the Reuters-21578 collection are conducted to show the effectiveness of our validation criterion.
No file

Dates and versions

inria-00332310 , version 1 (20-10-2008)

Identifiers

  • HAL Id : inria-00332310 , version 1

Cite

Randa Kassab, Jean-Charles Lamirel. Feature-Based Cluster Validation for High-Dimensional Data. Artificial Intelligence and Applications - AIA 2008, Feb 2008, Innsbruck, Austria, Austria. pp.232-239. ⟨inria-00332310⟩
3168 View
0 Download

Share

Gmail Facebook X LinkedIn More