| Estimating the number of clusters in a dataset via the Gap statistic (2000) | |||||||||||||||||
Abstract | |||||||||||||||||
| We propose a method (the Gap statistic") for estimating the number of clusters (groups) in a set of data. The technique uses the output of any clustering algorithm (e.g. k-means or hierarchical), comparing the change in within cluster dispersion to that expected under an appropriate reference null distribution. Some theory is developed for the proposal and a simulation study that shows that the Gap statistic usually outperforms other methods that have been proposed in the literature. We also briey explore application of the same technique to the problem for estimating the number of linear principal components. 1 Introduction Cluster analysis is an important tool for unsupervised" learning| the problem of nding groups in data without the help of a response variable. A major challenge in cluster analysis is estimation of the optimal number of clusters". Figure 1 (top right) shows a typical plot of an error measure W k (the within cluster dispersion dened below) for a clustering pr... | |||||||||||||||||
Publication details | |||||||||||||||||
| |||||||||||||||||
Publications citing this publication (3) | |||||||||||||||||
| |||||||||||||||||