Hierarchical Clustering

In this section, you will learn about hierarchical clustering and, in particular, agglomerative clustering. In contrast to K-means, this methodology does not require you to know the number of clusters in advance. This information is generated from a dendrogram created by the algorithm. As clusters of points are created, notions of the distance between two sets (that is, the "linkage') must be understood when applying this algorithm. You should already know how to compute the Euclidean distance between two points. This article also points out that there are many ways to compute the distance between points (Manhattan, maximum, Mahalanobois, etc.). We can also use these functions for point distances to compute the distance between two sets of points. For example, single linkage computes set distances by choosing the two closest points. Complete linkage chooses the two most distant points. Average distance computes the average of all distances between all points from both sets and so on. Read through this article to get an overview of hierarchical clustering.

In data mining and statistics, hierarchical clustering (also called hierarchical cluster analysis or HCA) is a method of cluster analysis that seeks to build a hierarchy of clusters. Strategies for hierarchical clustering generally fall into two types:

Agglomerative: This is a "bottom-up" approach: each observation starts in its own cluster, and pairs of clusters are merged as one moves up the hierarchy.
Divisive: This is a "top-down" approach: all observations start in one cluster, and splits are performed recursively as one moves down the hierarchy.

In general, the merges and splits are determined in a greedy manner. The results of hierarchical clustering are usually presented in a dendrogram.

The standard algorithm for hierarchical agglomerative clustering (HAC) has a time complexity of $O (n^3)$ and requires $Ω (n^ 2)$ memory, which makes it too slow for even medium data sets. However, for some special cases, optimal efficient agglomerative methods (of complexity $O(n^2)$ are known: SLINK for single-linkage and CLINK for complete-linkage clustering. With a heap, the runtime of the general case can be reduced to $O(n^2 \text{log ⁡} n )$ , an improvement on the aforementioned bound of $O(n^3)$ , at the cost of further increasing the memory requirements. In many cases, the memory overheads of this approach are too large to make it practically usable.

Except for the special case of single-linkage, none of the algorithms (except exhaustive search in $O(2^n)$ can be guaranteed to find the optimum solution.

Divisive clustering with an exhaustive search is $O(2^n)$ , but it is common to use faster heuristics to choose splits, such as k-means.

Cluster dissimilarity

In order to decide which clusters should be combined (for agglomerative), or where a cluster should be split (for divisive), a measure of dissimilarity between sets of observations is required. In most methods of hierarchical clustering, this is achieved by the use of an appropriate metric (a measure of distance between pairs of observations), and a linkage criterion that specifies the dissimilarity of sets as a function of the pairwise distances of observations in the sets.

Metric

The choice of an appropriate metric will influence the shape of the clusters, as some elements may be relatively closer to one another under one metric than another. For example, in two dimensions, under the Manhattan distance metric, the distance between the origin (0,0) and (0.5, 0.5) is the same as the distance between the origin and (0, 1), while under the Euclidean distance metric the latter is strictly greater.

Some commonly used metrics for hierarchical clustering are:

Names	Formula
Euclidean distance	$\\|a-b\\|_{2}={\sqrt {\sum _{i}(a_{i}-b_{i})^{2}}}$
Squared Euclidean distance	$\\|a-b\\|_{2}^{2}=\sum _{i}(a_{i}-b_{i})^{2}$
Manhattan (or city block ) distance	$\\|a-b\\|_{1}=\sum _{i}\|a_{i}-b_{i}\|$
Maximum distance (or Chebyshev distance)	$\\|a-b\\|_{\infty }=\max _{i}\|a_{i}-b_{i}\|$
Mahalanobis distance	${\sqrt {(a-b)^{\top }S^{-1}(a-b)}}$ = where S is the Covariance matrix

For text or other non-numeric data, metrics such as the Hamming distance or Levenshtein distance are often used.

Euclidean and Manhattan distances are the special cases of generalized Minkowski distance with p = 1 (for Manhattan) and p = 2 (for Euclidean).

Several other dissimilarity measures exist. Particularly, correlation-based distances - Pearson, Eisen cosine, Spearman, Kendall correlation distances, which are widely used for gene expression data analyses. Correlation-based distance is defined by subtracting the correlation coefficient from 1. Strictly speaking, correlation-based distances cannot be used as metrics, while the square root of it can be.

A review of cluster analysis in health psychology research found that the most common distance measure in published studies in that research area is the Euclidean distance or the squared Euclidean distance.

Linkage criteria

The linkage criterion determines the distance between sets of observations as a function of the pairwise distances between observations.

Some commonly used linkage criteria between two sets of observations A and B, are:

Names	Formula
Maximum or complete-linkage clustering	$\max \,\{\,d(a,b):a\in A,\,b\in B\,\}.$
Minimum or single-linkage clustering	$\min \,\{\,d(a,b):a\in A,\,b\in B\,\}..$
Unweighted average linkage clustering (or UPGMA)	${\frac {1}{\|A\|\cdot \|B\|}}\sum _{a\in A}\sum _{b\in B}d(a,b).$
Weighted average linkage clustering (or WPGMA)	$d(i\cup j,k)={\frac {d(i,k)+d(j,k)}{2}}.$
Centroid linkage clustering, or UPGMC	$d(c_{A},c_{B})$ where $c_{A}$ and $c_{B}$ are the centroids of clusters A and B, respectively.
Minimum energy clustering	${\displaystyle {\frac {2}{nm}}\sum _{i,j=1}^{n,m}\\|a_{i}-b_{j}\\|_{2}-{\frac {1}{n^{2}}}\sum _{i,j=1}^{n}\\|a_{i}-a_{j}\\|_{2}-{\frac {1}{m^{2}}}\sum _{i,j=1}^{m}\\|b_{i}-b_{j}\\|_{2}}$

where

$d$ is the chosen metric. Other linkage criteria include:

The sum of all intra-cluster variance.
The increase in variance for the cluster being merged (Ward's criterion).
The probability that candidate clusters spawn from the same distribution function (V-linkage).
The product of in-degree and out-degree on a k-nearest-neighbor graph (graph degree linkage).
The increment of some cluster descriptor (i.e., a quantity defined for measuring the quality of a cluster) after merging two clusters.

Discussion

Hierarchical clustering has the distinct advantage that any valid measure of distance can be used. In fact, the observations themselves are not required: all that is used is a matrix of distances.

Agglomerative clustering example

Raw data

For example, suppose this data is to be clustered, and the Euclidean distance is the distance metric.

The hierarchical clustering dendrogram would be as such:

Traditional representation

Cutting the tree at a given height will give partitioning clustering at a selected precision. In this example, cutting after the second row (from the top) of the dendrogram will yield clusters {a} {b c} {d e} {f}. Cutting after the third row will yield clusters {a} {b c} {d e f}, which is a coarser clustering with a smaller number but larger clusters.

This method builds the hierarchy from the individual elements by progressively merging clusters. In our example, we have six elements {a} {b} {c} {d} {e} and {f}. The first step is to determine which elements to merge in a cluster. Usually, we want to take the two closest elements according to the chosen distance.

Optionally, one can also construct a distance matrix at this stage, where the number in the i-th row j-th column is the distance between the i-th and j-th elements. Then, as clustering progresses, rows and columns are merged as the clusters are merged and the distances are updated. This is a common way to implement this type of clustering and has the benefit of caching distances between clusters. A simple agglomerative clustering algorithm is described on the single-linkage clustering page; it can easily be adapted to different types of linkage (see below).

Suppose we have merged the two closest elements b and c, we now have the following clusters {a}, {b, c}, {d}, {e} and {f}, and want to merge them further. To do that, we need to take the distance between {a} and {b c} and therefore define the distance between two clusters. Usually, the distance between two clusters

$\mathcal A$

$\mathcal {B}$ is one of the following:

The maximum distance between elements of each cluster (also called complete-linkage clustering): $\max\{\,d(x,y):x\in {\mathcal {A}},\,y\in {\mathcal {B}}\,\}$
The minimum distance between elements of each cluster (also called single-linkage clustering): $\min\{\,d(x,y):x\in {\mathcal {A}},\,y\in {\mathcal {B}}\,\}$ .
The mean distance between elements of each cluster (also called average linkage clustering, used e.g. in UPGMA): ${1 \over {|{\mathcal {A}}|\cdot |{\mathcal {B}}|}}\sum _{{x\in {\mathcal {A}}}}\sum _{{y\in {\mathcal {B}}}}d(x,y)$ .
The sum of all intra-cluster variance.
The increase in variance for the cluster being merged (Ward's method)
The probability that candidate clusters spawn from the same distribution function (V-linkage).

In the case of tied minimum distances, a pair is randomly chosen, thus being able to generate several structurally different dendrograms. Alternatively, all tied pairs may be joined at the same time, generating a unique dendrogram.

One can always decide to stop clustering when there is a sufficiently small number of clusters (number criterion). Some linkages may also guarantee that agglomeration occurs at a greater distance between clusters than the previous agglomeration, and then one can stop clustering when the clusters are too far apart to be merged (distance criterion). However, this is not the case of, e.g., the centroid linkage where the so-called reversals (inversions, departures from ultrametricity) may occur.

Divisive clustering

The basic principle of divisive clustering was published as the DIANA (DIvisive ANAlysis Clustering) algorithm. Initially, all data is in the same cluster, and the largest cluster is split until every object is separate. Because there exist $O ( 2 ^n )$ ways of splitting each cluster, heuristics are needed. DIANA chooses the object with the maximum average dissimilarity and then moves all objects to this cluster that are more similar to the new cluster than to the remainder.

Software

Open source implementations

Hierarchical clustering dendrogram of the Iris dataset (using R).

Hierarchical clustering and interactive dendrogram visualization in Orange data mining suite.

ALGLIB implements several hierarchical clustering algorithms (single-link, complete-link, Ward) in C++ and C# with O(n²) memory and O(n³) run time.
ELKI includes multiple hierarchical clustering algorithms, various linkage strategies, efficient SLINK, CLINK, and Anderberg algorithms, flexible cluster extraction from dendrograms, and various other cluster analysis algorithms.
Julia has an implementation inside the Clustering.jl package.
Octave, the GNU analog to MATLAB, implements hierarchical clustering in function "linkage".
Orange, a data mining software suite, includes hierarchical clustering with interactive dendrogram visualization.
R has built-in functions and packages that provide functions for hierarchical clustering.
SciPy implements hierarchical clustering in Python, including the efficient SLINK algorithm.
scikit-learn also implements hierarchical clustering in Python.
Weka includes hierarchical cluster analysis.

Commercial implementations

MATLAB includes hierarchical cluster analysis.
SAS includes hierarchical cluster analysis in PROC CLUSTER.
Mathematica includes a Hierarchical Clustering Package.
NCSS includes hierarchical cluster analysis.
SPSS includes hierarchical cluster analysis.
Qlucore Omics Explorer includes hierarchical cluster analysis.
Stata includes hierarchical cluster analysis.
CrimeStat includes a nearest neighbor hierarchical cluster algorithm with a graphical output for a Geographic Information System.

Source: Wikipedia, https://en.wikipedia.org/wiki/Hierarchical_clustering
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.

Last modified: Tuesday, March 5, 2024, 12:48 PM