Scikit Learn

Unsupervised Learning Reference

Targets scikit-learn 1.9.1. Snippets with caller-supplied data/columns are illustrative; fit all learned preprocessing inside the training folds when estimating predictive performance.

Overview

Unsupervised learning discovers patterns in unlabeled data through clustering, dimensionality reduction, and density estimation.

Clustering

K-Means

KMeans (sklearn.cluster.KMeans)

• Partition-based clustering into K clusters

• Key parameters:

  • n_clusters: Number of clusters to form

  • init: Initialization method ('k-means++', 'random')

  • n_init: Number of initializations (default='auto'; one run with k-means++, ten with random init). Set an integer explicitly for repeatable search effort

  • max_iter: Maximum iterations

• Use when: Know number of clusters, spherical cluster shapes

• Fast and scalable

• Example:

from sklearn.cluster import KMeans

model = KMeans(n_clusters=3, init='k-means++', n_init=10, random_state=42)
labels = model.fit_predict(X)
centers = model.cluster_centers_

# Inertia (sum of squared distances to nearest center)
print(f"Inertia: {model.inertia_}")

MiniBatchKMeans

• Faster K-Means using mini-batches

• Use when: Large datasets, need faster training

• Slightly less accurate than K-Means

• Example:

from sklearn.cluster import MiniBatchKMeans

model = MiniBatchKMeans(n_clusters=3, batch_size=100, random_state=42)
labels = model.fit_predict(X)

Density-Based Clustering

DBSCAN (sklearn.cluster.DBSCAN)

• Density-Based Spatial Clustering

• Key parameters:

  • eps: Maximum distance between two samples to be neighbors

  • min_samples: Minimum samples in neighborhood to form core point

  • metric: Distance metric

• Use when: Arbitrary cluster shapes, presence of noise/outliers

• Automatically determines number of clusters

• Labels noise points as -1

• Example:

from sklearn.cluster import DBSCAN

model = DBSCAN(eps=0.5, min_samples=5, metric='euclidean')
labels = model.fit_predict(X)

# Number of clusters (excluding noise)
n_clusters = len(set(labels)) - (1 if -1 in labels else 0)
n_noise = list(labels).count(-1)
print(f"Clusters: {n_clusters}, Noise points: {n_noise}")

HDBSCAN (sklearn.cluster.HDBSCAN)

• Hierarchical DBSCAN with adaptive epsilon (added in scikit-learn 1.3)

• Handles varying density; performance and semantics still depend on data and parameters

• scikit-learn includes the point itself in min_samples; the contrib hdbscan package does not. Do not transfer that parameter unchanged between implementations

• Key parameter: min_cluster_size

• Use when: Varying density clusters

• Example:

from sklearn.cluster import HDBSCAN

model = HDBSCAN(min_cluster_size=10, min_samples=5)
labels = model.fit_predict(X)

OPTICS (sklearn.cluster.OPTICS)

• Ordering points to identify clustering structure

• Similar to DBSCAN but doesn't require eps parameter

• Key parameters: min_samples, max_eps

• Use when: Varying density, exploratory analysis

• Example:

from sklearn.cluster import OPTICS

model = OPTICS(min_samples=5, max_eps=0.5)
labels = model.fit_predict(X)

Hierarchical Clustering

AgglomerativeClustering

• Bottom-up hierarchical clustering

• Key parameters:

  • n_clusters: Number of clusters (or use distance_threshold)

  • linkage: 'ward', 'complete', 'average', 'single'

  • metric: Distance metric

• Use when: Need dendrogram, hierarchical structure important

• Example:

from sklearn.cluster import AgglomerativeClustering

model = AgglomerativeClustering(n_clusters=3, linkage='ward')
labels = model.fit_predict(X)

# Create dendrogram using scipy
from scipy.cluster.hierarchy import dendrogram, linkage
Z = linkage(X, method='ward')
dendrogram(Z)

Other Clustering Methods

MeanShift

• Finds clusters by shifting points toward mode of density

• Automatically determines number of clusters

• Key parameter: bandwidth

• Use when: Don't know number of clusters, arbitrary shapes

• Example:

from sklearn.cluster import MeanShift, estimate_bandwidth

# Estimate bandwidth
bandwidth = estimate_bandwidth(X, quantile=0.2, n_samples=500)
model = MeanShift(bandwidth=bandwidth)
labels = model.fit_predict(X)

SpectralClustering

• Uses graph-based approach with eigenvalues

• Key parameters: n_clusters, affinity ('rbf', 'nearest_neighbors')

• Use when: Non-convex clusters, graph structure

• Example:

from sklearn.cluster import SpectralClustering

model = SpectralClustering(n_clusters=3, affinity='rbf', random_state=42)
labels = model.fit_predict(X)

AffinityPropagation

• Finds exemplars by message passing

• Automatically determines number of clusters

• Key parameters: damping, preference

• Use when: Don't know number of clusters

• Example:

from sklearn.cluster import AffinityPropagation

model = AffinityPropagation(damping=0.9, random_state=42)
labels = model.fit_predict(X)
n_clusters = len(model.cluster_centers_indices_)

BIRCH

• Balanced Iterative Reducing and Clustering using Hierarchies

• Memory efficient for large datasets

• Key parameters: n_clusters, threshold, branching_factor

• Use when: Very large datasets

• Example:

from sklearn.cluster import Birch

model = Birch(n_clusters=3, threshold=0.5)
labels = model.fit_predict(X)

Clustering Evaluation

Metrics when ground truth is known:

from sklearn.metrics import adjusted_rand_score, normalized_mutual_info_score
from sklearn.metrics import adjusted_mutual_info_score, fowlkes_mallows_score

# Compare predicted labels with true labels
ari = adjusted_rand_score(y_true, y_pred)
nmi = normalized_mutual_info_score(y_true, y_pred)
ami = adjusted_mutual_info_score(y_true, y_pred)
fmi = fowlkes_mallows_score(y_true, y_pred)

Metrics without ground truth: These require 2 <= n_labels < n_samples. Noise label -1 is otherwise treated as an ordinary cluster: explicitly decide whether to exclude noise and report retained coverage. Scores on different subsets are not directly comparable, and these Euclidean indices favor compact/convex clusters.

from sklearn.metrics import silhouette_score, calinski_harabasz_score
from sklearn.metrics import davies_bouldin_score

# Silhouette: [-1, 1], higher is better
silhouette = silhouette_score(X, labels)

# Calinski-Harabasz: higher is better
ch_score = calinski_harabasz_score(X, labels)

# Davies-Bouldin: lower is better
db_score = davies_bouldin_score(X, labels)

Elbow method for K-Means:

from sklearn.cluster import KMeans
import matplotlib.pyplot as plt

inertias = []
K_range = range(2, 11)
for k in K_range:
    model = KMeans(n_clusters=k, random_state=42)
    model.fit(X)
    inertias.append(model.inertia_)

plt.plot(K_range, inertias, 'bo-')
plt.xlabel('Number of clusters')
plt.ylabel('Inertia')
plt.title('Elbow Method')

Dimensionality Reduction

Principal Component Analysis (PCA)

PCA (sklearn.decomposition.PCA)

• Linear dimensionality reduction using SVD

• Key parameters:

  • n_components: Number of components (int or float for explained variance)

  • whiten: Whiten components to unit variance

• Use when: Linear relationships, want to explain variance

• Example:

from sklearn.decomposition import PCA

# Keep components explaining 95% variance
pca = PCA(n_components=0.95)
X_reduced = pca.fit_transform(X)

print(f"Original dimensions: {X.shape[1]}")
print(f"Reduced dimensions: {X_reduced.shape[1]}")
print(f"Explained variance ratio: {pca.explained_variance_ratio_}")
print(f"Total variance explained: {pca.explained_variance_ratio_.sum()}")

# Or specify exact number of components
pca = PCA(n_components=2)
X_2d = pca.fit_transform(X)

IncrementalPCA

• PCA for large datasets that don't fit in memory

• Processes data in batches

• Key parameter: n_components, batch_size

• Example:

from sklearn.decomposition import IncrementalPCA

pca = IncrementalPCA(n_components=50, batch_size=100)
X_reduced = pca.fit_transform(X)

KernelPCA

• Non-linear dimensionality reduction using kernels

• Key parameters: n_components, kernel ('linear', 'poly', 'rbf', 'sigmoid')

• Use when: Non-linear relationships

• Example:

from sklearn.decomposition import KernelPCA

pca = KernelPCA(n_components=2, kernel='rbf', gamma=0.1)
X_reduced = pca.fit_transform(X)

Manifold Learning

t-SNE (sklearn.manifold.TSNE)

• t-distributed Stochastic Neighbor Embedding

• Excellent for 2D/3D visualization

• Key parameters:

  • n_components: Usually 2 or 3

  • perplexity: Balance between local and global structure (5-50)

  • learning_rate: Usually 10-1000

  • max_iter: Number of iterations (minimum 250; renamed from n_iter in 1.5)

• Use when: Visualizing high-dimensional data

• perplexity must be less than the sample count. No transform() method; refitting on new data changes the embedding

• Example:

from sklearn.manifold import TSNE

tsne = TSNE(n_components=2, perplexity=30, learning_rate=200, max_iter=1000, random_state=42)
X_embedded = tsne.fit_transform(X)

# Visualize
import matplotlib.pyplot as plt
plt.scatter(X_embedded[:, 0], X_embedded[:, 1], c=labels, cmap='viridis')
plt.title('t-SNE visualization')

UMAP (not in scikit-learn, but compatible)

• Uniform Manifold Approximation and Projection

• Runtime and apparent geometry depend on dataset and settings; neither UMAP nor t-SNE reliably preserves global distances or proves clusters

• Install: uv pip install umap-learn

• Example:

from umap import UMAP

reducer = UMAP(n_components=2, n_neighbors=15, min_dist=0.1, random_state=42)
X_embedded = reducer.fit_transform(X)

Isomap

• Isometric Mapping

• Preserves geodesic distances

• Key parameters: n_components, n_neighbors

• Use when: Non-linear manifolds

• Example:

from sklearn.manifold import Isomap

isomap = Isomap(n_components=2, n_neighbors=5)
X_embedded = isomap.fit_transform(X)

Locally Linear Embedding (LLE)

• Preserves local neighborhood structure

• Key parameters: n_components, n_neighbors

• Example:

from sklearn.manifold import LocallyLinearEmbedding

lle = LocallyLinearEmbedding(n_components=2, n_neighbors=10)
X_embedded = lle.fit_transform(X)

MDS (Multidimensional Scaling)

• Preserves pairwise distances

• Key parameters: n_components, metric, metric_params, init (e.g. 'classical_mds' in 1.8+)

• Note: dissimilarity is deprecated; the former metric boolean was renamed to metric_mds

• Example:

from sklearn.manifold import MDS

mds = MDS(n_components=2, metric='euclidean', init='classical_mds', random_state=42)
X_embedded = mds.fit_transform(X)

ClassicalMDS (sklearn.manifold.ClassicalMDS)

• Classical (Torgerson) MDS via eigendecomposition of the double-centered distance matrix

• Added in scikit-learn 1.8; faster initialization path for MDS

• Example:

from sklearn.manifold import ClassicalMDS

cmds = ClassicalMDS(n_components=2)
X_embedded = cmds.fit_transform(X)

Matrix Factorization

NMF (Non-negative Matrix Factorization)

• Factorizes into non-negative matrices

• Key parameters: n_components, init ('nndsvd', 'random')

• Use when: Data is non-negative (images, text)

• Interpretable components

• Example:

from sklearn.decomposition import NMF

nmf = NMF(n_components=10, init='nndsvd', random_state=42)
W = nmf.fit_transform(X)  # Document-topic matrix
H = nmf.components_  # Topic-word matrix

TruncatedSVD

• SVD for sparse matrices

• Unlike PCA, does not center input, preserving sparsity; components need not match centered PCA

• Use when: Text data, sparse matrices

• Example:

from sklearn.decomposition import TruncatedSVD

svd = TruncatedSVD(n_components=100, random_state=42)
X_reduced = svd.fit_transform(X_sparse)
print(f"Explained variance: {svd.explained_variance_ratio_.sum()}")

FastICA

• Independent Component Analysis

• Separates multivariate signal into independent components

• Key parameter: n_components

• Use when: Signal separation (e.g., audio, EEG)

• Example:

from sklearn.decomposition import FastICA

ica = FastICA(n_components=10, random_state=42)
S = ica.fit_transform(X)  # Independent sources
A = ica.mixing_  # Mixing matrix

LatentDirichletAllocation (LDA)

• Topic modeling for text data

• Key parameters: n_components (number of topics), learning_method ('batch', 'online')

• Use when: Topic modeling, document clustering

• Example:

from sklearn.decomposition import LatentDirichletAllocation

lda = LatentDirichletAllocation(n_components=10, random_state=42)
doc_topics = lda.fit_transform(X_counts)  # Document-topic distribution

# Get top words for each topic
feature_names = vectorizer.get_feature_names_out()
for topic_idx, topic in enumerate(lda.components_):
    top_words = [feature_names[i] for i in topic.argsort()[-10:]]
    print(f"Topic {topic_idx}: {', '.join(top_words)}")

Outlier and Novelty Detection

Outlier Detection

IsolationForest

• Isolates anomalies using random trees

• Key parameters:

  • contamination: Expected proportion of outliers

  • n_estimators: Number of trees

• Use when: High-dimensional data, efficiency important

• Example:

from sklearn.ensemble import IsolationForest

model = IsolationForest(contamination=0.1, random_state=42)
predictions = model.fit_predict(X)  # -1 for outliers, 1 for inliers

LocalOutlierFactor

• Measures local density deviation

• Key parameters: n_neighbors, contamination

• Use when: Varying density regions

• Example:

from sklearn.neighbors import LocalOutlierFactor

lof = LocalOutlierFactor(n_neighbors=20, contamination=0.1)
predictions = lof.fit_predict(X)  # -1 for outliers, 1 for inliers
outlier_scores = lof.negative_outlier_factor_

For novelty detection use LocalOutlierFactor(novelty=True).fit(X_train) and call predict/score_samples only on unseen observations. Do not use novelty predictions to reproduce training fit_predict labels.

One-Class SVM

• Learns decision boundary around normal data

• nu bounds the fraction of training margin errors and lower-bounds the support-vector fraction; it is not an estimated test-set outlier prevalence. Other parameters: kernel, gamma

• Use when: Small training set of normal data

• Example:

from sklearn.svm import OneClassSVM

model = OneClassSVM(nu=0.1, kernel='rbf', gamma='auto')
model.fit(X_train)
predictions = model.predict(X_test)  # -1 for outliers, 1 for inliers

EllipticEnvelope

• Assumes Gaussian distribution

• Key parameter: contamination

• Use when: Data is Gaussian-distributed

• Example:

from sklearn.covariance import EllipticEnvelope

model = EllipticEnvelope(contamination=0.1, random_state=42)
predictions = model.fit_predict(X)

Gaussian Mixture Models

GaussianMixture

• Probabilistic clustering with mixture of Gaussians

• Key parameters:

  • n_components: Number of mixture components

  • covariance_type: 'full', 'tied', 'diag', 'spherical'

• Use when: Soft clustering, need probability estimates

• Example:

from sklearn.mixture import GaussianMixture

gmm = GaussianMixture(n_components=3, covariance_type='full', random_state=42)
gmm.fit(X)

# Predict cluster labels
labels = gmm.predict(X)

# Get probability of each cluster
probabilities = gmm.predict_proba(X)

# Information criteria for model selection
print(f"BIC: {gmm.bic(X)}")  # Lower is better
print(f"AIC: {gmm.aic(X)}")  # Lower is better

Choosing the Right Method

Clustering:

• Know K, spherical clusters: K-Means

• Arbitrary shapes, noise: DBSCAN, HDBSCAN

• Hierarchical structure: AgglomerativeClustering

• Very large data: MiniBatchKMeans, BIRCH

• Probabilistic: GaussianMixture

Dimensionality Reduction:

• Linear, variance explanation: PCA

• Non-linear, visualization: t-SNE, UMAP

• Non-negative data: NMF

• Sparse data: TruncatedSVD

• Topic modeling: LatentDirichletAllocation

Outlier Detection:

• High-dimensional: IsolationForest

• Varying density: LocalOutlierFactor

• Gaussian data: EllipticEnvelope

Upstream references

• https://scikit-learn.org/stable/modules/clustering.html

• https://scikit-learn.org/stable/modules/generated/sklearn.cluster.HDBSCAN.html

• https://scikit-learn.org/stable/modules/generated/sklearn.manifold.ClassicalMDS.html

• https://scikit-learn.org/stable/modules/outlier_detection.html

• https://umap-learn.readthedocs.io/en/latest/parameters.html