Number of clusters does not match with the number of selected areas in condensed_tree

[cyr31000]

Hi,

I have started to use hdbscan for a couple of weeks. Sometimes, I got for examples 3 clusters, but when I plot the condensed_tree with the option select_, clusters= True, I have seen that 6 clusters are selected (6 ellipses are drew) in the condensed_tree. It's like sometimes the condensed_tree is not updated. Any ideas? I've checked the corresponding .pyx files but no more informations...
Thank you,

cyr31000

[JelmerBot]

That sounds odd, indeed. Could you provide a small example script that reproduces this behaviour? And which version are you using? The latest release or a more recent branch?

This is the function in the current main branch that decides which ellipses to draw:

hdbscan/hdbscan/plots.py

Lines 235 to 245 in 8ecf239

	def _select_clusters(self):
	points_tree = self._raw_tree[self._raw_tree["child_size"] == 1]
	# Find lowest cluster segment id for each cluster label
	segments = points_tree['parent']
	labels = self._labels[points_tree['child']]
	order = np.argsort(labels)
	groups = np.split(segments[order], np.flatnonzero(np.diff(labels[order]) != 0) + 1)
	return [
	groups[label].min() for label in range(int(labels[order[0]] == -1), len(groups))
	]

It uses the cluster label vector to find the correct segment for each cluster. That cluster label vector is read from the HDBSCAN object each time the condensed_tree_ property is accessed. So, a CondensedTree object can become outdated if the HDBSCAN object changes after its creation:

# create clusterer and tree object
clusterer = HDBSCAN().fit(data)
tree = clusterer.condensed_tree_

# change clustering after tree creation
clusterer.cluster_selection_method='leaf'
clusterer.fit(data)

# outdated figure
tree.plot(select_clusters=True)
# updated figure
clusterer.condensed_tree_.plot(select_clusters=True)

[cyr31000]

Hello JelmerBot,
I'm using Python 3.11.10 and one of the last release of hdbscan (0.8.39). I used the PCA method before running hdbscan (data have been normalized before PCA). This is an extract of the script:

Then i ran the command to display the number of clusters:

So it displays 4 clusters. But when i ran the command spectr.condensed_tree_.plot(select_clusters=True), the following plot appeared:

It surrounded almost all of the leaves.

[JelmerBot]

It looks like you are using the cluster_selection_epsilon parameter. The CondensedTree in version 0.8.39 does not know about that parameter, so it selects different clusters when cluster_selection_epsilon is used. This issue should already be solved on the main branch using the code snippet in my previous comment.

Could you try to recreate the plot with HDBSCAN installed from git to see if it works as expected? If that is not feasible, you could test it by copying the plots.py file to your project and instantiating its CondensedTree manually:

from plots.py import CondensedTree

CondensedTree(spectr._condensed_tree, spectr.labels_).plot(select_clusters=True)