How to use Bertopic with a very specific and large data

[RamziRahli]

Hello,
I'm working on the classification of large (2M documents mainly in French and some English) and very noisy data from a specific domain (chats).
I already tested Bertopic on a sample and I felt that I could do interesting things with it.
Knowing that I would like to detect new topics in real time, which variant of Bertopic do you advise me to use? I also tested the GPU version but for reasons of memory I did not succeed in .fit all the dataset

[MaartenGr]

There are several options for doing this. First, you can .fit BERTopic on a subset of the data using the GPU acceleration. Depending on your hardware, I think 500k to 1M should generally be doable if you are using both HDBSCAN and UMAP from cuML. Second, there is an option for online topic modeling with BERTopic. This method allows you to iteratively train on the data using .partial_fit which makes it easier to handle out-of-core datasets.

[RamziRahli]

Thank you for the response.
I tested online topic modeling as recommended but I don't get the same results as with a fit. I want to keep the current topics and make sure that in the future the model can create a new cluster if documents don't match any old topic. I know that it is not possible to do a partial_fit after a fit, but is there any other trick?

[MaartenGr]

I tested online topic modeling as recommended but I don't get the same results as with a fit.

That is correct, online topic modeling uses different sub-models so the results will indeed be different.

I want to keep the current topics and make sure that in the future the model can create a new cluster if documents don't match any old topic. I know that it is not possible to do a partial_fit after a fit, but is there any other trick?

That is currently not possible as it depends fully on the underlying cluster model that is being used. HDBSCAN, for example, does not allow for online learning. It would be possible to gather all outliers and then train a new model based on the outliers.

[RamziRahli]

And what about using partial_fit after a fit_transform, is that possible ? Which means using incrémental topic modeling on the in coming data after fit_transform on the historical data ?

[MaartenGr]

You would still be using HDBSCAN, which does not allow for online learning, so that would also not be possible.

[Thiz-H]

Hello, I 'm also interessted in the answer. I would like to do temporal batch clustering, but I don't want to use Online Topic modeling because the results obtained are not satisfactory. What I want to do is to divide my dataset in several batches by time interval, make Bertopic on each batch with exactly the same configurations of each algorithm (I mean sentence model, clustering, UMAP), then merge the topics between these different batches if they are close based on the cosine similarity. Morever, I would like to perform classification on the incoming data by checking the cosine similaraty with the new texts and the existing clusters and create a new cluster if it is to high. I was wondering if I can do this onely using Bertopic with the GPU versions of UMAP and HDBSCAN. Tank you for your help

[Thiz-H]

Thank you for your reply. I have indeed tested this method by training three models over three batches because I beleive that it works for what I want to do but I have a lot of outliers each time (more than half of the data). I've tried the reduce_outliers methods but most of the documents are moved from class -1 to class 0 or to other clusters than the ones they're supposed to be moved to. I know my data is very domain specific but from the topics I'm getting I'm sure it's possible to get coherent clusters and improve topic representation.

1. Data description : very domain specific with a lot of abreviations and miss spelled word (French about 70% and English aout 30%). As pre-processing ,I am onlyperforming text normalization, deleting duplicates and lowercase all the data.
2. Models : each model is trained on a batch of 500000 documents and I am using the same configuration every time :

`sentence_model = SentenceTransformer('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')
 umap_model = UMAP(n_neighbors=80, n_components=10, metric='cosine', verbose=True)
 hdbscan_model = HDBSCAN(min_cluster_size=150, metric='euclidean', prediction_data=True, verbose=True)
 vectorizer_model = CountVectorizer(ngram_range=(1, 3),  stop_words=stopwords)
 ctfidf_model = ClassTfidfTransformer(reduce_frequent_words=True)
 representation_model = KeyBERTInspired()
 topic_model = BERTopic(
    embedding_model=sentence_model, 
    umap_model=umap_model, 
    hdbscan_model =hdbscan_model, 
    vectorizer_model=vectorizer_model, 
    ctfidf_model = ctfidf_model,
    representation_model=representation_model,
    calculate_probabilities=True, 
    nr_topics= 'auto', 
    verbose=True )

 topics, probs = topic_model.fit_transform(data)`

I don't know what I'm doing wrong but I get different results as I change some paremeters values and can't decide. So I hope you can help me or advice me. Thank you so much

[MaartenGr]

I've tried the reduce_outliers methods but most of the documents are moved from class -1 to class 0 or to other clusters than the ones they're supposed to be moved to. I

Have you tried all the different strategies for .reduce_outliers? Also, can you share your code?

I don't know what I'm doing wrong but I get different results as I change some paremeters values and can't decide

Make sure to set random_state in UMAP to get the same results each time. You can find more about that in the FAQ.

A tip, it might be worthwhile to not use nr_topics="auto" but to remove this parameter. It tends to aggregate a lot of individual topics which makes the assignment a bit more difficult. Instead, I would advise focusing on setting the min_cluster_size to control the number of clusters instead.

[Thiz-H]

Yeah, I tried to chain strategies c-tf-idf and 'distributions' but I get inconsistent clusters. For example, a document that is very similar to document tu cluster 30 will be classified in cluster 0. I also tried the embedding method but I also get inconsistent clusters.
Do you think that BERTopic may not be the best choice when dealing with domain data?

[MaartenGr]

It has actually worked well for me with domain data but it might be worthwhile to instead then use an embedding model that is specific to your domain. Since that is the main source for representing the data, using a different model might be better.

Have you also tried it without nr_topics="auto"? I think that should improve it quite a bit.

[Thiz-H]

I've already tried it without nr_topics="auto", but it didn't improve much. I'm testing a model fietuned on my data, I'll keep you informed about the results. Thanks again for your help.