FIX: propagate dtype to idf_ for large corpuses in PandasNormalizedTfidfVectorizer

[chrispyl]

Issue

Fitting PandasNormalizedTfidfVectorizer for large X returns results in idf_ having dtype np.float64 regardless of the provided dtype. This is due to TfidfTransformer which assumes the resulting dtype of np.log and doesn't use the dtype parameter explicitly.

Reproduction

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
import numpy as np
import uuid

#small data work fine
small_data=[str(uuid.uuid4()) for i in range(100)]
X = pd.Series(small_data)
vectorizer = TfidfVectorizer(dtype=np.float32)
vectorizer.fit(X)
print(vectorizer.idf_.dtype)

#large data do not preserve dtype
large_data=[str(uuid.uuid4()) for i in range(1000000)]
X = pd.Series(large_data)
vectorizer = TfidfVectorizer(dtype=np.float32)
vectorizer.fit(X)
print(vectorizer.idf_.dtype)

Solution

Set the type of idf_ right after fitting and before using it in any other operation to avoid surprises. This solution is temporary until the issue is fixed in scikit-learn.

Sklearn issue background

Not propagating dtype to TfidfTransformer is a known issue. Passing the dtype to np.log is not an option as in this thread they mention that it breaks functionality when dtype is integer. Adding:

.astype(dtype)

after the log operation was proposed but it was removed.

Have opened a new issue in scikit-learn.