I have two tfidf models which are based on word and character level
self.char_tfidf = TfidfVectorizer(analyzer='char', lowercase=True, max_features=2000000, ngram_range=char_ngram_range, dtype=np.float32) self.word_tfidf = TfidfVectorizer(analyzer='word', lowercase=True, max_features=1000000, ngram_range=(1, 1), dtype=np.float32, stop_words=stopwords.words('english'), token_pattern='[a-zA-Z0-9_]{1,}') .... LOGGER.info('Training generator...') LOGGER.info('Training Character-based TFIDF') self.char_tfidf.fit(tqdm(corpus)) LOGGER.info('Training Word-based TFIDF') self.word_tfidf.fit(tqdm(corpus)) LOGGER.info('Finish training.')
You can see the training character-based TFIDF just took 10 minutes but it got stuck there 2 hours until the training word-based starts.
I don't understand the behavior of sklearn behind this to improve the running time. Does anyone have any idea?
there is an image attached this post. I'm not sure you can see it because I don't see it either, but I can see it in private browser mode
Источник: https://stackoverflow.com/questions/780 ... sh-fitting
