Python Optimizing Topic Modeling: Saving and Loading Models for Faster Processing

  • Thread starter Thread starter EngWiPy
  • Start date Start date
  • Tags Tags
    Modeling Topic
Click For Summary
The discussion centers on optimizing the process of creating a topic model using Gensim's LDA implementation. The user initially faces slow performance when repeatedly running the code to read a large corpus and generate the model. A solution is found by saving the trained model, allowing for quicker loading in subsequent runs, thus significantly improving efficiency. This approach effectively addresses the issue of long processing times associated with model training.
EngWiPy
Messages
1,361
Reaction score
61
Hello,

I am running a program that takes relatively long time to read the corpus of some documents, and create the topic model. The code to do this is:

Code:
from gensim import models, corpora

corpus = corpora.BleiCorpus('./data/ap/ap.dat', './data/ap/vocab.txt')

#Creating the topic model
model = models.ldamodel.LdaModel(corpus, num_topics = 100, id2word = corpus.id2word)

I am testing different things with the code, and it's a little slow to run the code several times because of the above code's segment. How can I overcome this issue?

Thanks
 
Technology news on Phys.org
It turned out the model can be saved and loaded. Problem solved.
 
  • Like
Likes jim mcnamara and berkeman
Learn If you want to write code for Python Machine learning, AI Statistics/data analysis Scientific research Web application servers Some microcontrollers JavaScript/Node JS/TypeScript Web sites Web application servers C# Games (Unity) Consumer applications (Windows) Business applications C++ Games (Unreal Engine) Operating systems, device drivers Microcontrollers/embedded systems Consumer applications (Linux) Some more tips: Do not learn C++ (or any other dialect of C) as a...

Similar threads

  • · Replies 1 ·
Replies
1
Views
2K
  • · Replies 22 ·
Replies
22
Views
2K
  • · Replies 1 ·
Replies
1
Views
3K
  • · Replies 3 ·
Replies
3
Views
2K
  • · Replies 15 ·
Replies
15
Views
2K
Replies
4
Views
3K
  • · Replies 6 ·
Replies
6
Views
4K
  • · Replies 0 ·
Replies
0
Views
1K
  • · Replies 2 ·
Replies
2
Views
2K