On this tutorial, we current an entire end-to-end Pure Language Processing (NLP) pipeline constructed with Gensim and supporting libraries, designed to run seamlessly in Google Colab. It integrates a number of core methods in fashionable NLP, together with preprocessing, matter modeling with Latent Dirichlet Allocation (LDA), phrase embeddings with Word2Vec, TF-IDF-based similarity evaluation, and semantic search. The pipeline not solely demonstrates the way to prepare and consider these fashions but in addition showcases sensible visualizations, superior matter evaluation, and doc classification workflows. By combining statistical strategies with machine studying approaches, the tutorial gives a complete framework for understanding and experimenting with textual content knowledge at scale. Try the FULL CODES right here.
!pip set up --upgrade scipy==1.11.4
!pip set up gensim==4.3.2 nltk wordcloud matplotlib seaborn pandas numpy scikit-learn
!pip set up --upgrade setuptools
print("Please restart runtime after set up!")
print("Go to Runtime > Restart runtime, then run the subsequent cell")
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from wordcloud import WordCloud
import warnings
warnings.filterwarnings('ignore')
from gensim import corpora, fashions, similarities
from gensim.fashions import Word2Vec, LdaModel, TfidfModel, CoherenceModel
from gensim.parsing.preprocessing import preprocess_string, strip_tags, strip_punctuation, strip_multiple_whitespaces, strip_numeric, remove_stopwords, strip_short
import nltk
nltk.obtain('punkt', quiet=True)
nltk.obtain('stopwords', quiet=True)
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenizeWe set up and improve the mandatory libraries, similar to SciPy, Gensim, NLTK, and visualization instruments, to make sure compatibility. We then import all required modules for preprocessing, modeling, and evaluation. We additionally obtain NLTK sources to tokenize and deal with stopwords effectively, thereby organising the surroundings for our NLP pipeline. Try the FULL CODES right here.
class AdvancedGensimPipeline:
def __init__(self):
self.dictionary = None
self.corpus = None
self.lda_model = None
self.word2vec_model = None
self.tfidf_model = None
self.similarity_index = None
self.processed_docs = None
def create_sample_corpus(self):
"""Create a various pattern corpus for demonstration"""
paperwork = [ "Data science combines statistics, programming, and domain expertise to extract insights",
"Big data analytics helps organizations make data-driven decisions at scale",
"Cloud computing provides scalable infrastructure for modern applications and services",
"Cybersecurity protects digital systems from threats and unauthorized access attempts",
"Software engineering practices ensure reliable and maintainable code development",
"Database management systems store and organize large amounts of structured information",
"Python programming language is widely used for data analysis and machine learning",
"Statistical modeling helps identify patterns and relationships in complex datasets",
"Cross-validation techniques ensure robust model performance evaluation and selection",
"Recommendation systems suggest relevant items based on user preferences and behavior",
"Text mining extracts valuable insights from unstructured textual data sources",
"Image classification assigns predefined categories to visual content automatically",
"Reinforcement learning trains agents through interaction with dynamic environments"
]
return paperwork
def preprocess_documents(self, paperwork):
"""Superior doc preprocessing utilizing Gensim filters"""
print("Preprocessing paperwork...")
CUSTOM_FILTERS = [
strip_tags, strip_punctuation, strip_multiple_whitespaces,
strip_numeric, remove_stopwords, strip_short, lambda x: x.lower()
]
processed_docs = []
for doc in paperwork:
processed = preprocess_string(doc, CUSTOM_FILTERS)
stop_words = set(stopwords.phrases('english'))
processed = [word for word in processed if word not in stop_words and len(word) > 2]
processed_docs.append(processed)
self.processed_docs = processed_docs
print(f"Processed {len(processed_docs)} paperwork")
return processed_docs
def create_dictionary_and_corpus(self):
"""Create Gensim dictionary and corpus"""
print("Creating dictionary and corpus...")
self.dictionary = corpora.Dictionary(self.processed_docs)
self.dictionary.filter_extremes(no_below=2, no_above=0.8)
self.corpus = [self.dictionary.doc2bow(doc) for doc in self.processed_docs]
print(f"Dictionary dimension: {len(self.dictionary)}")
print(f"Corpus dimension: {len(self.corpus)}")
def train_word2vec_model(self):
"""Practice Word2Vec mannequin for phrase embeddings"""
print("Coaching Word2Vec mannequin...")
self.word2vec_model = Word2Vec(
sentences=self.processed_docs,
vector_size=100,
window=5,
min_count=2,
staff=4,
epochs=50
)
print("Word2Vec mannequin educated efficiently")
def analyze_word_similarities(self):
"""Analyze phrase similarities utilizing Word2Vec"""
print("n=== Word2Vec Similarity Evaluation ===")
test_words = ['machine', 'data', 'learning', 'computer']
for phrase in test_words:
if phrase in self.word2vec_model.wv:
similar_words = self.word2vec_model.wv.most_similar(phrase, topn=3)
print(f"Phrases just like '{phrase}': {similar_words}")
attempt:
if all(w in self.word2vec_model.wv for w in ['machine', 'computer', 'data']):
analogy = self.word2vec_model.wv.most_similar(
constructive=['computer', 'data'],
detrimental=['machine'],
topn=1
)
print(f"Analogy outcome: {analogy}")
besides:
print("Not sufficient vocabulary for advanced analogies")
def train_lda_model(self, num_topics=5):
"""Practice LDA matter mannequin"""
print(f"Coaching LDA mannequin with {num_topics} matters...")
self.lda_model = LdaModel(
corpus=self.corpus,
id2word=self.dictionary,
num_topics=num_topics,
random_state=42,
passes=10,
alpha="auto",
per_word_topics=True,
eval_every=None
)
print("LDA mannequin educated efficiently")
def evaluate_topic_coherence(self):
"""Consider matter mannequin coherence"""
print("Evaluating matter coherence...")
coherence_model = CoherenceModel(
mannequin=self.lda_model,
texts=self.processed_docs,
dictionary=self.dictionary,
coherence="c_v"
)
coherence_score = coherence_model.get_coherence()
print(f"Subject Coherence Rating: {coherence_score:.4f}")
return coherence_score
def display_topics(self):
"""Show found matters"""
print("n=== Found Subjects ===")
matters = self.lda_model.print_topics(num_words=8)
for idx, matter in enumerate(matters):
print(f"Subject {idx}: {matter[1]}")
def create_tfidf_model(self):
"""Create TF-IDF mannequin for doc similarity"""
print("Creating TF-IDF mannequin...")
self.tfidf_model = TfidfModel(self.corpus)
corpus_tfidf = self.tfidf_model[self.corpus]
self.similarity_index = similarities.MatrixSimilarity(corpus_tfidf)
print("TF-IDF mannequin and similarity index created")
def find_similar_documents(self, query_doc_idx=0):
"""Discover paperwork just like a question doc"""
print(f"n=== Doc Similarity Evaluation ===")
query_doc_tfidf = self.tfidf_model[self.corpus[query_doc_idx]]
similarities_scores = self.similarity_index[query_doc_tfidf]
sorted_similarities = sorted(enumerate(similarities_scores), key=lambda x: x[1], reverse=True)
print(f"Paperwork most just like doc {query_doc_idx}:")
for doc_idx, similarity in sorted_similarities[:5]:
print(f"Doc {doc_idx}: {similarity:.4f}")
def visualize_topics(self):
"""Create visualizations for matter evaluation"""
print("Creating matter visualizations...")
doc_topic_matrix = []
for doc_bow in self.corpus:
doc_topics = dict(self.lda_model.get_document_topics(doc_bow, minimum_probability=0))
topic_vec = [doc_topics.get(i, 0) for i in range(self.lda_model.num_topics)]
doc_topic_matrix.append(topic_vec)
doc_topic_df = pd.DataFrame(doc_topic_matrix, columns=[f'Topic_{i}' for i in range(self.lda_model.num_topics)])
plt.determine(figsize=(12, 8))
sns.heatmap(doc_topic_df.T, annot=True, cmap='Blues', fmt=".2f")
plt.title('Doc-Subject Distribution Heatmap')
plt.xlabel('Paperwork')
plt.ylabel('Subjects')
plt.tight_layout()
plt.present()
fig, axes = plt.subplots(2, 3, figsize=(15, 10))
axes = axes.flatten()
for topic_id in vary(min(6, self.lda_model.num_topics)):
topic_words = dict(self.lda_model.show_topic(topic_id, topn=20))
wordcloud = WordCloud(
width=300, top=200,
background_color="white",
colormap='viridis'
).generate_from_frequencies(topic_words)
axes[topic_id].imshow(wordcloud, interpolation='bilinear')
axes[topic_id].set_title(f'Subject {topic_id}')
axes[topic_id].axis('off')
for i in vary(self.lda_model.num_topics, 6):
axes[i].axis('off')
plt.tight_layout()
plt.present()
def advanced_topic_analysis(self):
"""Carry out superior matter evaluation"""
print("n=== Superior Subject Evaluation ===")
topic_distributions = []
for i, doc_bow in enumerate(self.corpus):
doc_topics = self.lda_model.get_document_topics(doc_bow)
dominant_topic = max(doc_topics, key=lambda x: x[1]) if doc_topics else (0, 0)
topic_distributions.append({
'doc_id': i,
'dominant_topic': dominant_topic[0],
'topic_probability': dominant_topic[1]
})
topic_df = pd.DataFrame(topic_distributions)
plt.determine(figsize=(10, 6))
topic_counts = topic_df['dominant_topic'].value_counts().sort_index()
plt.bar(vary(len(topic_counts)), topic_counts.values)
plt.xlabel('Subject ID')
plt.ylabel('Variety of Paperwork')
plt.title('Distribution of Dominant Subjects Throughout Paperwork')
plt.xticks(vary(len(topic_counts)), [f'Topic {i}' for i in topic_counts.index])
plt.present()
return topic_df
def document_classification_demo(self, new_document):
"""Classify a brand new doc utilizing educated fashions"""
print(f"n=== Doc Classification Demo ===")
print(f"Classifying: '{new_document[:50]}...'")
processed_new = preprocess_string(new_document, [
strip_tags, strip_punctuation, strip_multiple_whitespaces,
strip_numeric, remove_stopwords, strip_short, lambda x: x.lower()
])
new_doc_bow = self.dictionary.doc2bow(processed_new)
doc_topics = self.lda_model.get_document_topics(new_doc_bow)
print("Subject possibilities:")
for topic_id, prob in doc_topics:
print(f" Subject {topic_id}: {prob:.4f}")
new_doc_tfidf = self.tfidf_model[new_doc_bow]
similarities_scores = self.similarity_index[new_doc_tfidf]
most_similar = np.argmax(similarities_scores)
print(f"Most related doc: {most_similar} (similarity: {similarities_scores[most_similar]:.4f})")
return doc_topics, most_similar
def run_complete_pipeline(self):
"""Execute the entire NLP pipeline"""
print("=== Superior Gensim NLP Pipeline ===n")
raw_documents = self.create_sample_corpus()
self.preprocess_documents(raw_documents)
self.create_dictionary_and_corpus()
self.train_word2vec_model()
self.train_lda_model(num_topics=5)
self.create_tfidf_model()
self.analyze_word_similarities()
coherence_score = self.evaluate_topic_coherence()
self.display_topics()
self.visualize_topics()
topic_df = self.advanced_topic_analysis()
self.find_similar_documents(query_doc_idx=0)
new_doc = "Deep neural networks are highly effective machine studying fashions for sample recognition"
self.document_classification_demo(new_doc)
return {
'coherence_score': coherence_score,
'topic_distributions': topic_df,
'fashions': {
'lda': self.lda_model,
'word2vec': self.word2vec_model,
'tfidf': self.tfidf_model
}
}We outline the AdvancedGensimPipeline class as a modular framework to deal with each stage of textual content evaluation in a single place. It begins with making a pattern corpus, preprocessing it, after which constructing a dictionary and corpus representations. We prepare Word2Vec for embeddings, LDA for matter modeling, and TF-IDF for similarity, adopted by visualization, coherence analysis, and classification of recent paperwork. This fashion, we carry collectively the entire NLP workflow, from uncooked textual content to insights, right into a single reusable pipeline. Try the FULL CODES right here.
def compare_topic_models(pipeline, topic_range=[3, 5, 7, 10]):
print("n=== Subject Mannequin Comparability ===")
coherence_scores = []
perplexity_scores = []
for num_topics in topic_range:
lda_temp = LdaModel(
corpus=pipeline.corpus,
id2word=pipeline.dictionary,
num_topics=num_topics,
random_state=42,
passes=10,
alpha="auto"
)
coherence_model = CoherenceModel(
mannequin=lda_temp,
texts=pipeline.processed_docs,
dictionary=pipeline.dictionary,
coherence="c_v"
)
coherence = coherence_model.get_coherence()
coherence_scores.append(coherence)
perplexity = lda_temp.log_perplexity(pipeline.corpus)
perplexity_scores.append(perplexity)
print(f"Subjects: {num_topics}, Coherence: {coherence:.4f}, Perplexity: {perplexity:.4f}")
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 5))
ax1.plot(topic_range, coherence_scores, 'bo-')
ax1.set_xlabel('Variety of Subjects')
ax1.set_ylabel('Coherence Rating')
ax1.set_title('Mannequin Coherence vs Variety of Subjects')
ax1.grid(True)
ax2.plot(topic_range, perplexity_scores, 'ro-')
ax2.set_xlabel('Variety of Subjects')
ax2.set_ylabel('Perplexity')
ax2.set_title('Mannequin Perplexity vs Variety of Subjects')
ax2.grid(True)
plt.tight_layout()
plt.present()
return coherence_scores, perplexity_scoresThis operate compare_topic_models lets us systematically take a look at completely different numbers of matters for the LDA mannequin and evaluate their efficiency. We calculate coherence scores (to test matter interpretability) and perplexity scores (to test mannequin match) for every matter depend within the given vary. The outcomes are displayed as line plots, serving to us visually resolve essentially the most balanced variety of matters for our dataset. Try the FULL CODES right here.
def semantic_search_engine(pipeline, question, top_k=5):
"""Implement semantic search utilizing educated fashions"""
print(f"n=== Semantic Search: '{question}' ===")
processed_query = preprocess_string(question, [
strip_tags, strip_punctuation, strip_multiple_whitespaces,
strip_numeric, remove_stopwords, strip_short, lambda x: x.lower()
])
query_bow = pipeline.dictionary.doc2bow(processed_query)
query_tfidf = pipeline.tfidf_model[query_bow]
similarities_scores = pipeline.similarity_index[query_tfidf]
top_indices = np.argsort(similarities_scores)[::-1][:top_k]
print("High matching paperwork:")
for i, idx in enumerate(top_indices):
rating = similarities_scores[idx]
print(f"{i+1}. Doc {idx} (Rating: {rating:.4f})")
print(f" Content material: {' '.be a part of(pipeline.processed_docs[idx][:10])}...")
return top_indices, similarities_scores[top_indices]The semantic_search_engine operate provides a search layer to the pipeline by taking a question, preprocessing it, and changing it right into a bag-of-words and TF-IDF representations. It then compares the question in opposition to all paperwork utilizing the similarity index and returns the highest matches. This fashion, we will rapidly retrieve essentially the most related paperwork together with their similarity scores, making the pipeline helpful for sensible info retrieval and semantic search duties. Try the FULL CODES right here.
if __name__ == "__main__":
pipeline = AdvancedGensimPipeline()
outcomes = pipeline.run_complete_pipeline()
print("n" + "="*60)
coherence_scores, perplexity_scores = compare_topic_models(pipeline)
print("n" + "="*60)
search_results = semantic_search_engine(
pipeline,
"synthetic intelligence neural networks deep studying"
)
print("n" + "="*60)
print("Pipeline accomplished efficiently!")
print(f"Last coherence rating: {outcomes['coherence_score']:.4f}")
print(f"Vocabulary dimension: {len(pipeline.dictionary)}")
print(f"Word2Vec mannequin dimension: {pipeline.word2vec_model.wv.vector_size} dimensions")
print("nModels educated and prepared to be used!")
print("Entry fashions by way of: pipeline.lda_model, pipeline.word2vec_model, pipeline.tfidf_model")This foremost block ties every little thing collectively into an entire, executable pipeline. We initialize the AdvancedGensimPipeline, run the total workflow, after which consider matter fashions with completely different numbers of matters. Subsequent, we take a look at the semantic search engine with a question about synthetic intelligence and deep studying. Lastly, it prints out abstract metrics, such because the coherence rating, vocabulary dimension, and Word2Vec embedding dimensions, confirming that each one fashions are educated and prepared for additional use.
In conclusion, we acquire a strong, modular workflow that covers the complete spectrum of textual content evaluation, from cleansing and preprocessing uncooked paperwork to discovering hidden matters, visualizing outcomes, evaluating fashions, and performing semantic search. The inclusion of Word2Vec embeddings, TF-IDF similarity, and coherence analysis ensures that the pipeline is each versatile and strong, whereas visualizations and classification demos make the outcomes interpretable and actionable. This cohesive design permits learners, researchers, and practitioners to rapidly adapt the framework for real-world functions, making it a priceless basis for superior NLP experimentation and production-ready textual content analytics.
Try the FULL CODES right here. Be happy to take a look at our GitHub Web page for Tutorials, Codes and Notebooks. Additionally, be happy to observe us on Twitter and don’t overlook to hitch our 100k+ ML SubReddit and Subscribe to our E-newsletter.
Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is dedicated to harnessing the potential of Synthetic Intelligence for social good. His most up-to-date endeavor is the launch of an Synthetic Intelligence Media Platform, Marktechpost, which stands out for its in-depth protection of machine studying and deep studying information that’s each technically sound and simply comprehensible by a large viewers. The platform boasts of over 2 million month-to-month views, illustrating its reputation amongst audiences.
