Summary:<\/strong><\/p>\n You know how, back in the day, we used simple word\u2010count tricks to represent text? Well, things have come a long way since then. Now, when we talk about the evolution of embeddings, we mean numerical snapshots that capture not just which words appear but what they really mean, how they relate to each other in context, and even how they tie into images and other media. Embeddings power everything from search engines that understand your intent to recommendation systems that seem to read your mind. They\u2019re at the heart of cutting\u2010edge AI and machine\u2010learning applications, too. So, let\u2019s take a stroll through this evolution from raw counts to semantic vectors, exploring how each approach works, what it brings to the table, and where it falls short.<\/p>\n Most modern LLMs generate embeddings as intermediate outputs of their architectures. These can be extracted and fine-tuned for various downstream tasks, making LLM-based embeddings one of the most versatile tools available today.<\/p>\n To keep up with the fast-moving landscape, platforms like Hugging Face<\/a> have introduced resources like the Massive Text Embedding Benchmark (MTEB) Leaderboard. This leaderboard ranks embedding models based on their performance across a wide range of tasks, including classification, clustering, retrieval, and more. This is substantially helping practitioners identify the best models for their use cases.<\/p>\n Armed with these leaderboard insights, let\u2019s roll up our sleeves and dive into the vectorization toolbox \u2013 count vectors, TF\u2013IDF, and other classic methods, which still serve as the essential building blocks for today\u2019s sophisticated embeddings.<\/p>\n Count Vectorization<\/a> is one of the simplest techniques for representing text. It emerged from the need to convert raw text into numerical form so that machine learning models could process it. In this method, each document is transformed into a vector that reflects the count of each word appearing in it. This straightforward approach laid the groundwork for more complex representations and is still useful in scenarios where interpretability is key.<\/p>\n Output:<\/strong><\/p>\n One-hot encoding<\/a> is one of the earliest approaches to representing words as vectors. Developed alongside early digital computing techniques in the 1950s and 1960s, it transforms categorical data, such as words, into binary vectors. Each word is represented uniquely, ensuring that no two words share similar representations, though this comes at the expense of capturing semantic similarity.<\/p>\n Output:<\/strong><\/p>\n So, basically, you can view the difference between Count Vectorizer and One Hot Encoding. Count Vectorizer counts how many times a certain word exists in a sentence, whereas One Hot Encoding labels the word as 1 if it exists in a certain sentence\/document.<\/p>\n TF-IDF<\/a> was developed to improve upon raw count methods by counting word occurrences and weighing words based on their overall importance in a corpus. Introduced in the early 1970s, TF-IDF is a cornerstone in information retrieval systems and text mining applications. It helps highlight terms that are significant in individual documents while downplaying words that are common across all documents.<\/p>\n Output:<\/strong><\/p>\n Also Read: Implementing Count Vectorizer and TF-IDF in NLP using PySpark<\/a><\/p>\n Okapi BM25, developed in the 1990s, is a probabilistic model designed primarily for ranking documents in information retrieval systems rather than as an embedding method per se. BM25 is an enhanced version of TF-IDF, commonly used in search engines and information retrieval. It improves upon TF-IDF by considering document length normalization and saturation of term frequency (i.e., diminishing returns for repeated words).<\/p>\n Here we will be looking into the BM25 scoring mechanism:<\/p>\n BM25 introduces two parameters, k1 and b, which allow fine-tuning of the term frequency saturation and the length normalization, respectively. These parameters are crucial for optimizing the BM25 algorithm\u2019s performance in various search contexts.<\/p>\n Output:<\/strong><\/p>\n Output:<\/strong><\/p>\n Also Read: How to Create NLP Search Engine With BM25?<\/a><\/p>\n Introduced by Google in 2013, Word2Vec<\/a> revolutionized NLP by learning dense, low-dimensional vector representations of words. It moved beyond counting and weighting by training shallow neural networks that capture semantic and syntactic relationships based on word context. Word2Vec comes in two flavors: Continuous Bag-of-Words (CBOW) and Skip-gram.<\/p>\n Output:<\/strong><\/p>\n Visualizing the CBOW and Skip-gram:<\/p>\n Output:<\/strong><\/p>\n Output:<\/strong><\/p>\n To read more about Word2Vec read this<\/a> blog.<\/p>\n GloVe, developed at Stanford in 2014, builds on the ideas of Word2Vec by combining global co-occurrence statistics with local context information. It was designed to produce word embeddings that capture overall corpus-level statistics, offering improved consistency across different contexts.<\/p>\n This logic of Co-occurence matrices are also widely used in Computer Vision too, especially under the topic of GLCM(Gray-Level Co-occurrence Matrix). It is a statistical method used in image processing and computer vision for texture analysis that considers the spatial relationship between pixels.<\/p>\n<\/li>\n Output:<\/strong><\/p>\n This image will help you understand how this similarity looks like when plotted:<\/p>\n Do refer to this<\/a> for more in-depth information.<\/p>\n\n
Ranking of Embeddings in MTEB Leaderboards <\/h2>\n
![\"Ranking](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/04\/02-image9.webp\")
<\/figure>\n<\/div>\n![\"Ranking](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/04\/03-image39.webp\")
<\/figure>\n1. Count Vectorization<\/h2>\n
How It Works<\/h3>\n
\n
\n
Code Implementation<\/h3>\n
from sklearn.feature_extraction.text import CountVectorizer\n\nimport pandas as pd\n\n# Sample text documents with repeated words\n\ndocuments = [\n\n\t\"Natural Language Processing is fun and natural natural natural\",\n\n\t\"I really love love love Natural Language Processing Processing Processing\",\n\n\t\"Machine Learning is a part of AI AI AI AI\",\n\n\t\"AI and NLP NLP NLP are closely related related\"\n\n]\n\n# Initialize CountVectorizer\n\nvectorizer = CountVectorizer()\n\n# Fit and transform the text data\n\nX = vectorizer.fit_transform(documents)\n\n# Get feature names (unique words)\n\nfeature_names = vectorizer.get_feature_names_out()\n\n# Convert to DataFrame for better visualization\n\ndf = pd.DataFrame(X.toarray(), columns=feature_names)\n\n# Print the matrix\n\nprint(df)<\/code><\/pre>\n![\"Count](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/04\/04-image24.webp\")
<\/figure>\n<\/div>\nBenefits<\/h3>\n
\n
Shortcomings<\/h3>\n
\n
2. One-Hot Encoding<\/h2>\n
How It Works<\/h3>\n
\n
\n
Code Implementation<\/h3>\n
from sklearn.feature_extraction.text import CountVectorizer\n\nimport pandas as pd\n\n# Sample text documents\n\ndocuments = [\n\n\u00a0\u00a0\u00a0\"Natural Language Processing is fun and natural natural natural\",\n\n\u00a0\u00a0\u00a0\"I really love love love Natural Language Processing Processing Processing\",\n\n\u00a0\u00a0\u00a0\"Machine Learning is a part of AI AI AI AI\",\n\n\u00a0\u00a0\u00a0\"AI and NLP NLP NLP are closely related related\"\n\n]\n\n# Initialize CountVectorizer with binary=True for One-Hot Encoding\n\nvectorizer = CountVectorizer(binary=True)\n\n# Fit and transform the text data\n\nX = vectorizer.fit_transform(documents)\n\n# Get feature names (unique words)\n\nfeature_names = vectorizer.get_feature_names_out()\n\n# Convert to DataFrame for better visualization\n\ndf = pd.DataFrame(X.toarray(), columns=feature_names)\n\n# Print the one-hot encoded matrix\n\nprint(df)<\/code><\/pre>\n![\"One-Hot](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/04\/05-image19.webp\")
<\/figure>\n<\/div>\n![\"One-Hot](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/04\/06-image22.webp\")
<\/figure>\n<\/div>\nWhen to Use What?<\/h4>\n
\n
Benefits<\/h3>\n
\n
Shortcomings<\/h3>\n
\n
3. TF-IDF (Term Frequency-Inverse Document Frequency)<\/h2>\n
How It Works<\/h3>\n
\n
\n
\n
Code Implementation<\/h3>\n
from sklearn.feature_extraction.text import TfidfVectorizer\n\nimport pandas as pd\n\nimport numpy as np\n\n# Sample short sentences\n\ndocuments = [\n\n\u00a0\u00a0\u00a0\"cat sits here\",\n\n\u00a0\u00a0\u00a0\"dog barks loud\",\n\n\u00a0\u00a0\u00a0\"cat barks loud\"\n\n]\n\n# Initialize TfidfVectorizer to get both TF and IDF\n\nvectorizer = TfidfVectorizer()\n\n# Fit and transform the text data\n\nX = vectorizer.fit_transform(documents)\n\n# Extract feature names (unique words)\n\nfeature_names = vectorizer.get_feature_names_out()\n\n# Get TF matrix (raw term frequencies)\n\ntf_matrix = X.toarray()\n\n# Compute IDF values manually\n\nidf_values = vectorizer.idf_\n\n# Compute TF-IDF manually (TF * IDF)\n\ntfidf_matrix = tf_matrix * idf_values\n\n# Convert to DataFrames for better visualization\n\ndf_tf = pd.DataFrame(tf_matrix, columns=feature_names)\n\ndf_idf = pd.DataFrame([idf_values], columns=feature_names)\n\ndf_tfidf = pd.DataFrame(tfidf_matrix, columns=feature_names)\n\n# Print tables\n\nprint(\"\\n\ud83d\udd39 Term Frequency (TF) Matrix:\\n\", df_tf)\n\nprint(\"\\n\ud83d\udd39 Inverse Document Frequency (IDF) Values:\\n\", df_idf)\n\nprint(\"\\n\ud83d\udd39 TF-IDF Matrix (TF * IDF):\\n\", df_tfidf)<\/code><\/pre>\n![\"Evolution](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/04\/10-image40.webp\")
<\/figure>\n<\/div>\nBenefits<\/h3>\n
\n
Shortcomings<\/h3>\n
\n
4. Okapi BM25<\/h2>\n
How It Works<\/h3>\n
\n
\n
\n
Code Implementation<\/h3>\n
import numpy as np\n\nimport pandas as pd\n\nfrom sklearn.feature_extraction.text import CountVectorizer\n\n# Sample documents\n\ndocuments = [\n\n\u00a0\u00a0\u00a0\"cat sits here\",\n\n\u00a0\u00a0\u00a0\"dog barks loud\",\n\n\u00a0\u00a0\u00a0\"cat barks loud\"\n\n]\n\n# Compute Term Frequency (TF) using CountVectorizer\n\nvectorizer = CountVectorizer()\n\nX = vectorizer.fit_transform(documents)\n\ntf_matrix = X.toarray()\n\nfeature_names = vectorizer.get_feature_names_out()\n\n# Compute Inverse Document Frequency (IDF) for BM25\n\nN = len(documents)\u00a0 # Total number of documents\n\ndf = np.sum(tf_matrix > 0, axis=0)\u00a0 # Document Frequency (DF) for each term\n\nidf = np.log((N - df + 0.5) \/ (df + 0.5) + 1)\u00a0 # BM25 IDF formula\n\n# Compute BM25 scores\n\nk1 = 1.5\u00a0 # Smoothing parameter\n\nb = 0.75\u00a0 # Length normalization parameter\n\navgdl = np.mean([len(doc.split()) for doc in documents])\u00a0 # Average document length\n\ndoc_lengths = np.array([len(doc.split()) for doc in documents])\n\nbm25_matrix = np.zeros_like(tf_matrix, dtype=np.float64)\n\nfor i in range(N):\u00a0 # For each document\n\n\u00a0\u00a0\u00a0for j in range(len(feature_names)):\u00a0 # For each term\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0term_freq = tf_matrix[i, j]\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0num = term_freq * (k1 + 1)\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0denom = term_freq + k1 * (1 - b + b * (doc_lengths[i] \/ avgdl))\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0bm25_matrix[i, j] = idf[j] * (num \/ denom)\n\n# Convert to DataFrame for better visualization\n\ndf_tf = pd.DataFrame(tf_matrix, columns=feature_names)\n\ndf_idf = pd.DataFrame([idf], columns=feature_names)\n\ndf_bm25 = pd.DataFrame(bm25_matrix, columns=feature_names)\n\n# Display the results\n\nprint(\"\\n\ud83d\udd39 Term Frequency (TF) Matrix:\\n\", df_tf)\n\nprint(\"\\n\ud83d\udd39 BM25 Inverse Document Frequency (IDF):\\n\", df_idf)\n\nprint(\"\\n\ud83d\udd39 BM25 Scores:\\n\", df_bm25)<\/code><\/pre>\n![\"BN](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/04\/13-image13.webp\")
<\/figure>\n<\/div>\nCode Implementation (Info Retrieval)<\/h3>\n
!pip install bm25s\n\nimport bm25s\n\n# Create your corpus here\n\ncorpus = [\n\n\u00a0\u00a0\u00a0\"a cat is a feline and likes to purr\",\n\n\u00a0\u00a0\u00a0\"a dog is the human's best friend and loves to play\",\n\n\u00a0\u00a0\u00a0\"a bird is a beautiful animal that can fly\",\n\n\u00a0\u00a0\u00a0\"a fish is a creature that lives in water and swims\",\n\n]\n\n# Create the BM25 model and index the corpus\n\nretriever = bm25s.BM25(corpus=corpus)\n\nretriever.index(bm25s.tokenize(corpus))\n\n# Query the corpus and get top-k results\n\nquery = \"does the fish purr like a cat?\"\n\nresults, scores = retriever.retrieve(bm25s.tokenize(query), k=2)\n\n# Let's see what we got!\n\ndoc, score = results[0, 0], scores[0, 0]\n\nprint(f\"Rank {i+1} (score: {score:.2f}): {doc}\")<\/code><\/pre>\n![\"BN25](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/04\/14-image11.webp\")
<\/figure>\n<\/div>\nBenefits<\/h3>\n
\n
Shortcomings<\/h3>\n
\n
5. Word2Vec (CBOW and Skip-gram)<\/h2>\n
How It Works<\/h3>\n
\n
\n
\n
![\"Evolution](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/04\/15-image27.webp\")
<\/li>\n<\/ul>\n<\/li>\nCode Implementation<\/h3>\n
!pip install numpy==1.24.3\n\nfrom gensim.models import Word2Vec\n\nimport networkx as nx\n\nimport matplotlib.pyplot as plt\n\n# Sample corpus\n\nsentences = [\n\n\t[\"I\", \"love\", \"deep\", \"learning\"],\n\n\t[\"Natural\", \"language\", \"processing\", \"is\", \"fun\"],\n\n\t[\"Word2Vec\", \"is\", \"a\", \"great\", \"tool\"],\n\n\t[\"AI\", \"is\", \"the\", \"future\"],\n\n]\n\n# Train Word2Vec models\n\ncbow_model = Word2Vec(sentences, vector_size=10, window=2, min_count=1, sg=0)\u00a0 # CBOW\n\nskipgram_model = Word2Vec(sentences, vector_size=10, window=2, min_count=1, sg=1)\u00a0 # Skip-gram\n\n# Get word vectors\n\nword = \"is\"\n\nprint(f\"CBOW Vector for '{word}':\\n\", cbow_model.wv[word])\n\nprint(f\"\\nSkip-gram Vector for '{word}':\\n\", skipgram_model.wv[word])\n\n# Get most similar words\n\nprint(\"\\n\ud83d\udd39 CBOW Most Similar Words:\", cbow_model.wv.most_similar(word))\n\nprint(\"\\n\ud83d\udd39 Skip-gram Most Similar Words:\", skipgram_model.wv.most_similar(word))\n<\/code><\/pre>\n![\"Word2vec](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/04\/17-image21.webp\")
<\/figure>\n<\/div>\ndef visualize_cbow():\n\n\u00a0\u00a0\u00a0G = nx.DiGraph()\n\n\u00a0\u00a0\u00a0# Nodes\n\n\u00a0\u00a0\u00a0context_words = [\"Natural\", \"is\", \"fun\"]\n\n\u00a0\u00a0\u00a0target_word = \"learning\"\n\n\u00a0\u00a0\u00a0for word in context_words:\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0G.add_edge(word, \"Hidden Layer\")\n\n\u00a0\u00a0\u00a0G.add_edge(\"Hidden Layer\", target_word)\n\n\u00a0\u00a0\u00a0# Draw the network\n\n\u00a0\u00a0\u00a0pos = nx.spring_layout(G)\n\n\u00a0\u00a0\u00a0plt.figure(figsize=(6, 4))\n\n\u00a0\u00a0\u00a0nx.draw(G, pos, with_labels=True, node_size=3000, node_color=\"lightblue\", edge_color=\"gray\")\n\n\u00a0\u00a0\u00a0plt.title(\"CBOW Model Visualization\")\n\n\u00a0\u00a0\u00a0plt.show()\n\nvisualize_cbow()<\/code><\/pre>\n![\"CBOW](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/04\/18-image18.webp\")
<\/figure>\n<\/div>\ndef visualize_skipgram():\n\n\u00a0\u00a0\u00a0G = nx.DiGraph()\n\n\u00a0\u00a0\u00a0# Nodes\n\n\u00a0\u00a0\u00a0target_word = \"learning\"\n\n\u00a0\u00a0\u00a0context_words = [\"Natural\", \"is\", \"fun\"]\n\n\u00a0\u00a0\u00a0G.add_edge(target_word, \"Hidden Layer\")\n\n\u00a0\u00a0\u00a0for word in context_words:\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0G.add_edge(\"Hidden Layer\", word)\n\n\u00a0\u00a0\u00a0# Draw the network\n\n\u00a0\u00a0\u00a0pos = nx.spring_layout(G)\n\n\u00a0\u00a0\u00a0plt.figure(figsize=(6, 4))\n\n\u00a0\u00a0\u00a0nx.draw(G, pos, with_labels=True, node_size=3000, node_color=\"lightgreen\", edge_color=\"gray\")\n\n\u00a0\u00a0\u00a0plt.title(\"Skip-gram Model Visualization\")\n\n\u00a0\u00a0\u00a0plt.show()\n\nvisualize_skipgram()<\/code><\/pre>\n![\"Skip-gram](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/04\/19-image5.webp\")
<\/figure>\n<\/div>\nBenefits<\/h3>\n
\n
Shortcomings<\/h3>\n
\n
6. GloVe (Global Vectors for Word Representation)<\/h2>\n
How It Works<\/h3>\n
\n
\n
![\"\"\/](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/04\/20-image28.webp\")
\n\n
<\/strong>Unlike Word2Vec\u2019s purely predictive model, GloVe\u2019s approach allows the model to learn the ratios of word co-occurrences, which some studies have found to be more robust in capturing semantic similarities and analogies.<\/li>\n<\/ul>\nCode Implementation<\/h3>\n
import numpy as np\n\n# Load pre-trained GloVe embeddings\n\nglove_model = api.load(\"glove-wiki-gigaword-50\")\u00a0 # You can use \"glove-twitter-25\", \"glove-wiki-gigaword-100\", etc.\n\n# Example words\n\nword = \"king\"\n\nprint(f\"\ud83d\udd39 Vector representation for '{word}':\\n\", glove_model[word])\n\n# Find similar words\n\nsimilar_words = glove_model.most_similar(word, topn=5)\n\nprint(\"\\n\ud83d\udd39 Words similar to 'king':\", similar_words)\n\nword1 = \"king\"\n\nword2 = \"queen\"\n\nsimilarity = glove_model.similarity(word1, word2)\n\nprint(f\"\ud83d\udd39 Similarity between '{word1}' and '{word2}': {similarity:.4f}\")<\/code><\/pre>\n![\"GloVe](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/04\/21-image36.webp\")
<\/figure>\n<\/div>\n![\"GloVe](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/04\/22-image23.webp\")
<\/figure>\n<\/div>\n![\"GloVe](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/04\/23-image8.webp\")
<\/figure>\n<\/div>\nBenefits<\/h3>\n