{"id":177415,"date":"2025-04-08T14:28:14","date_gmt":"2025-04-08T14:28:14","guid":{"rendered":"https:\/\/peraltafinancing.com\/analytics\/bertscore-new-metrics-for-language-models\/"},"modified":"2025-04-08T14:28:14","modified_gmt":"2025-04-08T14:28:14","slug":"bertscore-new-metrics-for-language-models","status":"publish","type":"post","link":"https:\/\/fivemor.com\/?p=177415","title":{"rendered":"BERTScore: New Metrics for Language Models"},"content":{"rendered":"


\n<\/p>\n

\n

We all depend on LLMs for our everyday activities, but quantifying \u201cHow efficient they are<\/em>\u201d is a gigantic challenge. Conventional metrics such as BLEU<\/a>, ROUGE<\/a>, and METEOR<\/a> tend to fail in comprehending the real meaning of the text. They are too keen on matching similar words instead of comprehending the concept behind it. BERTScore reverses this by applying BERT embeddings to assess the quality of the text with better comprehension of meaning and context.<\/p>\n

Whether you\u2019re training a chatbot, translating, or making summaries, BERTScore makes it easier for you to evaluate your models better. It captures when two sentences convey the same thing despite using different words\u2014something older metrics completely miss. As we dive into how BERTScore operates, you\u2019ll learn how this brilliant evaluation approach ties together computer measurement and human intuition and revolutionizes the way we test and refine today\u2019s sophisticated language models.<\/p>\n

What is BERTScore?<\/h2>\n

BERTScore is a neural evaluation metric for text generation that uses contextual embeddings from pre-trained language models like BERT to calculate similarity scores between candidate and reference texts. Unlike traditional n-gram-based metrics, BERTScore can identify semantic equivalence even when different words are used, making it useful for evaluating language tasks where multiple valid outputs exist.<\/p>\n

\"\"<\/figure>\n

Formulated by Zhang et al. and presented in their 2019 paper \u201cBERTScore: Evaluating Text Generation with BERT<\/a>,\u201d this score has gained rapid acceptance within the NLP community due to its high correlation with human evaluation across a range of text generation tasks.<\/p>\n

BERTScore Architecture<\/h2>\n

BERTScore\u2019s architecture is elegantly simple yet powerful, consisting of three main components:<\/p>\n

    \n
  1. Embedding Generation<\/strong>: Each token in both reference and candidate texts is embedded using a pre-trained contextual embedding model (typically BERT).<\/li>\n
  2. Token Matching<\/strong>: The algorithm computes pairwise cosine similarities between all tokens in the reference and candidate texts, creating a similarity matrix.<\/li>\n
  3. Score Aggregation<\/strong>: These similarity scores are aggregated into precision, recall, and F1 measures that represent how well the candidate text matches the reference.<\/li>\n<\/ol>\n
    \"\"<\/figure>\n

    The beauty of BERTScore is that it leverages the contextual understanding of pre-trained models without requiring additional training for the evaluation task.<\/p>\n

    How to Use BERTScore?\u00a0<\/h2>\n

    BERTScore can be customized using several parameters to suit specific evaluation needs:<\/p>\n

    \n\n\n\n\n\n\n\n\n\n\n\n
    Parameter<\/b><\/td>\nDescription<\/b><\/td>\nDefault<\/b><\/td>\n<\/tr>\n
    model_type<\/td>\nPre-trained model to use (e.g., \u2018bert-base-uncased\u2019)<\/td>\n\u2018roberta-large\u2019<\/td>\n<\/tr>\n
    num_layers<\/td>\nWhich layer\u2019s embeddings to use<\/td>\n17 (for roberta-large)<\/td>\n<\/tr>\n
    idf<\/td>\nWhether to use IDF weighting for token importance<\/td>\nFalse<\/td>\n<\/tr>\n
    rescale_with_baseline<\/td>\nWhether to rescale scores based on a baseline<\/td>\nFalse<\/td>\n<\/tr>\n
    baseline_path<\/td>\nPath to baseline scores<\/td>\nNone<\/td>\n<\/tr>\n
    lang<\/td>\nLanguage of the texts being compared<\/td>\n\u2018en\u2019<\/td>\n<\/tr>\n
    use_fast_tokenizer<\/td>\nWhether to use HuggingFace\u2019s fast tokenizers<\/td>\nFalse<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

    These parameters allow researchers to fine-tune BERTScore for different languages, domains, and evaluation requirements.<\/p>\n

    How Does BERTScore Work?<\/h2>\n

    BERTScore evaluates the similarity between generated text and reference text through a token-level matching process using contextual embeddings. Here is a step-by-step breakdown of how it operates:<\/p>\n

    \"\"
    Source: BERTScore<\/a><\/figcaption><\/figure>\n
      \n
    1. Tokenization<\/strong>: Both candidate (generated) and reference texts are tokenized using the tokenizer corresponding to the pre-trained model being used (e.g., BERT, RoBERTa).<\/li>\n
    2. Contextual Embedding<\/strong>: Each token is then embedded using a pre-trained contextual model. Importantly, these embeddings capture the meaning of words in context rather than static word representations. For example, the word \u201cbank\u201d would have different embeddings in \u201criver bank\u201d versus \u201cfinancial bank.\u201d<\/li>\n
    3. Cosine Similarity Computation<\/strong>: For each token in the candidate text, BERTScore computes its cosine similarity with every token in the reference text, creating a similarity matrix.<\/li>\n
    4. Greedy Matching<\/strong>:\n