{"id":147556,"date":"2025-03-21T11:06:04","date_gmt":"2025-03-21T11:06:04","guid":{"rendered":"https:\/\/peraltafinancing.com\/analytics\/evaluating-language-models-with-bleu-metric\/"},"modified":"2025-03-21T11:06:04","modified_gmt":"2025-03-21T11:06:04","slug":"evaluating-language-models-with-bleu-metric","status":"publish","type":"post","link":"https:\/\/fivemor.com\/?p=147556","title":{"rendered":"Evaluating Language Models with BLEU Metric"},"content":{"rendered":"


\n<\/p>\n

\n

In artificial intelligence, evaluating the performance of language models presents a unique challenge. Unlike image recognition or numerical predictions, language quality assessment doesn\u2019t yield to simple binary measurements. Enter BLEU (Bilingual Evaluation Understudy), a metric that has become the cornerstone of machine translation evaluation since its introduction by IBM researchers in 2002.<\/p>\n

BLEU stands for a breakthrough in natural language processing for it is the very first evaluation method that manages to achieve a pretty high correlation with human judgment and yet retains the efficiency of automation. This article investigates the mechanics of BLEU, its applications, its limitations, and what the future holds for it in an increasingly AI-driven world that is preoccupied with richer nuances in language-generated output.<\/p>\n

Note: This is a series of Evaluation Metrics of LLMs and I will be covering all the Top 15 LLM Evaluation Metrics to Explore in 2025<\/a>.<\/em><\/p>\n

The Genesis of BLEU Metric: A Historical Perspective<\/h2>\n

Prior to BLEU, evaluating machine translations was primarily manual\u2014a resource-intensive process requiring lingual experts to manually assess each output. The introduction of BLEU by Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu at IBM Research represented a paradigm shift. Their 2002 paper, \u201cBLEU: a Method for Automatic Evaluation of Machine Translation,\u201d proposed an automated metric that could score translations with remarkable alignment to human judgment.<\/p>\n

The timing was pivotal. As statistical machine translation systems were gaining momentum, the field urgently needed standardized evaluation methods. BLEU filled this void, offering a reproducible, language-independent scoring mechanism that facilitated meaningful comparisons between different translation systems.<\/p>\n

How Does BLEU Metric Work?<\/h2>\n

At its core, BLEU operates on a simple principle: comparing machine-generated translations against reference translations (typically created by human translators). It has been observed that the BLEU score decreases as the sentence length increases, though it might vary depending on the model used for translations. However, its implementation involves sophisticated computational linguistics concepts:<\/p>\n

\"Source:
Source: Author<\/figcaption><\/figure>\n

N-gram Precision<\/strong><\/h3>\n

BLEU\u2019s foundation lies in n-gram precision\u2014the percentage of word sequences in the machine translation that appear in any reference translation. Rather than limiting itself to individual words (unigrams), BLEU examines contiguous sequences of various lengths:<\/p>\n

    \n
  • Unigrams (single words) Modified Precision: Measuring vocabulary accuracy<\/li>\n
  • Bigrams (two-word sequences) Modified Precision: Capturing basic phrasal correctness<\/li>\n
  • Trigrams and 4-grams Modified Precision: Evaluating grammatical structure and word order<\/li>\n<\/ul>\n

    BLEU calculates modified precision for each n-gram length by:<\/p>\n

      \n
    1. Counting n-gram matches between the candidate and reference translations<\/li>\n
    2. Applying a \u201cclipping\u201d mechanism to prevent overinflation from repeated words<\/li>\n
    3. Dividing by the total number of n-grams in the candidate translation<\/li>\n<\/ol>\n

      Brevity Penalty<\/strong><\/h3>\n

      To prevent systems from gaming the metric by producing extremely short translations (which could achieve high precision by including only easily matched words), BLEU incorporates a brevity penalty that reduces scores for translations shorter than their references.<\/p>\n

      The penalty is calculated as:<\/p>\n

      BP = exp(1 - r\/c) if c <\/code><\/pre>\n
      Where r is the reference length and c is the candidate translation length.<\/p>\n
      The Final BLEU Score<\/strong><\/h3>\n
      The final BLEU score combines these components into a single value between 0 and 1 (often presented as a percentage):<\/p>\n
      BLEU = BP \u00d7 exp(\u2211 wn log pn)<\/code><\/pre>\n
      Where:<\/p>\n
        \n
      • BP is the brevity penalty<\/li>\n
      • wn represents weights for each n-gram precision (typically uniform)<\/li>\n
      • pn is the modified precision for n-grams of length n<\/li>\n<\/ul>\n
        Implementing BLEU Metric<\/h2>\n
        Understanding BLEU conceptually is one thing; implementing it correctly requires attention to detail. Here\u2019s a practical guide to using BLEU effectively:<\/p>\n
        Required Inputs<\/h3>\n
        BLEU requires two primary inputs:<\/p>\n
          \n
        1. Candidate translations<\/strong>: The machine-generated translations you want to evaluate<\/li>\n
        2. Reference translations<\/strong>: One or more human-created translations for each source sentence<\/li>\n<\/ol>\n
          Both inputs must undergo consistent preprocessing:<\/p>\n
            \n
          • Tokenization<\/strong>: Breaking text into words or subwords<\/li>\n
          • Case normalization<\/strong>: Typically lowercasing all text<\/li>\n
          • Punctuation handling<\/strong>: Either removing punctuation or treating punctuation marks as separate tokens<\/li>\n<\/ul>\n
            Implementation Steps<\/h3>\n
            A typical BLEU implementation follows these steps:<\/p>\n
              \n
            1. Preprocess all translations<\/strong>: Apply consistent tokenization and normalization<\/li>\n
            2. Calculate n-gram precision<\/strong> for n=1 to N (typically N=4):\n
                \n
              • Count all n-grams in the candidate translation<\/li>\n
              • Count matching n-grams in reference translations (with clipping)<\/li>\n
              • Compute precision as (matches \/ total candidate n-grams)<\/li>\n<\/ul>\n<\/li>\n
              • Calculate brevity penalty<\/strong>:\n
                  \n
                • Determine effective reference length (shortest ref length in original BLEU)<\/li>\n
                • Compared to the candidate length<\/li>\n
                • Apply brevity penalty formula<\/li>\n<\/ul>\n<\/li>\n
                • Combine components<\/strong> into the final score:\n
                    \n
                  • Apply weighted geometric mean of n-gram precisions<\/li>\n
                  • Multiply by brevity penalty<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n
                    Several libraries provide ready-to-use BLEU implementations:<\/p>\n
                    NLTK: Python\u2019s Natural Language Toolkit offers a simple BLEU implementation<\/h3>\n
                    from nltk.translate.bleu_score import sentence_bleu, corpus_bleu\n\nfrom nltk.translate.bleu_score import SmoothingFunction\n\n# Create a smoothing function to avoid zero scores due to missing n-grams\n\nsmoothie = SmoothingFunction().method1\n\n# Example 1: Single reference, good match\n\nreference = [['this', 'is', 'a', 'test']]\n\ncandidate = ['this', 'is', 'a', 'test']\n\nscore = sentence_bleu(reference, candidate)\n\nprint(f\"Perfect match BLEU score: {score}\")\n\n# Example 2: Single reference, partial match\n\nreference = [['this', 'is', 'a', 'test']]\n\ncandidate = ['this', 'is', 'test']\n\n# Using smoothing to avoid zero scores\n\nscore = sentence_bleu(reference, candidate, smoothing_function=smoothie)\n\nprint(f\"Partial match BLEU score: {score}\")\n\n# Example 3: Multiple references (corrected format)\n\nreferences = [[['this', 'is', 'a', 'test']], [['this', 'is', 'an', 'evaluation']]]\n\ncandidates = [['this', 'is', 'an', 'assessment']]\n\n# The format for corpus_bleu is different - references need restructuring\n\ncorrect_references = [[['this', 'is', 'a', 'test'], ['this', 'is', 'an', 'evaluation']]]\n\nscore = corpus_bleu(correct_references, candidates, smoothing_function=smoothie)\n\nprint(f\"Multiple reference BLEU score: {score}\")<\/code><\/pre>\n
                    Output<\/h3>\n
                    Perfect match BLEU score: 1.0Partial match BLEU score: 0.19053627645285995Multiple reference BLEU score: 0.3976353643835253<\/pre>\n
                    SacreBLEU: A standardized BLEU implementation that addresses reproducibility concerns<\/h3>\n
                    import sacrebleu\n\n# For sentence-level BLEU with SacreBLEU\n\nreference = [\"this is a test\"]\u00a0 # List containing a single reference\n\ncandidate = \"this is a test\"\u00a0 \u00a0 # String containing the hypothesis\n\nscore = sacrebleu.sentence_bleu(candidate, reference)\n\nprint(f\"Perfect match SacreBLEU score: {score}\")\n\n# Partial match example\n\nreference = [\"this is a test\"]\n\ncandidate = \"this is test\"\n\nscore = sacrebleu.sentence_bleu(candidate, reference)\n\nprint(f\"Partial match SacreBLEU score: {score}\")\n\n# Multiple references example\n\nreferences = [\"this is a test\", \"this is a quiz\"]\u00a0 # List of multiple references\n\ncandidate = \"this is an exam\"\n\nscore = sacrebleu.sentence_bleu(candidate, references)\n\nprint(f\"Multiple references SacreBLEU score: {score}\")<\/code><\/pre>\n
                    Output<\/h3>\n
                    Perfect match SacreBLEU score: BLEU = 100.00 100.0\/100.0\/100.0\/100.0 (BP =1.000 ratio = 1.000 hyp_len = 4 ref_len = 4)
                    Partial match SacreBLEU score: BLEU = 45.14 100.0\/50.0\/50.0\/0.0 (BP = 0.717ratio = 0.750 hyp_len = 3 ref_len = 4)<\/p>
                    Multiple references SacreBLEU score: BLEU = 31.95 50.0\/33.3\/25.0\/25.0 (BP =1.000 ratio = 1.000 hyp_len = 4 ref_len = 4)<\/p><\/pre>\n
                    Hugging Face Evaluate: Modern implementation integrated with ML pipelines<\/h3>\n
                    from evaluate import load\n\nbleu = load('bleu')\n\n# Example 1: Perfect match\n\npredictions = [\"this is a test\"]\n\nreferences = [[\"this is a test\"]]\n\nresults = bleu.compute(predictions=predictions, references=references)\n\nprint(f\"Perfect match HF Evaluate BLEU score: {results}\")\n\n# Example 2: Multi-sentence evaluation\n\npredictions = [\"the cat is on the mat\", \"there is a dog in the park\"]\n\nreferences = [[\"the cat sits on the mat\"], [\"a dog is running in the park\"]]\n\nresults = bleu.compute(predictions=predictions, references=references)\n\nprint(f\"Multi-sentence HF Evaluate BLEU score: {results}\")\n\n# Example 3: More complex real-world translations\n\npredictions = [\"The agreement on the European Economic Area was signed in August 1992.\"]\n\nreferences = [[\"The agreement on the European Economic Area was signed in August 1992.\", \"An agreement on the European Economic Area was signed in August of 1992.\"]]\n\nresults = bleu.compute(predictions=predictions, references=references)\n\nprint(f\"Complex example HF Evaluate BLEU score: {results}\")<\/code><\/pre>\n
                    Output<\/h3>\n
                    Perfect match HF Evaluate BLEU score: {'bleu': 1.0, 'precisions': [1.0, 1.0,1.0, 1.0], 'brevity_penalty': 1.0, 'length_ratio': 1.0,'translation_length': 4, 'reference_length': 4}
                    Multi-sentence HF Evaluate BLEU score: {'bleu': 0.0, 'precisions':[0.8461538461538461, 0.5454545454545454, 0.2222222222222222, 0.0],'brevity_penalty': 1.0, 'length_ratio': 1.0, 'translation_length': 13,'reference_length': 13}<\/p>
                    Complex example HF Evaluate BLEU score: {'bleu': 1.0, 'precisions': [1.0,1.0, 1.0, 1.0], 'brevity_penalty': 1.0, 'length_ratio': 1.0,'translation_length': 13, 'reference_length': 13}<\/p><\/pre>\n
                    Interpreting BLEU Outputs<\/h2>\n
                    BLEU scores typically range from 0 to 1 (or 0 to 100 when presented as percentages):<\/p>\n
                      \n
                    • 0<\/strong>: No matches between candidate and references<\/li>\n
                    • 1 (or 100%)<\/strong>: Perfect match with references<\/li>\n
                    • Typical ranges<\/strong>:\n
                        \n
                      • 0-15: Poor translation<\/li>\n
                      • 15-30: Understandable but flawed translation<\/li>\n
                      • 30-40: Good translation<\/li>\n
                      • 40-50: High-quality translation<\/li>\n
                      • 50+: Exceptional translation (potentially approaching human quality)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
                        However, these ranges vary significantly between language pairs. For instance, translations between English and Chinese typically score lower than English-French pairs, due to linguistic differences rather than actual quality differences.<\/p>\n
                        Score Variants<\/h3>\n
                        Different BLEU implementations may produce varying scores due to:<\/p>\n
                          \n
                        • Smoothing methods<\/strong>: Addressing zero precision values<\/li>\n
                        • Tokenization differences<\/strong>: Especially important for languages without clear word boundaries<\/li>\n
                        • N-gram weighting schemes<\/strong>: Standard BLEU uses uniform weights, but alternatives exist<\/li>\n<\/ul>\n
                          For more information watch this video:<\/p>\n