{"id":169396,"date":"2025-04-03T16:23:32","date_gmt":"2025-04-03T16:23:32","guid":{"rendered":"https:\/\/peraltafinancing.com\/analytics\/how-to-evaluate-llms-using-hugging-face-evaluate\/"},"modified":"2025-04-03T16:23:32","modified_gmt":"2025-04-03T16:23:32","slug":"how-to-evaluate-llms-using-hugging-face-evaluate","status":"publish","type":"post","link":"https:\/\/fivemor.com\/?p=169396","title":{"rendered":"How to Evaluate LLMs Using Hugging Face Evaluate"},"content":{"rendered":"


\n<\/p>\n

\n

Evaluating large language models (LLMs)<\/a> is essential. You need to understand how well they perform and ensure they meet your standards. The Hugging Face Evaluate library offers a helpful set of tools for this task. This guide shows you how to use the Evaluate library to assess LLMs with practical code examples.<\/p>\n

Understanding the Hugging Face Evaluate Library<\/h2>\n

The Hugging Face Evaluate library provides tools for different evaluation needs. These tools fall into three main categories:<\/p>\n

    \n
  1. Metrics<\/strong>: These measure a model\u2019s performance by comparing its predictions to ground truth labels. Examples include accuracy, F1-score, BLEU, and ROUGE.<\/li>\n
  2. Comparisons<\/strong>: These help compare two models, often by examining how their predictions align with each other or with reference labels.<\/li>\n
  3. Measurements<\/strong>: These tools investigate the properties of datasets themselves, like calculating text complexity or label distributions.<\/li>\n<\/ol>\n

    You can access all these evaluation modules using a single function: evaluate.load().<\/p>\n

    Getting Started<\/h2>\n

    Installation<\/h3>\n

    First, you need to install the library. Open your terminal or command prompt and run:<\/p>\n

    pip install evaluate\n\npip install rouge_score # Needed for text generation metrics\n\npip install evaluate[visualization] # For plotting capabilities<\/code><\/pre>\n
    These commands install the core evaluate library, the rouge_score package (required for the ROUGE<\/a> metric often used in summarization), and optional dependencies for visualization like radar plots.<\/p>\n
    Loading an Evaluation Module<\/h3>\n
    To use a specific evaluation tool, you load it by name. For instance, to load the accuracy metric:<\/p>\n
    import evaluate\n\naccuracy_metric = evaluate.load(\"accuracy\")\n\nprint(\"Accuracy metric loaded.\")<\/code><\/pre>\n
    Output:<\/strong><\/p>\n
    \"\"<\/figure>\n
    This code imports the evaluate library and loads the accuracy metric object. You will use this object to compute accuracy scores.<\/p>\n
    Basic Evaluation Examples<\/h2>\n
    Let\u2019s walk through some common evaluation scenarios.<\/p>\n
    Computing Accuracy Directly<\/h3>\n
    You can compute a metric by providing all references (ground truth) and predictions at once.<\/p>\n
    import evaluate\n\n# Load the accuracy metric\n\naccuracy_metric = evaluate.load(\"accuracy\")\n\n# Sample ground truth and predictions\n\nreferences = [0, 1, 0, 1]\n\npredictions = [1, 0, 0, 1]\n\n# Compute accuracy\n\nresult = accuracy_metric.compute(references=references, predictions=predictions)\n\nprint(f\"Direct computation result: {result}\")\n\n# Example with exact_match metric\n\nexact_match_metric = evaluate.load('exact_match')\n\nmatch_result = exact_match_metric.compute(references=['hello world'], predictions=['hello world'])\n\nno_match_result = exact_match_metric.compute(references=['hello'], predictions=['hell'])\n\nprint(f\"Exact match result (match): {match_result}\")\n\nprint(f\"Exact match result (no match): {no_match_result}\")<\/code><\/pre>\n
    Output:<\/strong><\/p>\n
    \"\"<\/figure>\n
    Explanation:<\/strong><\/p>\n
      \n
    1. We define two lists: references holds the correct labels, and predictions holds the model\u2019s outputs.<\/li>\n
    2. The compute method takes these lists and calculates the accuracy, returning the result as a dictionary.<\/li>\n
    3. We also show the exact_match metric, which checks if the prediction perfectly matches the reference.<\/li>\n<\/ol>\n
      Incremental Evaluation (Using add_batch)<\/h3>\n
      For large datasets, processing predictions in batches can be more memory-efficient. You can add batches incrementally and compute the final score at the end.<\/p>\n
      import evaluate\n\n# Load the accuracy metric\n\naccuracy_metric = evaluate.load(\"accuracy\")\n\n# Sample batches of refrences and predictions\n\nreferences_batch1 = [0, 1]\n\npredictions_batch1 = [1, 0]\n\nreferences_batch2 = [0, 1]\n\npredictions_batch2 = [0, 1]\n\n# Add batches incrementally\n\naccuracy_metric.add_batch(references=references_batch1, predictions=predictions_batch1)\n\naccuracy_metric.add_batch(references=references_batch2, predictions=predictions_batch2)\n\n# Compute final accuracy\n\nfinal_result = accuracy_metric.compute()\n\nprint(f\"Incremental computation result: {final_result}\")<\/code><\/pre>\n
      Output:<\/strong><\/p>\n
      \"\"<\/figure>\n
      Explanation:<\/strong><\/p>\n
        \n
      1. We simulate processing data in two batches.<\/li>\n
      2. add_batch updates the metric\u2019s internal state with each batch.<\/li>\n
      3. Calling compute() without arguments calculates the metric over all added batches.<\/li>\n<\/ol>\n
        Combining Multiple Metrics<\/h3>\n
        You often want to calculate several metrics simultaneously (e.g., accuracy, F1, precision, recall for classification). The evaluate.combine<\/em> function simplifies this.<\/p>\n
        import evaluate\n\n# Combine multiple classification metrics\n\nclf_metrics = evaluate.combine([\"accuracy\", \"f1\", \"precision\", \"recall\"])\n\n# Sample data\n\npredictions = [0, 1, 0]\n\nreferences = [0, 1, 1] # Note: The last prediction is incorrect\n\n# Compute all metrics at once\n\nresults = clf_metrics.compute(predictions=predictions, references=references)\n\nprint(f\"Combined metrics result: {results}\")<\/code><\/pre>\n
        Output:<\/strong><\/p>\n
        \"\"<\/figure>\n
        Explanation:<\/strong><\/p>\n
          \n
        1. evaluate.combine<\/em> takes a list of metric names and returns a combined evaluation object.<\/li>\n
        2. Calling compute on this object calculates all the specified metrics using the same input data.<\/li>\n<\/ol>\n
          Using Measurements<\/h3>\n
          Measurements can be used to analyze datasets. Here\u2019s how to use the word_length measurement:<\/p>\n
          import evaluate\n\n# Load the word_length measurement\n\n# Note: May require NLTK data download on first run\n\ntry:\n\n\u00a0\u00a0\u00a0word_length = evaluate.load(\"word_length\", module_type=\"measurement\")\n\n\u00a0\u00a0\u00a0data = [\"hello world\", \"this is another sentence\"]\n\n\u00a0\u00a0\u00a0results = word_length.compute(data=data)\n\n\u00a0\u00a0\u00a0print(f\"Word length measurement result: {results}\")\n\nexcept Exception as e:\n\n\u00a0\u00a0\u00a0print(f\"Could not run word_length measurement, possibly NLTK data missing: {e}\")\n\n\u00a0\u00a0\u00a0print(\"Attempting NLTK download...\")\n\n\u00a0\u00a0\u00a0import nltk\n\n\u00a0\u00a0\u00a0nltk.download('punkt') # Uncomment and run if needed<\/code><\/pre>\n
          Output:<\/strong><\/p>\n
          \"\"<\/figure>\n
          Explanation:<\/strong><\/p>\n
            \n
          1. We load word_length and specify module_type=\u201dmeasurement\u201d.<\/li>\n
          2. The compute method takes the dataset (a list of strings here) as input.<\/li>\n
          3. It returns statistics about the word lengths in the provided data. (Note: Requires nltk and its \u2018punkt\u2019 tokenizer data).<\/li>\n<\/ol>\n
            Evaluating Specific NLP Tasks<\/h2>\n
            Different NLP tasks require specific metrics. Hugging Face Evaluate includes many standard ones.<\/p>\n
            Machine Translation (BLEU)<\/h3>\n
            BLEU (Bilingual Evaluation Understudy) is common for translation quality. It measures n-gram overlap between the model\u2019s translation (hypothesis) and reference translations.<\/p>\n
            import evaluate\n\ndef evaluate_machine_translation(hypotheses, references):\n\n\u00a0\u00a0\u00a0\"\"\"Calculates BLEU score for machine translation.\"\"\"\n\n\u00a0\u00a0\u00a0bleu_metric = evaluate.load(\"bleu\")\n\n\u00a0\u00a0\u00a0results = bleu_metric.compute(predictions=hypotheses, references=references)\n\n\u00a0\u00a0\u00a0# Extract the main BLEU score\n\n\u00a0\u00a0\u00a0bleu_score = results[\"bleu\"]\n\n\u00a0\u00a0\u00a0return bleu_score\n\n# Example hypotheses (model translations)\n\nhypotheses = [\"the cat sat on mat.\", \"the dog played in garden.\"]\n\n# Example references (correct translations, can have multiple per hypothesis)\n\nreferences = [[\"the cat sat on the mat.\"], [\"the dog played in the garden.\"]]\n\nbleu_score = evaluate_machine_translation(hypotheses, references)\n\nprint(f\"BLEU Score: {bleu_score:.4f}\") # Format for readability<\/code><\/pre>\n
            Output:<\/strong><\/p>\n
            \"\"<\/figure>\n
            Explanation:<\/strong><\/p>\n
              \n
            1. The function loads the BLEU metric.<\/li>\n
            2. It computes the score comparing predicted translations (hypotheses) against one or more correct references.<\/li>\n
            3. A higher BLEU score (closer to 1.0) generally indicates better translation quality, suggesting more overlap with reference translations. A score around 0.51 suggests moderate overlap.<\/li>\n<\/ol>\n
              Named Entity Recognition (NER \u2013 using seqeval)<\/h3>\n
              For sequence labeling tasks like NER, metrics like precision, recall, and F1-score per entity type are useful. The seqeval metric handles this format (e.g., B-PER, I-PER, O tags).<\/p>\n
              To run the following code, seqeval library would be required. It could be installed by running the following command:<\/p>\n
              pip install seqeval<\/code><\/pre>\n
              Code:<\/strong><\/p>\n
              import evaluate\n\n# Load the seqeval metric\ntry:\n\n\u00a0\u00a0\u00a0seqeval_metric = evaluate.load(\"seqeval\")\n\n\u00a0\u00a0\u00a0# Example labels (using IOB format)\n\u00a0\u00a0\u00a0true_labels = [['O', 'B-PER', 'I-PER', 'O'], ['B-LOC', 'I-LOC', 'O']]\n\n\u00a0\u00a0\u00a0predicted_labels = [['O', 'B-PER', 'I-PER', 'O'], ['B-LOC', 'I-LOC', 'O']] # Example: Perfect prediction here\n\n\u00a0\u00a0\u00a0results = seqeval_metric.compute(predictions=predicted_labels, references=true_labels)\n\n\u00a0\u00a0\u00a0print(\"Seqeval Results (per entity type):\")\n\n\u00a0\u00a0\u00a0# Print results nicely\n\n\u00a0\u00a0\u00a0for key, value in results.items():\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0if isinstance(value, dict):\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0print(f\"\u00a0 {key}: Precision={value['precision']:.2f}, Recall={value['recall']:.2f}, F1={value['f1']:.2f}, Number={value['number']}\")\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0else:\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0print(f\"\u00a0 {key}: {value:.4f}\")\n\nexcept ModuleNotFoundError:\n\n\u00a0\u00a0\u00a0print(\"Seqeval metric not installed. Run: pip install seqeval\")<\/code><\/pre>\n
              Output:<\/strong><\/p>\n
              \"\"<\/figure>\n
              Explanation:<\/strong><\/p>\n