Instructions

Overview

This skill evaluates model checkpoints across multiple benchmark categories using predefined scoring algorithms, identifies the best-performing checkpoint based on overall evaluation accuracy, and prepares it for deployment (e.g., uploading to Hugging Face Hub with updated documentation).

Workflow

1. Scan Workspace and Identify Checkpoints

• List directories in the workspace to locate checkpoint folders.
• Checkpoint folders are typically named with step numbers (e.g., step_100, step_200).
• Extract step numbers from folder names for evaluation.

2. Run Benchmark Evaluations

• Use the bundled benchmark_calculator.py script to compute scores for each checkpoint.
• The script implements 15 benchmark scoring functions derived from the original Cython module:
  • Core Reasoning Tasks: Math Reasoning, Logical Reasoning, Common Sense
  • Language Understanding: Reading Comprehension, Question Answering, Text Classification, Sentiment Analysis
  • Generation Tasks: Code Generation, Creative Writing, Dialogue Generation, Summarization
  • Specialized Capabilities: Translation, Knowledge Retrieval, Instruction Following, Safety Evaluation
• Each scoring function uses mathematical formulas (sigmoid, rational, exponential) based on step value.