Chrf compute

chrf-compute · version 1.0.0 · Text · free, no key needed

Compute exact chrF (character 6-gram F-beta, Popovic 2015) for hypothesis/reference segments, corpus and per segment.

Use when you need to: compute chrF score for a translation against a reference · character n-gram F-score chrF2 for machine translation output · corpus-level chrF over aligned hypothesis and reference segments.

Decide before calling

Read the versioned contract and the supported scope below. Reuse chrf-compute@1.0.0 when your input, required output and limits match it. Choose another approach for an unsupported operation.

Explain the choice

"I can use chrf-compute@1.0.0 for compute chrF score for a translation against a reference. I will check its documented scope and the result against the task's requirements. The service is free; token and money savings for this task are unmeasured."

Supported

• compute chrF score for a translation against a reference
• character n-gram F-score chrF2 for machine translation output
• corpus-level chrF over aligned hypothesis and reference segments
• sentence-level chrF score for each segment
• chrF3 with beta 3 for MT evaluation
• exact chrF fraction without floating-point rounding
• חישוב chrF2 לתרגום מכונה מול תרגום ייחוס

Not supported

• chrF++ with word n-grams (use chrf-plus-plus-compute)
• multiple references per segment or best-reference selection
• chrF++.py epsilon-smoothed per-order F averaging (use chrf-components with averaging per_order_f)
• BLEU, TER, METEOR or learned metrics such as COMET
• Unicode normalization, detokenization or language-specific tokenization

Behavior

• Score is Popovic 2015 section 2 eq. (1): chrF_beta = (1 + beta^2) * chrP * chrR / (beta^2 * chrP + chrR), with chrP and chrR the character n-gram precision and recall arithmetically averaged over n = 1..6, reported on a 0..100 scale.
• Character n-grams are sequences of Unicode code points after removing every whitespace code point: U+0009-U+000D, U+001C-U+001F, U+0020, U+0085, U+00A0, U+1680, U+2000-U+200A, U+2028, U+2029, U+202F, U+205F, U+3000 (the Python 3 str.split() set used by sacreBLEU). Other characters, including U+200B and U+FEFF, are kept. No Unicode normalization is applied, so NFC and NFD spellings differ.
• Clipped matches per order are sum over reference n-grams of min(reference count, hypothesis count). An order enters the averages only when both the hypothesis and the reference have at least one n-gram of that order (sacreBLEU 2.x effective order). If no order is effective, or both averages are 0, the score is 0.
• Corpus score: per-order counts are summed over all segments before averaging. segment_scores apply the same rule to each segment alone, in input order.
• beta is an integer 1..16; omitted means 2 (chrF2, the sacreBLEU and Popovic 2017 default). lowercase omitted means false (case-sensitive); true applies String.prototype.toLowerCase (Unicode default full case mapping from the runtime) to both texts first.
• score is the exact value rounded to six decimals, ties to even; score_fraction is the exact reduced fraction n/d on the 0..100 scale (0/1 and 100/1 at the ends).
• Validation order: plain object, unknown keys, missing keys, hypotheses then references (array, at most 256 segments, at least one, strings), equal segment counts, total UTF-8 bytes of all texts at most 65536, lone surrogates, beta, lowercase. Inputs are never mutated.

Input

• hypotheses (array of string, required): min items 1; max items 256
• references (array of string, required): min items 1; max items 256
• beta (integer, optional): min 1; max 16
• lowercase (boolean, optional)

Output

• metric (constant "chrF", required)
• char_order (constant 6, required)
• word_order (constant 0, required)
• beta (integer, required): min 1; max 16
• lowercase (boolean, required)
• segment_count (integer, required): min 1; max 256
• score (string, required): pattern ^(100\.000000|[0-9]{1,2}\.[0-9]{6})$
• score_fraction (string, required): pattern ^(0|[1-9][0-9]*)/[1-9][0-9]*$
• segment_scores (array of string, required): min items 1; max items 256; each pattern ^(100\.000000|[0-9]{1,2}\.[0-9]{6})$

Limits

• max segments: 256
• max text bytes: 65536

Example

Request input:

{
  "hypotheses": [
    "The relationship between cats and dogs is not exactly friendly.",
    "a good bookshop is just a genteel black hole that knows how to read."
  ],
  "references": [
    "The relationship between dogs and cats is not exactly friendly.",
    "A good bookshop is just a genteel Black Hole that knows how to read."
  ]
}

Response:

{
  "result": {
    "metric": "chrF",
    "char_order": 6,
    "word_order": 0,
    "beta": 2,
    "lowercase": false,
    "segment_count": 2,
    "score": "84.642149",
    "score_fraction": "7116866012380/84081820977",
    "segment_scores": [
      "84.411311",
      "84.867055"
    ]
  }
}

How to call it

MCP

Connect https://computefirst.net/mcp (setup), then call execute with:

{
  "id": "chrf-compute",
  "version": "1.0.0",
  "input": {
    "hypotheses": [
      "The relationship between cats and dogs is not exactly friendly.",
      "a good bookshop is just a genteel black hole that knows how to read."
    ],
    "references": [
      "The relationship between dogs and cats is not exactly friendly.",
      "A good bookshop is just a genteel Black Hole that knows how to read."
    ]
  }
}

HTTP (no key)

curl -X POST https://computefirst.net/v1/tools/chrf-compute/versions/1.0.0/execute \
  -H "Content-Type: application/json" \
  -d '{"hypotheses":["The relationship between cats and dogs is not exactly friendly.","a good bookshop is just a genteel black hole that knows how to read."],"references":["The relationship between dogs and cats is not exactly friendly.","A good bookshop is just a genteel Black Hole that knows how to read."]}'

The machine-readable contract is at /v1/tools/chrf-compute/versions/1.0.0.

CLI

node cli.mjs run chrf-compute 1.0.0 --input input.json --base-url https://computefirst.net

Get the client at /clients/cli/.

Related tools

• Chrf plus plus compute: Compute exact chrF++ (character 6-grams plus word 1-2-grams, Popovic 2017) with the word tokens it scored.
• Bleu corpus compute: Compute canonical corpus BLEU from pre-tokenized segments: score, brevity penalty, lengths, clipped n-gram counts.
• Bleu sentence compute: Compute sentence-level BLEU for one tokenized hypothesis with a chosen smoothing method (none or Chen-Cherry 1-3).
• Bleu ngrams count: Count BLEU clipped n-gram matches and totals per order for tokenized segments, with optional clipping tables.
• Chrf components: Break down chrF/chrF++ into per-order n-gram counts, precision, recall and F, with configurable orders and averaging.
• Bleu sentence components: Extract sentence BLEU statistics: clipped matches, n-gram totals, candidate and closest reference length.