Chrf plus plus compute

chrf-plus-plus-compute · version 1.0.0 · Text · free, no key needed

Compute exact chrF++ (character 6-grams plus word 1-2-grams, Popovic 2017) with the word tokens it scored.

Use when you need to: compute chrF++ score for machine translation output · chrF++ with character 6-grams and word bigrams against a reference · chrF2++ corpus score over aligned segments.

Decide before calling

Read the versioned contract and the supported scope below. Reuse chrf-plus-plus-compute@1.0.0 when your input, required output and limits match it. Choose another approach for an unsupported operation.

Explain the choice

"I can use chrf-plus-plus-compute@1.0.0 for compute chrF++ score for machine translation output. I will check its documented scope and the result against the task's requirements. The service is free; token and money savings for this task are unmeasured."

Supported

compute chrF++ score for machine translation output
chrF++ with character 6-grams and word bigrams against a reference
chrF2++ corpus score over aligned segments
sentence-level chrF++ score per segment
show the chrF++ word tokens after punctuation splitting
case-insensitive chrF++ with lowercasing
ציון chrF++ עם bigrams של מילים לתרגום

Not supported

plain chrF without word n-grams (use chrf-compute)
configurable n-gram orders or keeping whitespace (use chrf-components)
multiple references per segment (chrF++.py "*#" syntax) or best-reference selection
language-specific word tokenization such as Moses, 13a or CJK segmentation
non-ASCII punctuation splitting such as U+3002 or U+00BF

Behavior

Score is Popovic 2017 section 2 eq. (1): F_beta = (1 + beta^2) * P * R / (beta^2 * P + R), where P and R are the precisions and recalls of character n-grams n = 1..6 and word n-grams n = 1..2 arithmetically averaged together (8 orders), reported on a 0..100 scale.
Character n-grams are Unicode code point sequences of the text with every whitespace code point removed: U+0009-U+000D, U+001C-U+001F, U+0020, U+0085, U+00A0, U+1680, U+2000-U+200A, U+2028, U+2029, U+202F, U+205F, U+3000. No Unicode normalization is applied.
Word tokens follow separate_punctuation in Popovic chrF++.py (sacreBLEU _remove_punctuation): split the text on runs of the same whitespace set; a word of one code point is kept; otherwise if its last code point is one of the 32 ASCII punctuation characters !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~ it becomes [rest, mark]; else if its first code point is one, [mark, rest]; else it is kept. At most one mark is split per word, so "(hi)" gives "(hi" and ")", and "..." gives ".." and ".".
Clipped matches per order are sum over reference n-grams of min(reference count, hypothesis count); word bigrams are adjacent token pairs. An order enters the averages only when both sides have at least one n-gram of that order (sacreBLEU 2.x effective order); if none does, or both averages are 0, the score is 0.
Corpus score sums per-order counts over all segments; segment_scores and word_tokens are per segment in input order. beta omitted means 2; lowercase omitted means false, and true applies String.prototype.toLowerCase (runtime Unicode default case mapping) before tokenization, so word_tokens are lowercased too.
score is the exact value rounded to six decimals, ties to even; score_fraction is the exact reduced fraction n/d on the 0..100 scale.
Validation order: plain object, unknown keys, missing keys, hypotheses then references (array, at most 256 segments, at least one, strings), equal segment counts, total UTF-8 bytes of all texts at most 65536, lone surrogates, beta, lowercase. Inputs are never mutated.

Input

hypotheses (array of string, required): min items 1; max items 256
references (array of string, required): min items 1; max items 256
beta (integer, optional): min 1; max 16
lowercase (boolean, optional)

Output

metric (constant "chrF++", required)
char_order (constant 6, required)
word_order (constant 2, required)
beta (integer, required): min 1; max 16
lowercase (boolean, required)
segment_count (integer, required): min 1; max 256
score (string, required): pattern ^(100\.000000|[0-9]{1,2}\.[0-9]{6})$
score_fraction (string, required): pattern ^(0|[1-9][0-9]*)/[1-9][0-9]*$
segment_scores (array of string, required): min items 1; max items 256; each pattern ^(100\.000000|[0-9]{1,2}\.[0-9]{6})$
word_tokens (array of object, required): min items 1; max items 256

Limits

max segments: 256
max text bytes: 65536

Example

Request input:

{
  "hypotheses": [
    "The relationship between cats and dogs is not exactly friendly.",
    "a good bookshop is just a genteel black hole that knows how to read."
  ],
  "references": [
    "The relationship between dogs and cats is not exactly friendly.",
    "A good bookshop is just a genteel Black Hole that knows how to read."
  ]
}

Response:

{
  "result": {
    "metric": "chrF++",
    "char_order": 6,
    "word_order": 2,
    "beta": 2,
    "lowercase": false,
    "segment_count": 2,
    "score": "82.872637",
    "score_fraction": "40260030811255/485806076756",
    "segment_scores": [
      "83.308483",
      "82.578862"
    ],
    "word_tokens": [
      {
        "hypothesis": [
          "The",
          "relationship",
          "between",
          "cats",
          "and",
          "dogs",
          "is",
          "not",
          "exactly",
          "friendly",
          "."
        ],
        "reference": [
          "The",
          "relationship",
          "between",
          "dogs",
          "and",
          "cats",
          "is",
          "not",
          "exactly",
          "friendly",
          "."
        ]
      },
      {
        "hypothesis": [
          "a",
          "good",
          "bookshop",
          "is",
          "just",
          "a",
          "genteel",
          "black",
          "hole",
          "that",
          "knows",
          "how",
          "to",
          "read",
          "."
        ],
        "reference": [
          "A",
          "good",
          "bookshop",
          "is",
          "just",
          "a",
          "genteel",
          "Black",
          "Hole",
          "that",
          "knows",
          "how",
          "to",
          "read",
          "."
        ]
      }
    ]
  }
}

How to call it

MCP

Connect https://computefirst.net/mcp (setup), then call execute with:

{
  "id": "chrf-plus-plus-compute",
  "version": "1.0.0",
  "input": {
    "hypotheses": [
      "The relationship between cats and dogs is not exactly friendly.",
      "a good bookshop is just a genteel black hole that knows how to read."
    ],
    "references": [
      "The relationship between dogs and cats is not exactly friendly.",
      "A good bookshop is just a genteel Black Hole that knows how to read."
    ]
  }
}

HTTP (no key)

curl -X POST https://computefirst.net/v1/tools/chrf-plus-plus-compute/versions/1.0.0/execute \
  -H "Content-Type: application/json" \
  -d '{"hypotheses":["The relationship between cats and dogs is not exactly friendly.","a good bookshop is just a genteel black hole that knows how to read."],"references":["The relationship between dogs and cats is not exactly friendly.","A good bookshop is just a genteel Black Hole that knows how to read."]}'

The machine-readable contract is at /v1/tools/chrf-plus-plus-compute/versions/1.0.0.

CLI

node cli.mjs run chrf-plus-plus-compute 1.0.0 --input input.json --base-url https://computefirst.net

Get the client at /clients/cli/.

Chrf compute: Compute exact chrF (character 6-gram F-beta, Popovic 2015) for hypothesis/reference segments, corpus and per segment.
Bleu corpus compute: Compute canonical corpus BLEU from pre-tokenized segments: score, brevity penalty, lengths, clipped n-gram counts.
Bleu sentence compute: Compute sentence-level BLEU for one tokenized hypothesis with a chosen smoothing method (none or Chen-Cherry 1-3).
Bleu ngrams count: Count BLEU clipped n-gram matches and totals per order for tokenized segments, with optional clipping tables.
Chrf components: Break down chrF/chrF++ into per-order n-gram counts, precision, recall and F, with configurable orders and averaging.
Bleu brevity penalty: Compute the BLEU corpus brevity penalty from candidate and reference lengths using closest reference lengths.