Bleu corpus compute

bleu-corpus-compute · version 1.0.0 · Text · free, no key needed

Compute canonical corpus BLEU from pre-tokenized segments: score, brevity penalty, lengths, clipped n-gram counts.

Use when you need to: corpus BLEU score from tokenized segments · system-level BLEU for a machine translation test set · compute BLEU-4 over a whole corpus without smoothing.

Decide before calling

Read the versioned contract and the supported scope below. Reuse bleu-corpus-compute@1.0.0 when your input, required output and limits match it. Choose another approach for an unsupported operation.

Explain the choice

"I can use bleu-corpus-compute@1.0.0 for corpus BLEU score from tokenized segments. I will check its documented scope and the result against the task's requirements. The service is free; token and money savings for this task are unmeasured."

Supported

• corpus BLEU score from tokenized segments
• system-level BLEU for a machine translation test set
• compute BLEU-4 over a whole corpus without smoothing
• Papineni corpus BLEU with multiple references per segment
• micro-averaged BLEU across many hypothesis segments
• document-level BLEU with brevity penalty and n-gram precisions
• חישוב ציון איכות תרגום לקורפוס שלם

Not supported

• tokenizing raw text (13a, intl, char or any other tokenizer); tokens must be supplied
• sentence-level smoothing (Chen and Cherry methods) or effective-order rescaling
• averaging per-sentence BLEU scores (macro average)
• custom non-uniform n-gram weights, chrF, TER, METEOR or case-insensitive matching

Behavior

• Input is pre-tokenized: hypotheses[i] is the token list of segment i and references[i] lists 1 to 8 reference token lists for it. Tokens are compared by exact string equality: no case folding, no Unicode normalization, no trimming.
• For each order n = 1..max_order (max_order omitted means 4) the clipped matches m_n and totals l_n are summed over all segments before dividing (Papineni 2002 section 2.1.1); clipping uses the largest count of the n-gram in any single reference.
• candidate_length c is the total hypothesis token count; reference_length r sums, per segment, the reference length closest to the hypothesis length, ties going to the shorter reference.
• brevity_penalty is 1 when c > r, 0 when c = 0 (even if r = 0), otherwise exp(1 - r/c).
• bleu = brevity_penalty * exp((1/N) * sum of ln(m_n / l_n)) with N = max_order, summed in order n = 1..N; it is exactly 0 when brevity_penalty is 0 or any order has m_n = 0 (including l_n = 0). No smoothing is applied.
• decimals (0-15) rounds bleu and brevity_penalty with Number.prototype.toFixed (nearest, computed on the exact binary value); omitted means full IEEE-754 double precision. Counts are exact integers.
• Scores are on the 0-1 scale; multiply by 100 for the sacreBLEU percentage scale.

Input

• hypotheses (array of array of string, required): min items 1; max items 1000; each each min length 1; each each max length 128
• references (array of array of array of string, required): min items 1; max items 1000; each min items 1; each max items 8; each each each min length 1; each each each max length 128
• max_order (integer, optional): min 1; max 8
• decimals (integer, optional): min 0; max 15

Output

• bleu (number, required): min 0; max 1
• brevity_penalty (number, required): min 0; max 1
• candidate_length (integer, required): min 0
• reference_length (integer, required): min 0
• max_order (integer, required): min 1; max 8
• matches (array of integer, required): min items 1; max items 8; each min 0
• totals (array of integer, required): min items 1; max items 8; each min 0

Limits

• max segments: 1000
• max references per segment: 8
• max total tokens: 50000
• max token bytes: 128
• max total token bytes: 1048576

Example

Request input:

{
  "hypotheses": [
    [
      "this",
      "is",
      "a",
      "test"
    ]
  ],
  "references": [
    [
      [
        "this",
        "is",
        "a",
        "test"
      ]
    ]
  ]
}

Response:

{
  "result": {
    "bleu": 1,
    "brevity_penalty": 1,
    "candidate_length": 4,
    "reference_length": 4,
    "max_order": 4,
    "matches": [
      4,
      3,
      2,
      1
    ],
    "totals": [
      4,
      3,
      2,
      1
    ]
  }
}

How to call it

MCP

Connect https://computefirst.net/mcp (setup), then call execute with:

{
  "id": "bleu-corpus-compute",
  "version": "1.0.0",
  "input": {
    "hypotheses": [
      [
        "this",
        "is",
        "a",
        "test"
      ]
    ],
    "references": [
      [
        [
          "this",
          "is",
          "a",
          "test"
        ]
      ]
    ]
  }
}

HTTP (no key)

curl -X POST https://computefirst.net/v1/tools/bleu-corpus-compute/versions/1.0.0/execute \
  -H "Content-Type: application/json" \
  -d '{"hypotheses":[["this","is","a","test"]],"references":[[["this","is","a","test"]]]}'

The machine-readable contract is at /v1/tools/bleu-corpus-compute/versions/1.0.0.

CLI

node cli.mjs run bleu-corpus-compute 1.0.0 --input input.json --base-url https://computefirst.net

Get the client at /clients/cli/.

Related tools

• Chrf compute: Compute exact chrF (character 6-gram F-beta, Popovic 2015) for hypothesis/reference segments, corpus and per segment.
• Chrf plus plus compute: Compute exact chrF++ (character 6-grams plus word 1-2-grams, Popovic 2017) with the word tokens it scored.
• Bleu brevity penalty: Compute the BLEU corpus brevity penalty from candidate and reference lengths using closest reference lengths.
• Bleu ngrams count: Count BLEU clipped n-gram matches and totals per order for tokenized segments, with optional clipping tables.
• Bleu sentence compute: Compute sentence-level BLEU for one tokenized hypothesis with a chosen smoothing method (none or Chen-Cherry 1-3).
• Bleu sentence components: Extract sentence BLEU statistics: clipped matches, n-gram totals, candidate and closest reference length.