{"id":"bleu-corpus-compute","version":"1.0.0","description":"Compute canonical corpus BLEU from pre-tokenized segments: score, brevity penalty, lengths, clipped n-gram counts.","supported_operations":["corpus BLEU score from tokenized segments","system-level BLEU for a machine translation test set","compute BLEU-4 over a whole corpus without smoothing","Papineni corpus BLEU with multiple references per segment","micro-averaged BLEU across many hypothesis segments","document-level BLEU with brevity penalty and n-gram precisions","חישוב ציון איכות תרגום לקורפוס שלם"],"unsupported_operations":["tokenizing raw text (13a, intl, char or any other tokenizer); tokens must be supplied","sentence-level smoothing (Chen and Cherry methods) or effective-order rescaling","averaging per-sentence BLEU scores (macro average)","custom non-uniform n-gram weights, chrF, TER, METEOR or case-insensitive matching"],"semantics":["Input is pre-tokenized: hypotheses[i] is the token list of segment i and references[i] lists 1 to 8 reference token lists for it. Tokens are compared by exact string equality: no case folding, no Unicode normalization, no trimming.","For each order n = 1..max_order (max_order omitted means 4) the clipped matches m_n and totals l_n are summed over all segments before dividing (Papineni 2002 section 2.1.1); clipping uses the largest count of the n-gram in any single reference.","candidate_length c is the total hypothesis token count; reference_length r sums, per segment, the reference length closest to the hypothesis length, ties going to the shorter reference.","brevity_penalty is 1 when c > r, 0 when c = 0 (even if r = 0), otherwise exp(1 - r/c).","bleu = brevity_penalty * exp((1/N) * sum of ln(m_n / l_n)) with N = max_order, summed in order n = 1..N; it is exactly 0 when brevity_penalty is 0 or any order has m_n = 0 (including l_n = 0). No smoothing is applied.","decimals (0-15) rounds bleu and brevity_penalty with Number.prototype.toFixed (nearest, computed on the exact binary value); omitted means full IEEE-754 double precision. Counts are exact integers.","Scores are on the 0-1 scale; multiply by 100 for the sacreBLEU percentage scale."],"limits":{"max_segments":1000,"max_references_per_segment":8,"max_total_tokens":50000,"max_token_bytes":128,"max_total_token_bytes":1048576},"pricing":{"status":"unpriced","charge_usd":null},"input_schema":{"type":"object","additionalProperties":false,"required":["hypotheses","references"],"properties":{"hypotheses":{"type":"array","minItems":1,"maxItems":1000,"items":{"type":"array","items":{"type":"string","minLength":1,"maxLength":128}}},"references":{"type":"array","minItems":1,"maxItems":1000,"items":{"type":"array","minItems":1,"maxItems":8,"items":{"type":"array","items":{"type":"string","minLength":1,"maxLength":128}}}},"max_order":{"type":"integer","minimum":1,"maximum":8},"decimals":{"type":"integer","minimum":0,"maximum":15}}},"output_schema":{"type":"object","additionalProperties":false,"required":["bleu","brevity_penalty","candidate_length","reference_length","max_order","matches","totals"],"properties":{"bleu":{"type":"number","minimum":0,"maximum":1},"brevity_penalty":{"type":"number","minimum":0,"maximum":1},"candidate_length":{"type":"integer","minimum":0},"reference_length":{"type":"integer","minimum":0},"max_order":{"type":"integer","minimum":1,"maximum":8},"matches":{"type":"array","minItems":1,"maxItems":8,"items":{"type":"integer","minimum":0}},"totals":{"type":"array","minItems":1,"maxItems":8,"items":{"type":"integer","minimum":0}}}},"examples":[{"input":{"hypotheses":[["this","is","a","test"]],"references":[[["this","is","a","test"]]]},"output":{"bleu":1,"brevity_penalty":1,"candidate_length":4,"reference_length":4,"max_order":4,"matches":[4,3,2,1],"totals":[4,3,2,1]}},{"input":{"hypotheses":[["The","dog","bit","the","man","."],["It","wasn't","surprising","."],["The","man","had","just","bitten","him","."]],"references":[[["The","dog","bit","the","man","."],["The","dog","had","bit","the","man","."]],[["It","was","not","unexpected","."],["No","one","was","surprised","."]],[["The","man","bit","him","first","."],["The","man","had","bitten","the","dog","."]]],"decimals":6},"output":{"bleu":0.485308,"brevity_penalty":0.942873,"candidate_length":17,"reference_length":18,"max_order":4,"matches":[14,7,5,3],"totals":[17,14,11,8]}},{"input":{"hypotheses":[[]],"references":[[["the","cat"]]]},"output":{"bleu":0,"brevity_penalty":0,"candidate_length":0,"reference_length":2,"max_order":4,"matches":[0,0,0,0],"totals":[0,0,0,0]}}],"execute_url":"/v1/tools/bleu-corpus-compute/versions/1.0.0/execute"}