{"id":"chrf-compute","version":"1.0.0","description":"Compute exact chrF (character 6-gram F-beta, Popovic 2015) for hypothesis/reference segments, corpus and per segment.","supported_operations":["compute chrF score for a translation against a reference","character n-gram F-score chrF2 for machine translation output","corpus-level chrF over aligned hypothesis and reference segments","sentence-level chrF score for each segment","chrF3 with beta 3 for MT evaluation","exact chrF fraction without floating-point rounding","חישוב chrF2 לתרגום מכונה מול תרגום ייחוס"],"unsupported_operations":["chrF++ with word n-grams (use chrf-plus-plus-compute)","multiple references per segment or best-reference selection","chrF++.py epsilon-smoothed per-order F averaging (use chrf-components with averaging per_order_f)","BLEU, TER, METEOR or learned metrics such as COMET","Unicode normalization, detokenization or language-specific tokenization"],"semantics":["Score is Popovic 2015 section 2 eq. (1): chrF_beta = (1 + beta^2) * chrP * chrR / (beta^2 * chrP + chrR), with chrP and chrR the character n-gram precision and recall arithmetically averaged over n = 1..6, reported on a 0..100 scale.","Character n-grams are sequences of Unicode code points after removing every whitespace code point: U+0009-U+000D, U+001C-U+001F, U+0020, U+0085, U+00A0, U+1680, U+2000-U+200A, U+2028, U+2029, U+202F, U+205F, U+3000 (the Python 3 str.split() set used by sacreBLEU). Other characters, including U+200B and U+FEFF, are kept. No Unicode normalization is applied, so NFC and NFD spellings differ.","Clipped matches per order are sum over reference n-grams of min(reference count, hypothesis count). An order enters the averages only when both the hypothesis and the reference have at least one n-gram of that order (sacreBLEU 2.x effective order). If no order is effective, or both averages are 0, the score is 0.","Corpus score: per-order counts are summed over all segments before averaging. segment_scores apply the same rule to each segment alone, in input order.","beta is an integer 1..16; omitted means 2 (chrF2, the sacreBLEU and Popovic 2017 default). lowercase omitted means false (case-sensitive); true applies String.prototype.toLowerCase (Unicode default full case mapping from the runtime) to both texts first.","score is the exact value rounded to six decimals, ties to even; score_fraction is the exact reduced fraction n/d on the 0..100 scale (0/1 and 100/1 at the ends).","Validation order: plain object, unknown keys, missing keys, hypotheses then references (array, at most 256 segments, at least one, strings), equal segment counts, total UTF-8 bytes of all texts at most 65536, lone surrogates, beta, lowercase. Inputs are never mutated."],"limits":{"max_segments":256,"max_text_bytes":65536},"pricing":{"status":"unpriced","charge_usd":null},"input_schema":{"type":"object","additionalProperties":false,"properties":{"hypotheses":{"type":"array","minItems":1,"maxItems":256,"items":{"type":"string"}},"references":{"type":"array","minItems":1,"maxItems":256,"items":{"type":"string"}},"beta":{"type":"integer","minimum":1,"maximum":16},"lowercase":{"type":"boolean"}},"required":["hypotheses","references"]},"output_schema":{"type":"object","additionalProperties":false,"properties":{"metric":{"type":"string","const":"chrF"},"char_order":{"type":"integer","const":6},"word_order":{"type":"integer","const":0},"beta":{"type":"integer","minimum":1,"maximum":16},"lowercase":{"type":"boolean"},"segment_count":{"type":"integer","minimum":1,"maximum":256},"score":{"type":"string","pattern":"^(100\\.000000|[0-9]{1,2}\\.[0-9]{6})$"},"score_fraction":{"type":"string","pattern":"^(0|[1-9][0-9]*)/[1-9][0-9]*$"},"segment_scores":{"type":"array","minItems":1,"maxItems":256,"items":{"type":"string","pattern":"^(100\\.000000|[0-9]{1,2}\\.[0-9]{6})$"}}},"required":["metric","char_order","word_order","beta","lowercase","segment_count","score","score_fraction","segment_scores"]},"examples":[{"input":{"hypotheses":["The relationship between cats and dogs is not exactly friendly.","a good bookshop is just a genteel black hole that knows how to read."],"references":["The relationship between dogs and cats is not exactly friendly.","A good bookshop is just a genteel Black Hole that knows how to read."]},"output":{"metric":"chrF","char_order":6,"word_order":0,"beta":2,"lowercase":false,"segment_count":2,"score":"84.642149","score_fraction":"7116866012380/84081820977","segment_scores":["84.411311","84.867055"]}},{"input":{"hypotheses":["aa"],"references":["ab"],"beta":3},"output":{"metric":"chrF","char_order":6,"word_order":0,"beta":3,"lowercase":false,"segment_count":1,"score":"25.000000","score_fraction":"25/1","segment_scores":["25.000000"]}}],"execute_url":"/v1/tools/chrf-compute/versions/1.0.0/execute"}