{"id":"chrf-plus-plus-compute","version":"1.0.0","description":"Compute exact chrF++ (character 6-grams plus word 1-2-grams, Popovic 2017) with the word tokens it scored.","supported_operations":["compute chrF++ score for machine translation output","chrF++ with character 6-grams and word bigrams against a reference","chrF2++ corpus score over aligned segments","sentence-level chrF++ score per segment","show the chrF++ word tokens after punctuation splitting","case-insensitive chrF++ with lowercasing","ציון chrF++ עם bigrams של מילים לתרגום"],"unsupported_operations":["plain chrF without word n-grams (use chrf-compute)","configurable n-gram orders or keeping whitespace (use chrf-components)","multiple references per segment (chrF++.py \"*#\" syntax) or best-reference selection","language-specific word tokenization such as Moses, 13a or CJK segmentation","non-ASCII punctuation splitting such as U+3002 or U+00BF"],"semantics":["Score is Popovic 2017 section 2 eq. (1): F_beta = (1 + beta^2) * P * R / (beta^2 * P + R), where P and R are the precisions and recalls of character n-grams n = 1..6 and word n-grams n = 1..2 arithmetically averaged together (8 orders), reported on a 0..100 scale.","Character n-grams are Unicode code point sequences of the text with every whitespace code point removed: U+0009-U+000D, U+001C-U+001F, U+0020, U+0085, U+00A0, U+1680, U+2000-U+200A, U+2028, U+2029, U+202F, U+205F, U+3000. No Unicode normalization is applied.","Word tokens follow separate_punctuation in Popovic chrF++.py (sacreBLEU _remove_punctuation): split the text on runs of the same whitespace set; a word of one code point is kept; otherwise if its last code point is one of the 32 ASCII punctuation characters !\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~ it becomes [rest, mark]; else if its first code point is one, [mark, rest]; else it is kept. At most one mark is split per word, so \"(hi)\" gives \"(hi\" and \")\", and \"...\" gives \"..\" and \".\".","Clipped matches per order are sum over reference n-grams of min(reference count, hypothesis count); word bigrams are adjacent token pairs. An order enters the averages only when both sides have at least one n-gram of that order (sacreBLEU 2.x effective order); if none does, or both averages are 0, the score is 0.","Corpus score sums per-order counts over all segments; segment_scores and word_tokens are per segment in input order. beta omitted means 2; lowercase omitted means false, and true applies String.prototype.toLowerCase (runtime Unicode default case mapping) before tokenization, so word_tokens are lowercased too.","score is the exact value rounded to six decimals, ties to even; score_fraction is the exact reduced fraction n/d on the 0..100 scale.","Validation order: plain object, unknown keys, missing keys, hypotheses then references (array, at most 256 segments, at least one, strings), equal segment counts, total UTF-8 bytes of all texts at most 65536, lone surrogates, beta, lowercase. Inputs are never mutated."],"limits":{"max_segments":256,"max_text_bytes":65536},"pricing":{"status":"unpriced","charge_usd":null},"input_schema":{"type":"object","additionalProperties":false,"properties":{"hypotheses":{"type":"array","minItems":1,"maxItems":256,"items":{"type":"string"}},"references":{"type":"array","minItems":1,"maxItems":256,"items":{"type":"string"}},"beta":{"type":"integer","minimum":1,"maximum":16},"lowercase":{"type":"boolean"}},"required":["hypotheses","references"]},"output_schema":{"type":"object","additionalProperties":false,"properties":{"metric":{"type":"string","const":"chrF++"},"char_order":{"type":"integer","const":6},"word_order":{"type":"integer","const":2},"beta":{"type":"integer","minimum":1,"maximum":16},"lowercase":{"type":"boolean"},"segment_count":{"type":"integer","minimum":1,"maximum":256},"score":{"type":"string","pattern":"^(100\\.000000|[0-9]{1,2}\\.[0-9]{6})$"},"score_fraction":{"type":"string","pattern":"^(0|[1-9][0-9]*)/[1-9][0-9]*$"},"segment_scores":{"type":"array","minItems":1,"maxItems":256,"items":{"type":"string","pattern":"^(100\\.000000|[0-9]{1,2}\\.[0-9]{6})$"}},"word_tokens":{"type":"array","minItems":1,"maxItems":256,"items":{"type":"object","additionalProperties":false,"properties":{"hypothesis":{"type":"array","items":{"type":"string","minLength":1}},"reference":{"type":"array","items":{"type":"string","minLength":1}}},"required":["hypothesis","reference"]}}},"required":["metric","char_order","word_order","beta","lowercase","segment_count","score","score_fraction","segment_scores","word_tokens"]},"examples":[{"input":{"hypotheses":["The relationship between cats and dogs is not exactly friendly.","a good bookshop is just a genteel black hole that knows how to read."],"references":["The relationship between dogs and cats is not exactly friendly.","A good bookshop is just a genteel Black Hole that knows how to read."]},"output":{"metric":"chrF++","char_order":6,"word_order":2,"beta":2,"lowercase":false,"segment_count":2,"score":"82.872637","score_fraction":"40260030811255/485806076756","segment_scores":["83.308483","82.578862"],"word_tokens":[{"hypothesis":["The","relationship","between","cats","and","dogs","is","not","exactly","friendly","."],"reference":["The","relationship","between","dogs","and","cats","is","not","exactly","friendly","."]},{"hypothesis":["a","good","bookshop","is","just","a","genteel","black","hole","that","knows","how","to","read","."],"reference":["A","good","bookshop","is","just","a","genteel","Black","Hole","that","knows","how","to","read","."]}]}},{"input":{"hypotheses":["a."],"references":["a ."]},"output":{"metric":"chrF++","char_order":6,"word_order":2,"beta":2,"lowercase":false,"segment_count":1,"score":"100.000000","score_fraction":"100/1","segment_scores":["100.000000"],"word_tokens":[{"hypothesis":["a","."],"reference":["a","."]}]}}],"execute_url":"/v1/tools/chrf-plus-plus-compute/versions/1.0.0/execute"}