{"id":"text-shingle-similarity","version":"1.0.0","description":"Exact shingle-set resemblance (Jaccard) and containment of two texts, with counts and shared shingles.","supported_operations":["jaccard similarity of two texts using shingles","how similar are these two answers word for word","compute broder resemblance and containment","is text a mostly contained in text b","compare two documents with k-shingles","shared word n-grams between two passages","measure near-duplicate overlap of two strings"],"unsupported_operations":["semantic or embedding similarity (only literal shingle overlap)","MinHash estimates for many documents (use text-minhash-cluster)","bag (multiset) shingling with repeated shingles counted (sets only)","edit distance or diffs (use the text diff tools)"],"semantics":["Normalization (optional fields): lowercase (default false) applies String.prototype.toLowerCase (Unicode case mapping of the runtime, locale-independent); collapse_whitespace (default true) replaces each run of Unicode White_Space code points with one U+0020 and trims both ends.","shingle_unit (default 'char'): 'char' uses Unicode code points, 'word' uses maximal runs of non-White_Space code points. shingle_size k (default 5, 1 to 32). The shingle set holds every run of k consecutive units (Broder 1997 option B: a set, duplicates counted once); a text with 1 to k-1 units has one shingle made of all its units; a text with no units has the empty set. Word shingles join words with one U+0020.","intersection = |A ∩ B|, union = |A ∪ B|; resemblance = intersection / union (Broder r(A,B), MMDS Jaccard similarity), and 1 when both sets are empty (r(A,A) = 1). containment_a_in_b = intersection / |A| (Broder c(A,B)) and 1 when A is empty (an empty text is a contiguous subsequence of any text); containment_b_in_a likewise.","Ratios are IEEE-754 doubles from one correctly rounded division of the exact integer counts, which are also returned.","shared_shingles lists the intersection in Unicode code point order, at most max_shared (optional, default 10, 0 to 100) entries; shared_truncated is true when more exist."],"limits":{"max_text_bytes":65536,"max_shingle_size":32,"max_shared":100},"pricing":{"status":"unpriced","charge_usd":null},"input_schema":{"type":"object","additionalProperties":false,"required":["text_a","text_b"],"properties":{"text_a":{"type":"string","maxLength":65536},"text_b":{"type":"string","maxLength":65536},"shingle_unit":{"type":"string","enum":["char","word"]},"shingle_size":{"type":"integer","minimum":1,"maximum":32},"lowercase":{"type":"boolean"},"collapse_whitespace":{"type":"boolean"},"max_shared":{"type":"integer","minimum":0,"maximum":100}}},"output_schema":{"type":"object","additionalProperties":false,"required":["shingles_a","shingles_b","intersection","union","resemblance","containment_a_in_b","containment_b_in_a","shared_shingles","shared_truncated"],"properties":{"shingles_a":{"type":"integer","minimum":0},"shingles_b":{"type":"integer","minimum":0},"intersection":{"type":"integer","minimum":0},"union":{"type":"integer","minimum":0},"resemblance":{"type":"number","minimum":0,"maximum":1},"containment_a_in_b":{"type":"number","minimum":0,"maximum":1},"containment_b_in_a":{"type":"number","minimum":0,"maximum":1},"shared_shingles":{"type":"array","maxItems":100,"items":{"type":"string"}},"shared_truncated":{"type":"boolean"}}},"examples":[{"input":{"text_a":"a rose is a rose is a rose","text_b":"a rose is a flower which is a rose","shingle_unit":"word","shingle_size":2},"output":{"shingles_a":3,"shingles_b":6,"intersection":3,"union":6,"resemblance":0.5,"containment_a_in_b":1,"containment_b_in_a":0.5,"shared_shingles":["a rose","is a","rose is"],"shared_truncated":false}},{"input":{"text_a":"abcdabd","text_b":"abcd","shingle_size":2},"output":{"shingles_a":5,"shingles_b":3,"intersection":3,"union":5,"resemblance":0.6,"containment_a_in_b":0.6,"containment_b_in_a":1,"shared_shingles":["ab","bc","cd"],"shared_truncated":false}}],"execute_url":"/v1/tools/text-shingle-similarity/versions/1.0.0/execute"}