{"id":"text-near-duplicates-find","version":"1.0.0","description":"Find every pair of texts whose exact shingle Jaccard similarity reaches a threshold, in a fixed order.","supported_operations":["find near duplicate agent outputs","which of these answers are almost the same","detect near-duplicate documents with exact jaccard","list pairs of similar texts above a threshold","deduplicate model responses by shingle overlap","find repeated paragraphs across many outputs"],"unsupported_operations":["approximate MinHash estimates or clustering (use text-minhash-cluster)","semantic paraphrase detection (only literal shingle overlap)","comparing one pair in detail with containment and shared shingles (use text-shingle-similarity)"],"semantics":["Shingling is identical to text-shingle-similarity: optional lowercase (default false), collapse_whitespace (default true), shingle_unit (default 'char') and shingle_size (default 5); sets, not bags; a text shorter than k units has one shingle; an empty text has the empty set.","Every pair (a, b) with a < b (0-based document indexes) is compared exactly: intersection = |A ∩ B|, union = |A ∪ B|, resemblance = intersection / union as a correctly rounded IEEE-754 double, and 1 when both sets are empty.","A pair is reported when resemblance >= threshold (optional, default 0.8, 0 to 1), comparing the double resemblance with the double threshold.","Order: higher exact resemblance first (fractions compared by cross-multiplication, not by rounded doubles), then smaller a, then smaller b. At most 1000 pairs are returned; pair_count is the number that qualified and pairs_truncated is true when pairs were cut.","shingle_counts[i] is the number of distinct shingles of document i; pairs_compared is n(n-1)/2."],"limits":{"max_documents":200,"max_total_bytes":131072,"max_shingle_size":32,"max_pairs":1000},"pricing":{"status":"unpriced","charge_usd":null},"input_schema":{"type":"object","additionalProperties":false,"required":["documents"],"properties":{"documents":{"type":"array","minItems":2,"maxItems":200,"items":{"type":"string"}},"threshold":{"type":"number","minimum":0,"maximum":1},"shingle_unit":{"type":"string","enum":["char","word"]},"shingle_size":{"type":"integer","minimum":1,"maximum":32},"lowercase":{"type":"boolean"},"collapse_whitespace":{"type":"boolean"}}},"output_schema":{"type":"object","additionalProperties":false,"required":["pairs","pair_count","pairs_truncated","pairs_compared","shingle_counts"],"properties":{"pairs":{"type":"array","maxItems":1000,"items":{"type":"object","additionalProperties":false,"required":["a","b","resemblance","intersection","union"],"properties":{"a":{"type":"integer","minimum":0},"b":{"type":"integer","minimum":1},"resemblance":{"type":"number","minimum":0,"maximum":1},"intersection":{"type":"integer","minimum":0},"union":{"type":"integer","minimum":0}}}},"pair_count":{"type":"integer","minimum":0},"pairs_truncated":{"type":"boolean"},"pairs_compared":{"type":"integer","minimum":1},"shingle_counts":{"type":"array","items":{"type":"integer","minimum":0}}}},"examples":[{"input":{"documents":["a d","b d e","a c d"],"shingle_unit":"word","shingle_size":1,"threshold":0.25},"output":{"pairs":[{"a":0,"b":2,"resemblance":0.6666666666666666,"intersection":2,"union":3},{"a":0,"b":1,"resemblance":0.25,"intersection":1,"union":4}],"pair_count":2,"pairs_truncated":false,"pairs_compared":3,"shingle_counts":[2,3,3]}},{"input":{"documents":["The build passed on main.","The build  passed on main!","Deploy failed: timeout."]},"output":{"pairs":[{"a":0,"b":1,"resemblance":0.9090909090909091,"intersection":20,"union":22}],"pair_count":1,"pairs_truncated":false,"pairs_compared":3,"shingle_counts":[21,21,19]}}],"execute_url":"/v1/tools/text-near-duplicates-find/versions/1.0.0/execute"}