{"id":"text-minhash-cluster","version":"1.0.0","description":"Cluster near-duplicate texts by fixed-seed MinHash signatures, with estimated Jaccard per pair.","supported_operations":["cluster near duplicate llm outputs with minhash","group almost identical responses together","compute minhash signatures for a batch of texts","estimate jaccard similarity with minhash","reproduce a textbook minhash signature matrix","collapse repeated agent answers into groups"],"unsupported_operations":["exact Jaccard values (use text-near-duplicates-find or text-shingle-similarity)","locality-sensitive hashing bands or approximate candidate pruning (every pair is compared)","random or time-seeded hash functions (all seeds are fixed)"],"semantics":["Shingling as text-shingle-similarity (lowercase default false, collapse_whitespace default true, shingle_unit default 'char', shingle_size default 5, sets).","Element id x of a shingle: element_ids 'fnv1a32' (default) is FNV-1a 32-bit (RFC 9923: offset basis 0x811C9DC5, prime 0x01000193, one byte at a time) over the shingle's UTF-8 bytes, an unsigned value below 2^32; 'universe_row' is the 0-based row of the shingle in the characteristic matrix whose rows are the distinct shingles of all documents in Unicode code point order (MMDS 3.3.1).","Hash functions h_i(x) = (a_i * x + b_i) mod m, computed exactly with non-negative integers (MMDS 3.3.5). Default m = 4294967311 (2^32 + 15, prime) and num_hashes = 64 (1 to 256) functions with a_i = 1 + (i * 2654435761 mod (m - 1)) and b_i = (i * 2246822519) mod m for i = 0 .. num_hashes - 1, so h_0(x) = x. Alternatively hash_functions gives explicit [{a, b}] (1 to 256, 1 <= a < m, 0 <= b < m) and modulus gives m (2 to 4294967311); num_hashes and hash_functions are exclusive.","Signature component i of a document = min over its shingles of h_i(x); a document with no shingles gets m (a value no h_i can produce) in every component. matching_hashes = number of equal components; estimate = matching_hashes / num_hashes as a correctly rounded IEEE-754 double.","A pair (a < b) qualifies when estimate >= threshold (optional, default 0.8, 0 to 1). clusters are the connected components of the qualifying pairs (single linkage), each listed in ascending index order, components ordered by their smallest index; every document is in exactly one cluster (singletons included).","pairs lists qualifying pairs by matching_hashes descending, then a, then b; at most 1000 (pair_count, pairs_truncated). signatures is the full signature matrix (one row per document) when include_signatures is true (default false), else null."],"limits":{"max_documents":200,"max_total_bytes":131072,"max_shingle_size":32,"max_hashes":256,"max_pairs":1000,"max_signature_values":16384},"pricing":{"status":"unpriced","charge_usd":null},"input_schema":{"type":"object","additionalProperties":false,"required":["documents"],"properties":{"documents":{"type":"array","minItems":2,"maxItems":200,"items":{"type":"string"}},"threshold":{"type":"number","minimum":0,"maximum":1},"num_hashes":{"type":"integer","minimum":1,"maximum":256},"hash_functions":{"type":"array","minItems":1,"maxItems":256,"items":{"type":"object","additionalProperties":false,"required":["a","b"],"properties":{"a":{"type":"integer","minimum":1},"b":{"type":"integer","minimum":0}}}},"modulus":{"type":"integer","minimum":2,"maximum":4294967311},"element_ids":{"type":"string","enum":["fnv1a32","universe_row"]},"include_signatures":{"type":"boolean"},"shingle_unit":{"type":"string","enum":["char","word"]},"shingle_size":{"type":"integer","minimum":1,"maximum":32},"lowercase":{"type":"boolean"},"collapse_whitespace":{"type":"boolean"}}},"output_schema":{"type":"object","additionalProperties":false,"required":["num_hashes","modulus","clusters","pairs","pair_count","pairs_truncated","shingle_counts","signatures"],"properties":{"num_hashes":{"type":"integer","minimum":1,"maximum":256},"modulus":{"type":"integer","minimum":2,"maximum":4294967311},"clusters":{"type":"array","items":{"type":"array","minItems":1,"items":{"type":"integer","minimum":0}}},"pairs":{"type":"array","maxItems":1000,"items":{"type":"object","additionalProperties":false,"required":["a","b","matching_hashes","estimate"],"properties":{"a":{"type":"integer","minimum":0},"b":{"type":"integer","minimum":1},"matching_hashes":{"type":"integer","minimum":0},"estimate":{"type":"number","minimum":0,"maximum":1}}}},"pair_count":{"type":"integer","minimum":0},"pairs_truncated":{"type":"boolean"},"shingle_counts":{"type":"array","items":{"type":"integer","minimum":0}},"signatures":{"type":["array","null"],"items":{"type":"array","items":{"type":"integer","minimum":0}}}}},"examples":[{"input":{"documents":["a d","c","b d e","a c d"],"shingle_unit":"word","shingle_size":1,"element_ids":"universe_row","modulus":5,"hash_functions":[{"a":1,"b":1},{"a":3,"b":1}],"threshold":0.5,"include_signatures":true},"output":{"num_hashes":2,"modulus":5,"clusters":[[0,2,3],[1]],"pairs":[{"a":0,"b":3,"matching_hashes":2,"estimate":1},{"a":0,"b":2,"matching_hashes":1,"estimate":0.5},{"a":2,"b":3,"matching_hashes":1,"estimate":0.5}],"pair_count":3,"pairs_truncated":false,"shingle_counts":[2,1,3,3],"signatures":[[1,0],[3,2],[0,0],[1,0]]}},{"input":{"documents":["a","foobar"],"shingle_unit":"word","num_hashes":1,"include_signatures":true},"output":{"num_hashes":1,"modulus":4294967311,"clusters":[[0],[1]],"pairs":[],"pair_count":0,"pairs_truncated":false,"shingle_counts":[1,1],"signatures":[[3826002220],[3214735720]]}}],"execute_url":"/v1/tools/text-minhash-cluster/versions/1.0.0/execute"}