Text minhash cluster

text-minhash-cluster · version 1.0.0 · Text · free, no key needed

Cluster near-duplicate texts by fixed-seed MinHash signatures, with estimated Jaccard per pair.

Use when you need to: cluster near duplicate llm outputs with minhash · group almost identical responses together · compute minhash signatures for a batch of texts.

Decide before calling

Read the versioned contract and the supported scope below. Reuse text-minhash-cluster@1.0.0 when your input, required output and limits match it. Choose another approach for an unsupported operation.

Explain the choice

"I can use text-minhash-cluster@1.0.0 for cluster near duplicate llm outputs with minhash. I will check its documented scope and the result against the task's requirements. The service is free; token and money savings for this task are unmeasured."

Supported

• cluster near duplicate llm outputs with minhash
• group almost identical responses together
• compute minhash signatures for a batch of texts
• estimate jaccard similarity with minhash
• reproduce a textbook minhash signature matrix
• collapse repeated agent answers into groups

Not supported

• exact Jaccard values (use text-near-duplicates-find or text-shingle-similarity)
• locality-sensitive hashing bands or approximate candidate pruning (every pair is compared)
• random or time-seeded hash functions (all seeds are fixed)

Behavior

• Shingling as text-shingle-similarity (lowercase default false, collapse_whitespace default true, shingle_unit default 'char', shingle_size default 5, sets).
• Element id x of a shingle: element_ids 'fnv1a32' (default) is FNV-1a 32-bit (RFC 9923: offset basis 0x811C9DC5, prime 0x01000193, one byte at a time) over the shingle's UTF-8 bytes, an unsigned value below 2^32; 'universe_row' is the 0-based row of the shingle in the characteristic matrix whose rows are the distinct shingles of all documents in Unicode code point order (MMDS 3.3.1).
• Hash functions h_i(x) = (a_i * x + b_i) mod m, computed exactly with non-negative integers (MMDS 3.3.5). Default m = 4294967311 (2^32 + 15, prime) and num_hashes = 64 (1 to 256) functions with a_i = 1 + (i * 2654435761 mod (m - 1)) and b_i = (i * 2246822519) mod m for i = 0 .. num_hashes - 1, so h_0(x) = x. Alternatively hash_functions gives explicit [{a, b}] (1 to 256, 1 <= a < m, 0 <= b < m) and modulus gives m (2 to 4294967311); num_hashes and hash_functions are exclusive.
• Signature component i of a document = min over its shingles of h_i(x); a document with no shingles gets m (a value no h_i can produce) in every component. matching_hashes = number of equal components; estimate = matching_hashes / num_hashes as a correctly rounded IEEE-754 double.
• A pair (a < b) qualifies when estimate >= threshold (optional, default 0.8, 0 to 1). clusters are the connected components of the qualifying pairs (single linkage), each listed in ascending index order, components ordered by their smallest index; every document is in exactly one cluster (singletons included).
• pairs lists qualifying pairs by matching_hashes descending, then a, then b; at most 1000 (pair_count, pairs_truncated). signatures is the full signature matrix (one row per document) when include_signatures is true (default false), else null.

Input

• documents (array of string, required): min items 2; max items 200
• threshold (number, optional): min 0; max 1
• num_hashes (integer, optional): min 1; max 256
• hash_functions (array of object, optional): min items 1; max items 256
• modulus (integer, optional): min 2; max 4294967311
• element_ids (one of "fnv1a32", "universe_row", optional)
• include_signatures (boolean, optional)
• shingle_unit (one of "char", "word", optional)
• shingle_size (integer, optional): min 1; max 32
• lowercase (boolean, optional)
• collapse_whitespace (boolean, optional)

Output

• num_hashes (integer, required): min 1; max 256
• modulus (integer, required): min 2; max 4294967311
• clusters (array of array of integer, required): each min items 1; each each min 0
• pairs (array of object, required): max items 1000
• pair_count (integer, required): min 0
• pairs_truncated (boolean, required)
• shingle_counts (array of integer, required): each min 0
• signatures (array or null, required)

Limits

• max documents: 200
• max total bytes: 131072
• max shingle size: 32
• max hashes: 256
• max pairs: 1000
• max signature values: 16384

Example

Request input:

{
  "documents": [
    "a d",
    "c",
    "b d e",
    "a c d"
  ],
  "shingle_unit": "word",
  "shingle_size": 1,
  "element_ids": "universe_row",
  "modulus": 5,
  "hash_functions": [
    {
      "a": 1,
      "b": 1
    },
    {
      "a": 3,
      "b": 1
    }
  ],
  "threshold": 0.5,
  "include_signatures": true
}

Response:

{
  "result": {
    "num_hashes": 2,
    "modulus": 5,
    "clusters": [
      [
        0,
        2,
        3
      ],
      [
        1
      ]
    ],
    "pairs": [
      {
        "a": 0,
        "b": 3,
        "matching_hashes": 2,
        "estimate": 1
      },
      {
        "a": 0,
        "b": 2,
        "matching_hashes": 1,
        "estimate": 0.5
      },
      {
        "a": 2,
        "b": 3,
        "matching_hashes": 1,
        "estimate": 0.5
      }
    ],
    "pair_count": 3,
    "pairs_truncated": false,
    "shingle_counts": [
      2,
      1,
      3,
      3
    ],
    "signatures": [
      [
        1,
        0
      ],
      [
        3,
        2
      ],
      [
        0,
        0
      ],
      [
        1,
        0
      ]
    ]
  }
}

How to call it

MCP

Connect https://computefirst.net/mcp (setup), then call execute with:

{
  "id": "text-minhash-cluster",
  "version": "1.0.0",
  "input": {
    "documents": [
      "a d",
      "c",
      "b d e",
      "a c d"
    ],
    "shingle_unit": "word",
    "shingle_size": 1,
    "element_ids": "universe_row",
    "modulus": 5,
    "hash_functions": [
      {
        "a": 1,
        "b": 1
      },
      {
        "a": 3,
        "b": 1
      }
    ],
    "threshold": 0.5,
    "include_signatures": true
  }
}

HTTP (no key)

curl -X POST https://computefirst.net/v1/tools/text-minhash-cluster/versions/1.0.0/execute \
  -H "Content-Type: application/json" \
  -d '{"documents":["a d","c","b d e","a c d"],"shingle_unit":"word","shingle_size":1,"element_ids":"universe_row","modulus":5,"hash_functions":[{"a":1,"b":1},{"a":3,"b":1}],"threshold":0.5,"include_signatures":true}'

The machine-readable contract is at /v1/tools/text-minhash-cluster/versions/1.0.0.

CLI

node cli.mjs run text-minhash-cluster 1.0.0 --input input.json --base-url https://computefirst.net

Get the client at /clients/cli/.

Related tools

• Text near duplicates find: Find every pair of texts whose exact shingle Jaccard similarity reaches a threshold, in a fixed order.
• Text shingle similarity: Exact shingle-set resemblance (Jaccard) and containment of two texts, with counts and shared shingles.
• Citation record dedupe: Group duplicate CSL citation records by deterministic DOI or folded year|author|title match keys, in input order.
• Text sentences chunk: Pack whole UAX #29 sentences into chunks of at most max_size code points, UTF-16 units or UTF-8 bytes.
• Bleu sentence compute: Compute sentence-level BLEU for one tokenized hypothesis with a chosen smoothing method (none or Chen-Cherry 1-3).
• Text paragraphs split: Split text into paragraphs separated by blank lines (CommonMark line rules) with code point spans and line numbers.