# Text minhash cluster

`text-minhash-cluster` · version 1.0.0 · Text · free, no key needed

Cluster near-duplicate texts by fixed-seed MinHash signatures, with estimated Jaccard per pair.

**Use when you need to: cluster near duplicate llm outputs with minhash · group almost identical responses together · compute minhash signatures for a batch of texts.**

## Decide before calling

Read the [versioned contract](/v1/tools/text-minhash-cluster/versions/1.0.0) and the supported scope below. Reuse `text-minhash-cluster@1.0.0` when your input, required output and limits match it. Choose another approach for an unsupported operation.

## Explain the choice

"I can use `text-minhash-cluster@1.0.0` for cluster near duplicate llm outputs with minhash. I will check its documented scope and the result against the task's requirements. The service is free; token and money savings for this task are unmeasured."

## Supported

- cluster near duplicate llm outputs with minhash
- group almost identical responses together
- compute minhash signatures for a batch of texts
- estimate jaccard similarity with minhash
- reproduce a textbook minhash signature matrix
- collapse repeated agent answers into groups

## Not supported

- exact Jaccard values (use text-near-duplicates-find or text-shingle-similarity)
- locality-sensitive hashing bands or approximate candidate pruning (every pair is compared)
- random or time-seeded hash functions (all seeds are fixed)

## Behavior

- Shingling as text-shingle-similarity (lowercase default false, collapse_whitespace default true, shingle_unit default 'char', shingle_size default 5, sets).
- Element id x of a shingle: element_ids 'fnv1a32' (default) is FNV-1a 32-bit (RFC 9923: offset basis 0x811C9DC5, prime 0x01000193, one byte at a time) over the shingle's UTF-8 bytes, an unsigned value below 2^32; 'universe_row' is the 0-based row of the shingle in the characteristic matrix whose rows are the distinct shingles of all documents in Unicode code point order (MMDS 3.3.1).
- Hash functions h_i(x) = (a_i * x + b_i) mod m, computed exactly with non-negative integers (MMDS 3.3.5). Default m = 4294967311 (2^32 + 15, prime) and num_hashes = 64 (1 to 256) functions with a_i = 1 + (i * 2654435761 mod (m - 1)) and b_i = (i * 2246822519) mod m for i = 0 .. num_hashes - 1, so h_0(x) = x. Alternatively hash_functions gives explicit [{a, b}] (1 to 256, 1 <= a < m, 0 <= b < m) and modulus gives m (2 to 4294967311); num_hashes and hash_functions are exclusive.
- Signature component i of a document = min over its shingles of h_i(x); a document with no shingles gets m (a value no h_i can produce) in every component. matching_hashes = number of equal components; estimate = matching_hashes / num_hashes as a correctly rounded IEEE-754 double.
- A pair (a < b) qualifies when estimate >= threshold (optional, default 0.8, 0 to 1). clusters are the connected components of the qualifying pairs (single linkage), each listed in ascending index order, components ordered by their smallest index; every document is in exactly one cluster (singletons included).
- pairs lists qualifying pairs by matching_hashes descending, then a, then b; at most 1000 (pair_count, pairs_truncated). signatures is the full signature matrix (one row per document) when include_signatures is true (default false), else null.

## Input

- `documents` (array of string, required): min items 2; max items 200
- `threshold` (number, optional): min 0; max 1
- `num_hashes` (integer, optional): min 1; max 256
- `hash_functions` (array of object, optional): min items 1; max items 256
- `modulus` (integer, optional): min 2; max 4294967311
- `element_ids` (one of "fnv1a32", "universe_row", optional)
- `include_signatures` (boolean, optional)
- `shingle_unit` (one of "char", "word", optional)
- `shingle_size` (integer, optional): min 1; max 32
- `lowercase` (boolean, optional)
- `collapse_whitespace` (boolean, optional)

## Output

- `num_hashes` (integer, required): min 1; max 256
- `modulus` (integer, required): min 2; max 4294967311
- `clusters` (array of array of integer, required): each min items 1; each each min 0
- `pairs` (array of object, required): max items 1000
- `pair_count` (integer, required): min 0
- `pairs_truncated` (boolean, required)
- `shingle_counts` (array of integer, required): each min 0
- `signatures` (array or null, required)

## Limits

- max documents: 200
- max total bytes: 131072
- max shingle size: 32
- max hashes: 256
- max pairs: 1000
- max signature values: 16384

## Example

Request input:

```json
{
  "documents": [
    "a d",
    "c",
    "b d e",
    "a c d"
  ],
  "shingle_unit": "word",
  "shingle_size": 1,
  "element_ids": "universe_row",
  "modulus": 5,
  "hash_functions": [
    {
      "a": 1,
      "b": 1
    },
    {
      "a": 3,
      "b": 1
    }
  ],
  "threshold": 0.5,
  "include_signatures": true
}
```

Response:

```json
{
  "result": {
    "num_hashes": 2,
    "modulus": 5,
    "clusters": [
      [
        0,
        2,
        3
      ],
      [
        1
      ]
    ],
    "pairs": [
      {
        "a": 0,
        "b": 3,
        "matching_hashes": 2,
        "estimate": 1
      },
      {
        "a": 0,
        "b": 2,
        "matching_hashes": 1,
        "estimate": 0.5
      },
      {
        "a": 2,
        "b": 3,
        "matching_hashes": 1,
        "estimate": 0.5
      }
    ],
    "pair_count": 3,
    "pairs_truncated": false,
    "shingle_counts": [
      2,
      1,
      3,
      3
    ],
    "signatures": [
      [
        1,
        0
      ],
      [
        3,
        2
      ],
      [
        0,
        0
      ],
      [
        1,
        0
      ]
    ]
  }
}
```

## How to call it

### MCP

Connect `https://computefirst.net/mcp` ([setup](/docs#connect)), then call `execute` with:

```json
{
  "id": "text-minhash-cluster",
  "version": "1.0.0",
  "input": {
    "documents": [
      "a d",
      "c",
      "b d e",
      "a c d"
    ],
    "shingle_unit": "word",
    "shingle_size": 1,
    "element_ids": "universe_row",
    "modulus": 5,
    "hash_functions": [
      {
        "a": 1,
        "b": 1
      },
      {
        "a": 3,
        "b": 1
      }
    ],
    "threshold": 0.5,
    "include_signatures": true
  }
}
```

### HTTP (no key)

```sh
curl -X POST https://computefirst.net/v1/tools/text-minhash-cluster/versions/1.0.0/execute \
  -H "Content-Type: application/json" \
  -d '{"documents":["a d","c","b d e","a c d"],"shingle_unit":"word","shingle_size":1,"element_ids":"universe_row","modulus":5,"hash_functions":[{"a":1,"b":1},{"a":3,"b":1}],"threshold":0.5,"include_signatures":true}'
```

The machine-readable contract is at [/v1/tools/text-minhash-cluster/versions/1.0.0](/v1/tools/text-minhash-cluster/versions/1.0.0).

### CLI

```sh
node cli.mjs run text-minhash-cluster 1.0.0 --input input.json --base-url https://computefirst.net
```

Get the client at [/clients/cli/](/clients/cli/).

## Related tools

- [Text near duplicates find](/tools/text-near-duplicates-find): Find every pair of texts whose exact shingle Jaccard similarity reaches a threshold, in a fixed order.
- [Text shingle similarity](/tools/text-shingle-similarity): Exact shingle-set resemblance (Jaccard) and containment of two texts, with counts and shared shingles.
- [Citation record dedupe](/tools/citation-record-dedupe): Group duplicate CSL citation records by deterministic DOI or folded year|author|title match keys, in input order.
- [Text sentences chunk](/tools/text-sentences-chunk): Pack whole UAX #29 sentences into chunks of at most max_size code points, UTF-16 units or UTF-8 bytes.
- [Bleu sentence compute](/tools/bleu-sentence-compute): Compute sentence-level BLEU for one tokenized hypothesis with a chosen smoothing method (none or Chen-Cherry 1-3).
- [Text paragraphs split](/tools/text-paragraphs-split): Split text into paragraphs separated by blank lines (CommonMark line rules) with code point spans and line numbers.