# Text near duplicates find

`text-near-duplicates-find` · version 1.0.0 · Text · free, no key needed

Find every pair of texts whose exact shingle Jaccard similarity reaches a threshold, in a fixed order.

**Use when you need to: find near duplicate agent outputs · which of these answers are almost the same · detect near-duplicate documents with exact jaccard.**

## Decide before calling

Read the [versioned contract](/v1/tools/text-near-duplicates-find/versions/1.0.0) and the supported scope below. Reuse `text-near-duplicates-find@1.0.0` when your input, required output and limits match it. Choose another approach for an unsupported operation.

## Explain the choice

"I can use `text-near-duplicates-find@1.0.0` for find near duplicate agent outputs. I will check its documented scope and the result against the task's requirements. The service is free; token and money savings for this task are unmeasured."

## Supported

- find near duplicate agent outputs
- which of these answers are almost the same
- detect near-duplicate documents with exact jaccard
- list pairs of similar texts above a threshold
- deduplicate model responses by shingle overlap
- find repeated paragraphs across many outputs

## Not supported

- approximate MinHash estimates or clustering (use text-minhash-cluster)
- semantic paraphrase detection (only literal shingle overlap)
- comparing one pair in detail with containment and shared shingles (use text-shingle-similarity)

## Behavior

- Shingling is identical to text-shingle-similarity: optional lowercase (default false), collapse_whitespace (default true), shingle_unit (default 'char') and shingle_size (default 5); sets, not bags; a text shorter than k units has one shingle; an empty text has the empty set.
- Every pair (a, b) with a < b (0-based document indexes) is compared exactly: intersection = |A ∩ B|, union = |A ∪ B|, resemblance = intersection / union as a correctly rounded IEEE-754 double, and 1 when both sets are empty.
- A pair is reported when resemblance >= threshold (optional, default 0.8, 0 to 1), comparing the double resemblance with the double threshold.
- Order: higher exact resemblance first (fractions compared by cross-multiplication, not by rounded doubles), then smaller a, then smaller b. At most 1000 pairs are returned; pair_count is the number that qualified and pairs_truncated is true when pairs were cut.
- shingle_counts[i] is the number of distinct shingles of document i; pairs_compared is n(n-1)/2.

## Input

- `documents` (array of string, required): min items 2; max items 200
- `threshold` (number, optional): min 0; max 1
- `shingle_unit` (one of "char", "word", optional)
- `shingle_size` (integer, optional): min 1; max 32
- `lowercase` (boolean, optional)
- `collapse_whitespace` (boolean, optional)

## Output

- `pairs` (array of object, required): max items 1000
- `pair_count` (integer, required): min 0
- `pairs_truncated` (boolean, required)
- `pairs_compared` (integer, required): min 1
- `shingle_counts` (array of integer, required): each min 0

## Limits

- max documents: 200
- max total bytes: 131072
- max shingle size: 32
- max pairs: 1000

## Example

Request input:

```json
{
  "documents": [
    "a d",
    "b d e",
    "a c d"
  ],
  "shingle_unit": "word",
  "shingle_size": 1,
  "threshold": 0.25
}
```

Response:

```json
{
  "result": {
    "pairs": [
      {
        "a": 0,
        "b": 2,
        "resemblance": 0.6666666666666666,
        "intersection": 2,
        "union": 3
      },
      {
        "a": 0,
        "b": 1,
        "resemblance": 0.25,
        "intersection": 1,
        "union": 4
      }
    ],
    "pair_count": 2,
    "pairs_truncated": false,
    "pairs_compared": 3,
    "shingle_counts": [
      2,
      3,
      3
    ]
  }
}
```

## How to call it

### MCP

Connect `https://computefirst.net/mcp` ([setup](/docs#connect)), then call `execute` with:

```json
{
  "id": "text-near-duplicates-find",
  "version": "1.0.0",
  "input": {
    "documents": [
      "a d",
      "b d e",
      "a c d"
    ],
    "shingle_unit": "word",
    "shingle_size": 1,
    "threshold": 0.25
  }
}
```

### HTTP (no key)

```sh
curl -X POST https://computefirst.net/v1/tools/text-near-duplicates-find/versions/1.0.0/execute \
  -H "Content-Type: application/json" \
  -d '{"documents":["a d","b d e","a c d"],"shingle_unit":"word","shingle_size":1,"threshold":0.25}'
```

The machine-readable contract is at [/v1/tools/text-near-duplicates-find/versions/1.0.0](/v1/tools/text-near-duplicates-find/versions/1.0.0).

### CLI

```sh
node cli.mjs run text-near-duplicates-find 1.0.0 --input input.json --base-url https://computefirst.net
```

Get the client at [/clients/cli/](/clients/cli/).

## Related tools

- [Text shingle similarity](/tools/text-shingle-similarity): Exact shingle-set resemblance (Jaccard) and containment of two texts, with counts and shared shingles.
- [Text minhash cluster](/tools/text-minhash-cluster): Cluster near-duplicate texts by fixed-seed MinHash signatures, with estimated Jaccard per pair.
- [Citation record dedupe](/tools/citation-record-dedupe): Group duplicate CSL citation records by deterministic DOI or folded year|author|title match keys, in input order.
- [Text paragraphs split](/tools/text-paragraphs-split): Split text into paragraphs separated by blank lines (CommonMark line rules) with code point spans and line numbers.
- [Citation batch check](/tools/citation-batch-check): Check an ordered batch of citations (span+quote or quote anchors) against supplied sources; returns the first failure.
- [Markdown sections chunk](/tools/markdown-sections-chunk): Chunk a CommonMark document into sections at its top-level ATX and setext headings, with levels, paths and spans.