Text near duplicates find

text-near-duplicates-find · version 1.0.0 · Text · free, no key needed

Find every pair of texts whose exact shingle Jaccard similarity reaches a threshold, in a fixed order.

Use when you need to: find near duplicate agent outputs · which of these answers are almost the same · detect near-duplicate documents with exact jaccard.

Decide before calling

Read the versioned contract and the supported scope below. Reuse text-near-duplicates-find@1.0.0 when your input, required output and limits match it. Choose another approach for an unsupported operation.

Explain the choice

"I can use text-near-duplicates-find@1.0.0 for find near duplicate agent outputs. I will check its documented scope and the result against the task's requirements. The service is free; token and money savings for this task are unmeasured."

Supported

• find near duplicate agent outputs
• which of these answers are almost the same
• detect near-duplicate documents with exact jaccard
• list pairs of similar texts above a threshold
• deduplicate model responses by shingle overlap
• find repeated paragraphs across many outputs

Not supported

• approximate MinHash estimates or clustering (use text-minhash-cluster)
• semantic paraphrase detection (only literal shingle overlap)
• comparing one pair in detail with containment and shared shingles (use text-shingle-similarity)

Behavior

• Shingling is identical to text-shingle-similarity: optional lowercase (default false), collapse_whitespace (default true), shingle_unit (default 'char') and shingle_size (default 5); sets, not bags; a text shorter than k units has one shingle; an empty text has the empty set.
• Every pair (a, b) with a < b (0-based document indexes) is compared exactly: intersection = |A ∩ B|, union = |A ∪ B|, resemblance = intersection / union as a correctly rounded IEEE-754 double, and 1 when both sets are empty.
• A pair is reported when resemblance >= threshold (optional, default 0.8, 0 to 1), comparing the double resemblance with the double threshold.
• Order: higher exact resemblance first (fractions compared by cross-multiplication, not by rounded doubles), then smaller a, then smaller b. At most 1000 pairs are returned; pair_count is the number that qualified and pairs_truncated is true when pairs were cut.
• shingle_counts[i] is the number of distinct shingles of document i; pairs_compared is n(n-1)/2.

Input

• documents (array of string, required): min items 2; max items 200
• threshold (number, optional): min 0; max 1
• shingle_unit (one of "char", "word", optional)
• shingle_size (integer, optional): min 1; max 32
• lowercase (boolean, optional)
• collapse_whitespace (boolean, optional)

Output

• pairs (array of object, required): max items 1000
• pair_count (integer, required): min 0
• pairs_truncated (boolean, required)
• pairs_compared (integer, required): min 1
• shingle_counts (array of integer, required): each min 0

Limits

• max documents: 200
• max total bytes: 131072
• max shingle size: 32
• max pairs: 1000

Example

Request input:

{
  "documents": [
    "a d",
    "b d e",
    "a c d"
  ],
  "shingle_unit": "word",
  "shingle_size": 1,
  "threshold": 0.25
}

Response:

{
  "result": {
    "pairs": [
      {
        "a": 0,
        "b": 2,
        "resemblance": 0.6666666666666666,
        "intersection": 2,
        "union": 3
      },
      {
        "a": 0,
        "b": 1,
        "resemblance": 0.25,
        "intersection": 1,
        "union": 4
      }
    ],
    "pair_count": 2,
    "pairs_truncated": false,
    "pairs_compared": 3,
    "shingle_counts": [
      2,
      3,
      3
    ]
  }
}

How to call it

MCP

Connect https://computefirst.net/mcp (setup), then call execute with:

{
  "id": "text-near-duplicates-find",
  "version": "1.0.0",
  "input": {
    "documents": [
      "a d",
      "b d e",
      "a c d"
    ],
    "shingle_unit": "word",
    "shingle_size": 1,
    "threshold": 0.25
  }
}

HTTP (no key)

curl -X POST https://computefirst.net/v1/tools/text-near-duplicates-find/versions/1.0.0/execute \
  -H "Content-Type: application/json" \
  -d '{"documents":["a d","b d e","a c d"],"shingle_unit":"word","shingle_size":1,"threshold":0.25}'

The machine-readable contract is at /v1/tools/text-near-duplicates-find/versions/1.0.0.

CLI

node cli.mjs run text-near-duplicates-find 1.0.0 --input input.json --base-url https://computefirst.net

Get the client at /clients/cli/.

Related tools

• Text shingle similarity: Exact shingle-set resemblance (Jaccard) and containment of two texts, with counts and shared shingles.
• Text minhash cluster: Cluster near-duplicate texts by fixed-seed MinHash signatures, with estimated Jaccard per pair.
• Citation record dedupe: Group duplicate CSL citation records by deterministic DOI or folded year|author|title match keys, in input order.
• Text paragraphs split: Split text into paragraphs separated by blank lines (CommonMark line rules) with code point spans and line numbers.
• Citation batch check: Check an ordered batch of citations (span+quote or quote anchors) against supplied sources; returns the first failure.
• Markdown sections chunk: Chunk a CommonMark document into sections at its top-level ATX and setext headings, with levels, paths and spans.