Text shingle similarity

text-shingle-similarity · version 1.0.0 · Text · free, no key needed

Exact shingle-set resemblance (Jaccard) and containment of two texts, with counts and shared shingles.

Use when you need to: jaccard similarity of two texts using shingles · how similar are these two answers word for word · compute broder resemblance and containment.

Decide before calling

Read the versioned contract and the supported scope below. Reuse text-shingle-similarity@1.0.0 when your input, required output and limits match it. Choose another approach for an unsupported operation.

Explain the choice

"I can use text-shingle-similarity@1.0.0 for jaccard similarity of two texts using shingles. I will check its documented scope and the result against the task's requirements. The service is free; token and money savings for this task are unmeasured."

Supported

• jaccard similarity of two texts using shingles
• how similar are these two answers word for word
• compute broder resemblance and containment
• is text a mostly contained in text b
• compare two documents with k-shingles
• shared word n-grams between two passages
• measure near-duplicate overlap of two strings

Not supported

• semantic or embedding similarity (only literal shingle overlap)
• MinHash estimates for many documents (use text-minhash-cluster)
• bag (multiset) shingling with repeated shingles counted (sets only)
• edit distance or diffs (use the text diff tools)

Behavior

• Normalization (optional fields): lowercase (default false) applies String.prototype.toLowerCase (Unicode case mapping of the runtime, locale-independent); collapse_whitespace (default true) replaces each run of Unicode White_Space code points with one U+0020 and trims both ends.
• shingle_unit (default 'char'): 'char' uses Unicode code points, 'word' uses maximal runs of non-White_Space code points. shingle_size k (default 5, 1 to 32). The shingle set holds every run of k consecutive units (Broder 1997 option B: a set, duplicates counted once); a text with 1 to k-1 units has one shingle made of all its units; a text with no units has the empty set. Word shingles join words with one U+0020.
• intersection = |A ∩ B|, union = |A ∪ B|; resemblance = intersection / union (Broder r(A,B), MMDS Jaccard similarity), and 1 when both sets are empty (r(A,A) = 1). containment_a_in_b = intersection / |A| (Broder c(A,B)) and 1 when A is empty (an empty text is a contiguous subsequence of any text); containment_b_in_a likewise.
• Ratios are IEEE-754 doubles from one correctly rounded division of the exact integer counts, which are also returned.
• shared_shingles lists the intersection in Unicode code point order, at most max_shared (optional, default 10, 0 to 100) entries; shared_truncated is true when more exist.

Input

• text_a (string, required): max length 65536
• text_b (string, required): max length 65536
• shingle_unit (one of "char", "word", optional)
• shingle_size (integer, optional): min 1; max 32
• lowercase (boolean, optional)
• collapse_whitespace (boolean, optional)
• max_shared (integer, optional): min 0; max 100

Output

• shingles_a (integer, required): min 0
• shingles_b (integer, required): min 0
• intersection (integer, required): min 0
• union (integer, required): min 0
• resemblance (number, required): min 0; max 1
• containment_a_in_b (number, required): min 0; max 1
• containment_b_in_a (number, required): min 0; max 1
• shared_shingles (array of string, required): max items 100
• shared_truncated (boolean, required)

Limits

• max text bytes: 65536
• max shingle size: 32
• max shared: 100

Example

Request input:

{
  "text_a": "a rose is a rose is a rose",
  "text_b": "a rose is a flower which is a rose",
  "shingle_unit": "word",
  "shingle_size": 2
}

Response:

{
  "result": {
    "shingles_a": 3,
    "shingles_b": 6,
    "intersection": 3,
    "union": 6,
    "resemblance": 0.5,
    "containment_a_in_b": 1,
    "containment_b_in_a": 0.5,
    "shared_shingles": [
      "a rose",
      "is a",
      "rose is"
    ],
    "shared_truncated": false
  }
}

How to call it

MCP

Connect https://computefirst.net/mcp (setup), then call execute with:

{
  "id": "text-shingle-similarity",
  "version": "1.0.0",
  "input": {
    "text_a": "a rose is a rose is a rose",
    "text_b": "a rose is a flower which is a rose",
    "shingle_unit": "word",
    "shingle_size": 2
  }
}

HTTP (no key)

curl -X POST https://computefirst.net/v1/tools/text-shingle-similarity/versions/1.0.0/execute \
  -H "Content-Type: application/json" \
  -d '{"text_a":"a rose is a rose is a rose","text_b":"a rose is a flower which is a rose","shingle_unit":"word","shingle_size":2}'

The machine-readable contract is at /v1/tools/text-shingle-similarity/versions/1.0.0.

CLI

node cli.mjs run text-shingle-similarity 1.0.0 --input input.json --base-url https://computefirst.net

Get the client at /clients/cli/.

Related tools

• Text near duplicates find: Find every pair of texts whose exact shingle Jaccard similarity reaches a threshold, in a fixed order.
• Text minhash cluster: Cluster near-duplicate texts by fixed-seed MinHash signatures, with estimated Jaccard per pair.
• Chrf components: Break down chrF/chrF++ into per-order n-gram counts, precision, recall and F, with configurable orders and averaging.
• Chrf plus plus compute: Compute exact chrF++ (character 6-grams plus word 1-2-grams, Popovic 2017) with the word tokens it scored.
• Citation record dedupe: Group duplicate CSL citation records by deterministic DOI or folded year|author|title match keys, in input order.
• Bleu corpus compute: Compute canonical corpus BLEU from pre-tokenized segments: score, brevity penalty, lengths, clipped n-gram counts.