# Citation record dedupe

`citation-record-dedupe` · version 1.0.0 · Text · free, no key needed

Group duplicate CSL citation records by deterministic DOI or folded year|author|title match keys, in input order.

**Use when you need to: deduplicate a list of CSL-JSON citation records · find duplicate references in a bibliography by DOI · group bibliography entries with the same year author and title.**

## Decide before calling

Read the [versioned contract](/v1/tools/citation-record-dedupe/versions/1.0.0) and the supported scope below. Reuse `citation-record-dedupe@1.0.0` when your input, required output and limits match it. Choose another approach for an unsupported operation.

## Explain the choice

"I can use `citation-record-dedupe@1.0.0` for deduplicate a list of CSL-JSON citation records. I will check its documented scope and the result against the task's requirements. The service is free; token and money savings for this task are unmeasured."

## Supported

- deduplicate a list of CSL-JSON citation records
- find duplicate references in a bibliography by DOI
- group bibliography entries with the same year author and title
- merge candidates for duplicate citations without network lookup
- count duplicate records in a reference list
- איתור כפילויות ברשימת מקורות ביבליוגרפית

## Not supported

- merging the duplicate records into one record
- fuzzy or similarity-based duplicate detection
- resolving DOIs or checking metadata against Crossref
- more than 200 records per call

## Behavior

- Every record is validated and normalized exactly as citation-record-normalize does, and keyed exactly as citation-record-key does; an error names the record by JSON Pointer (/records/<index>/...).
- match_on (optional) is doi_then_metadata by default: a record with a DOI is keyed by doi_key, otherwise by metadata_key. metadata_only ignores DOIs and keys every record by metadata_key. A record with no usable key forms its own group with match_key null and key_type none and never joins another group.
- Records with equal keys (exact string equality) form one group. Groups are listed in order of their first record; indices (0-based) and ids within a group are ascending in input order. duplicate_count = record_count - group_count.
- doi_key = "doi:" + the normalized DOI with ASCII a-z upper-cased and every other character unchanged (DOI Handbook section 3.3.4: DOI names are case-insensitive for Basic Latin letters only); null without a DOI.
- Folding for keys: NFKC; String.prototype.toLowerCase (Unicode default full lowercase mapping with Final_Sigma; not CaseFolding.txt, so U+00DF stays); NFKD; remove every p{M} mark; replace each run of code points that are not p{L} or p{N} with one space; trim; NFC. Unicode tables come from the runtime.
- metadata_key = "meta:" + year + "|" + first_author + "|" + title: year is issued.date_parts[0][0] as a base-10 string (empty if absent); first_author is the fold of the first remaining author name's family, else literal, else given (empty if absent); title is the fold of title. metadata_key is null when there is no title or it folds to empty. Every key is at most 2060 code points.
- Limits: input JSON at most 262144 UTF-8 bytes, at most 200 records (at least 1), and the per-record limits of citation-record-normalize: every string at most 1024 code points before and after NFC, NFKC or folding (limit_exceeded), at most 100 names per name variable.
- Grouping is a transparent deterministic heuristic, not bibliographic verification; review groups before deleting records.

## Input

- `records` (array of object, required): min items 1; max items 200
- `match_on` (one of "doi_then_metadata", "metadata_only", optional)

## Output

- `match_on` (one of "doi_then_metadata", "metadata_only", required)
- `record_count` (integer, required): min 1; max 200
- `group_count` (integer, required): min 1; max 200
- `duplicate_count` (integer, required): min 0; max 199
- `groups` (array of object, required): min items 1; max items 200

## Limits

- max input bytes: 262144
- max records: 200
- max field chars: 1024
- max names: 100

## Example

Request input:

```json
{
  "records": [
    {
      "id": "a",
      "type": "article-journal",
      "doi": "10.123/ABC"
    },
    {
      "id": "b",
      "type": "article-journal",
      "doi": "10.123/abc"
    },
    {
      "id": "c",
      "type": "article-journal",
      "doi": "https://doi.org/10.123/AbC"
    }
  ]
}
```

Response:

```json
{
  "result": {
    "match_on": "doi_then_metadata",
    "record_count": 3,
    "group_count": 1,
    "duplicate_count": 2,
    "groups": [
      {
        "match_key": "doi:10.123/ABC",
        "key_type": "doi",
        "indices": [
          0,
          1,
          2
        ],
        "ids": [
          "a",
          "b",
          "c"
        ]
      }
    ]
  }
}
```

## How to call it

### MCP

Connect `https://computefirst.net/mcp` ([setup](/docs#connect)), then call `execute` with:

```json
{
  "id": "citation-record-dedupe",
  "version": "1.0.0",
  "input": {
    "records": [
      {
        "id": "a",
        "type": "article-journal",
        "doi": "10.123/ABC"
      },
      {
        "id": "b",
        "type": "article-journal",
        "doi": "10.123/abc"
      },
      {
        "id": "c",
        "type": "article-journal",
        "doi": "https://doi.org/10.123/AbC"
      }
    ]
  }
}
```

### HTTP (no key)

```sh
curl -X POST https://computefirst.net/v1/tools/citation-record-dedupe/versions/1.0.0/execute \
  -H "Content-Type: application/json" \
  -d '{"records":[{"id":"a","type":"article-journal","doi":"10.123/ABC"},{"id":"b","type":"article-journal","doi":"10.123/abc"},{"id":"c","type":"article-journal","doi":"https://doi.org/10.123/AbC"}]}'
```

The machine-readable contract is at [/v1/tools/citation-record-dedupe/versions/1.0.0](/v1/tools/citation-record-dedupe/versions/1.0.0).

### CLI

```sh
node cli.mjs run citation-record-dedupe 1.0.0 --input input.json --base-url https://computefirst.net
```

Get the client at [/clients/cli/](/clients/cli/).

## Related tools

- [Citation record key](/tools/citation-record-key): Compute transparent deterministic match keys (DOI key and folded year|author|title key) for one CSL citation record.
- [Citation record normalize](/tools/citation-record-normalize): Normalize one CSL 1.0.2 citation record: NFC, whitespace, empty values, DOI wrapper, page dashes, dates; lists changes.
- [Text near duplicates find](/tools/text-near-duplicates-find): Find every pair of texts whose exact shingle Jaccard similarity reaches a threshold, in a fixed order.
- [Bleu ngrams count](/tools/bleu-ngrams-count): Count BLEU clipped n-gram matches and totals per order for tokenized segments, with optional clipping tables.
- [Citation batch check](/tools/citation-batch-check): Check an ordered batch of citations (span+quote or quote anchors) against supplied sources; returns the first failure.
- [Text shingle similarity](/tools/text-shingle-similarity): Exact shingle-set resemblance (Jaccard) and containment of two texts, with counts and shared shingles.