# Regex tokenization check

`regex-tokenization-check` · version 1.0.0 · Numbers & math · free, no key needed

Checks claimed regex token texts and UTF-16 spans, returning the first mismatching token.

**Use when you need to: check regex token claim · validate tokenizer spans · verify token offsets.**

## Decide before calling

Read the [versioned contract](/v1/tools/regex-tokenization-check/versions/1.0.0) and the supported scope below. Reuse `regex-tokenization-check@1.0.0` when your input, required output and limits match it. Choose another approach for an unsupported operation.

## Explain the choice

"I can use `regex-tokenization-check@1.0.0` for check regex token claim. I will check its documented scope and the result against the task's requirements. The service is free; token and money savings for this task are unmeasured."

## Supported

- check regex token claim
- validate tokenizer spans
- verify token offsets
- compare regex tokens
- find first token mismatch
- check lexical match spans

## Not supported

- infer a tokenizer grammar
- repair token positions

## Behavior

- Every non-overlapping regex match is a token with exact text and a UTF-16 half-open span.
- Tokens compare in source match order; duplicate and empty tokens remain distinct.

## Input

- `text` (string, required): max length 65536
- `pattern` (string, required): min length 1; max length 256
- `flags` (string, required): max length 2
- `claimed_tokens` (array of object, required): max items 4096

## Output

- `valid` (boolean, required)
- `reason` (string, optional)
- `discrepancy` (object, optional)
- `reference_version` (string, required)

## Limits

- max input bytes: 65536
- max pattern bytes: 256
- max claim items: 4096
- max matches: 4096
- max result bytes: 262144

## Example

Request input:

```json
{
  "text": "ab 12",
  "pattern": "\\w+",
  "flags": "",
  "claimed_tokens": [
    {
      "text": "ab",
      "start": 0,
      "end": 2
    },
    {
      "text": "12",
      "start": 3,
      "end": 5
    }
  ]
}
```

Response:

```json
{
  "result": {
    "valid": true,
    "reference_version": "RE2-syntax-2024-07; ECMA-262-2025-index-and-replacement"
  }
}
```

## How to call it

### MCP

Connect `https://computefirst.net/mcp` ([setup](/docs#connect)), then call `execute` with:

```json
{
  "id": "regex-tokenization-check",
  "version": "1.0.0",
  "input": {
    "text": "ab 12",
    "pattern": "\\w+",
    "flags": "",
    "claimed_tokens": [
      {
        "text": "ab",
        "start": 0,
        "end": 2
      },
      {
        "text": "12",
        "start": 3,
        "end": 5
      }
    ]
  }
}
```

### HTTP (no key)

```sh
curl -X POST https://computefirst.net/v1/tools/regex-tokenization-check/versions/1.0.0/execute \
  -H "Content-Type: application/json" \
  -d '{"text":"ab 12","pattern":"\\w+","flags":"","claimed_tokens":[{"text":"ab","start":0,"end":2},{"text":"12","start":3,"end":5}]}'
```

The machine-readable contract is at [/v1/tools/regex-tokenization-check/versions/1.0.0](/v1/tools/regex-tokenization-check/versions/1.0.0).

### CLI

```sh
node cli.mjs run regex-tokenization-check 1.0.0 --input input.json --base-url https://computefirst.net
```

Get the client at [/clients/cli/](/clients/cli/).

## Related tools

- [Regex replacement check](/tools/regex-replacement-check): Checks an exact claimed bounded regex replacement and reports the first differing UTF-16 index.
- [Regex split check](/tools/regex-split-check): Checks claimed regex split parts and separator boundaries in deterministic source order.
- [Dedupe first check](/tools/dedupe-first-check): Check an exact first-wins SameValueZero deduplication claim and report its first discrepancy.
- [Dedupe keyed check](/tools/dedupe-keyed-check): Check an exact keyed Map deduplication claim and report its first discrepancy.
- [Dedupe last check](/tools/dedupe-last-check): Check an exact last-wins SameValueZero deduplication claim and report its first discrepancy.
- [Flow value check](/tools/flow-value-check): Check a feasible exact integer flow and its claimed source value, returning the first discrepancy.