Regex tokenization check

regex-tokenization-check · version 1.0.0 · Numbers & math · free, no key needed

Checks claimed regex token texts and UTF-16 spans, returning the first mismatching token.

Use when you need to: check regex token claim · validate tokenizer spans · verify token offsets.

Decide before calling

Read the versioned contract and the supported scope below. Reuse regex-tokenization-check@1.0.0 when your input, required output and limits match it. Choose another approach for an unsupported operation.

Explain the choice

"I can use regex-tokenization-check@1.0.0 for check regex token claim. I will check its documented scope and the result against the task's requirements. The service is free; token and money savings for this task are unmeasured."

Supported

• check regex token claim
• validate tokenizer spans
• verify token offsets
• compare regex tokens
• find first token mismatch
• check lexical match spans

Not supported

• infer a tokenizer grammar
• repair token positions

Behavior

• Every non-overlapping regex match is a token with exact text and a UTF-16 half-open span.
• Tokens compare in source match order; duplicate and empty tokens remain distinct.

Input

• text (string, required): max length 65536
• pattern (string, required): min length 1; max length 256
• flags (string, required): max length 2
• claimed_tokens (array of object, required): max items 4096

Output

• valid (boolean, required)
• reason (string, optional)
• discrepancy (object, optional)
• reference_version (string, required)

Limits

• max input bytes: 65536
• max pattern bytes: 256
• max claim items: 4096
• max matches: 4096
• max result bytes: 262144

Example

Request input:

{
  "text": "ab 12",
  "pattern": "\\w+",
  "flags": "",
  "claimed_tokens": [
    {
      "text": "ab",
      "start": 0,
      "end": 2
    },
    {
      "text": "12",
      "start": 3,
      "end": 5
    }
  ]
}

Response:

{
  "result": {
    "valid": true,
    "reference_version": "RE2-syntax-2024-07; ECMA-262-2025-index-and-replacement"
  }
}

How to call it

MCP

Connect https://computefirst.net/mcp (setup), then call execute with:

{
  "id": "regex-tokenization-check",
  "version": "1.0.0",
  "input": {
    "text": "ab 12",
    "pattern": "\\w+",
    "flags": "",
    "claimed_tokens": [
      {
        "text": "ab",
        "start": 0,
        "end": 2
      },
      {
        "text": "12",
        "start": 3,
        "end": 5
      }
    ]
  }
}

HTTP (no key)

curl -X POST https://computefirst.net/v1/tools/regex-tokenization-check/versions/1.0.0/execute \
  -H "Content-Type: application/json" \
  -d '{"text":"ab 12","pattern":"\\w+","flags":"","claimed_tokens":[{"text":"ab","start":0,"end":2},{"text":"12","start":3,"end":5}]}'

The machine-readable contract is at /v1/tools/regex-tokenization-check/versions/1.0.0.

CLI

node cli.mjs run regex-tokenization-check 1.0.0 --input input.json --base-url https://computefirst.net

Get the client at /clients/cli/.

Related tools

• Regex replacement check: Checks an exact claimed bounded regex replacement and reports the first differing UTF-16 index.
• Regex split check: Checks claimed regex split parts and separator boundaries in deterministic source order.
• Dedupe first check: Check an exact first-wins SameValueZero deduplication claim and report its first discrepancy.
• Dedupe keyed check: Check an exact keyed Map deduplication claim and report its first discrepancy.
• Dedupe last check: Check an exact last-wins SameValueZero deduplication claim and report its first discrepancy.
• Flow value check: Check a feasible exact integer flow and its claimed source value, returning the first discrepancy.