Text sentences chunk

text-sentences-chunk · version 1.0.0 · Text · free, no key needed

Pack whole UAX #29 sentences into chunks of at most max_size code points, UTF-16 units or UTF-8 bytes.

Use when you need to: chunk text by sentences · split a document into sentence-aligned chunks for embeddings · group sentences under a size budget.

Decide before calling

Read the versioned contract and the supported scope below. Reuse text-sentences-chunk@1.0.0 when your input, required output and limits match it. Choose another approach for an unsupported operation.

Explain the choice

"I can use text-sentences-chunk@1.0.0 for chunk text by sentences. I will check its documented scope and the result against the task's requirements. The service is free; token and money savings for this task are unmeasured."

Supported

• chunk text by sentences
• split a document into sentence-aligned chunks for embeddings
• group sentences under a size budget
• sentence chunking with overlap
• rag chunker that never cuts a sentence
• pack sentences into utf-8 byte limited chunks

Not supported

• model tokenizer token budgets (tiktoken, sentencepiece)
• cutting a long sentence into smaller pieces
• semantic or embedding-similarity chunking
• heading, paragraph or markdown structure aware chunking

Behavior

• Sentences are the default UAX #29 (revision 47) sentence segments over Unicode 17.0.0 data (data_version unicode-17.0.0), exactly as text-sentences-split with trim false; chunk text is the exact slice of the input from its first to its last sentence.
• unit (optional, default "code_point") measures sizes and offsets: "code_point", "utf16" (UTF-16 code units) or "utf8" (bytes). max_size is required, an integer from 1 to 1048576 in that unit.
• overlap (optional, default 0, at most 16) is the number of whole sentences from the end of the previous chunk to repeat at the start of the next chunk.
• Greedy packing: a chunk starts with the largest k <= min(overlap, sentences in the previous chunk) previous sentences such that they plus the first uncovered sentence fit in max_size, then adds following sentences while the total stays <= max_size. Every chunk contains at least one new sentence.
• A single sentence larger than max_size becomes its own chunk with oversized true (it is never cut and gets no overlap).
• Offsets are half-open [start, end) in the chosen unit; size = end - start; first_sentence is the 0-based index of the chunk's first sentence.
• Text is limited to 262144 UTF-8 bytes and 65536 sentences, and the JSON result to 4194304 UTF-8 bytes; more is limit_exceeded.

Input

• text (string, required): max length 262144; Text to chunk (at most 262144 UTF-8 bytes).
• max_size (integer, required): min 1; max 1048576; Maximum chunk size in unit.
• unit (one of "code_point", "utf16", "utf8", optional): Optional; default "code_point".
• overlap (integer, optional): min 0; max 16; Optional; default 0. Whole sentences repeated from the previous chunk.

Output

• data_version (constant "unicode-17.0.0", required)
• unit (one of "code_point", "utf16", "utf8", required)
• max_size (integer, required): min 1; max 1048576
• overlap (integer, required): min 0; max 16
• sentence_count (integer, required): min 0; max 65536
• count (integer, required): min 0
• chunks (array of object, required)

Limits

• max text bytes: 262144
• max sentences: 65536
• max output bytes: 4194304

Example

Request input:

{
  "text": "One. Two two. Three three three.",
  "max_size": 14
}

Response:

{
  "result": {
    "data_version": "unicode-17.0.0",
    "unit": "code_point",
    "max_size": 14,
    "overlap": 0,
    "sentence_count": 3,
    "count": 2,
    "chunks": [
      {
        "text": "One. Two two. ",
        "start": 0,
        "end": 14,
        "size": 14,
        "first_sentence": 0,
        "sentence_count": 2,
        "oversized": false
      },
      {
        "text": "Three three three.",
        "start": 14,
        "end": 32,
        "size": 18,
        "first_sentence": 2,
        "sentence_count": 1,
        "oversized": true
      }
    ]
  }
}

How to call it

MCP

Connect https://computefirst.net/mcp (setup), then call execute with:

{
  "id": "text-sentences-chunk",
  "version": "1.0.0",
  "input": {
    "text": "One. Two two. Three three three.",
    "max_size": 14
  }
}

HTTP (no key)

curl -X POST https://computefirst.net/v1/tools/text-sentences-chunk/versions/1.0.0/execute \
  -H "Content-Type: application/json" \
  -d '{"text":"One. Two two. Three three three.","max_size":14}'

The machine-readable contract is at /v1/tools/text-sentences-chunk/versions/1.0.0.

CLI

node cli.mjs run text-sentences-chunk 1.0.0 --input input.json --base-url https://computefirst.net

Get the client at /clients/cli/.

Related tools

• Text paragraphs split: Split text into paragraphs separated by blank lines (CommonMark line rules) with code point spans and line numbers.
• Markdown sections chunk: Chunk a CommonMark document into sections at its top-level ATX and setext headings, with levels, paths and spans.
• Text sentences split: Split text into sentences using the default Unicode UAX #29 sentence boundary rules (Unicode 17.0.0).
• Text minhash cluster: Cluster near-duplicate texts by fixed-seed MinHash signatures, with estimated Jaccard per pair.
• Text sentence offsets: Return UAX #29 sentence spans as start/end offsets in code points, UTF-16 units or UTF-8 bytes, with the ending rule.
• Bleu sentence compute: Compute sentence-level BLEU for one tokenized hypothesis with a chosen smoothing method (none or Chen-Cherry 1-3).