{"id":"text-sentences-chunk","version":"1.0.0","description":"Pack whole UAX #29 sentences into chunks of at most max_size code points, UTF-16 units or UTF-8 bytes.","supported_operations":["chunk text by sentences","split a document into sentence-aligned chunks for embeddings","group sentences under a size budget","sentence chunking with overlap","rag chunker that never cuts a sentence","pack sentences into utf-8 byte limited chunks"],"unsupported_operations":["model tokenizer token budgets (tiktoken, sentencepiece)","cutting a long sentence into smaller pieces","semantic or embedding-similarity chunking","heading, paragraph or markdown structure aware chunking"],"semantics":["Sentences are the default UAX #29 (revision 47) sentence segments over Unicode 17.0.0 data (data_version unicode-17.0.0), exactly as text-sentences-split with trim false; chunk text is the exact slice of the input from its first to its last sentence.","unit (optional, default \"code_point\") measures sizes and offsets: \"code_point\", \"utf16\" (UTF-16 code units) or \"utf8\" (bytes). max_size is required, an integer from 1 to 1048576 in that unit.","overlap (optional, default 0, at most 16) is the number of whole sentences from the end of the previous chunk to repeat at the start of the next chunk.","Greedy packing: a chunk starts with the largest k <= min(overlap, sentences in the previous chunk) previous sentences such that they plus the first uncovered sentence fit in max_size, then adds following sentences while the total stays <= max_size. Every chunk contains at least one new sentence.","A single sentence larger than max_size becomes its own chunk with oversized true (it is never cut and gets no overlap).","Offsets are half-open [start, end) in the chosen unit; size = end - start; first_sentence is the 0-based index of the chunk's first sentence.","Text is limited to 262144 UTF-8 bytes and 65536 sentences, and the JSON result to 4194304 UTF-8 bytes; more is limit_exceeded."],"limits":{"max_text_bytes":262144,"max_sentences":65536,"max_output_bytes":4194304},"pricing":{"status":"unpriced","charge_usd":null},"input_schema":{"type":"object","additionalProperties":false,"required":["text","max_size"],"properties":{"text":{"type":"string","maxLength":262144,"description":"Text to chunk (at most 262144 UTF-8 bytes)."},"max_size":{"type":"integer","minimum":1,"maximum":1048576,"description":"Maximum chunk size in unit."},"unit":{"type":"string","enum":["code_point","utf16","utf8"],"description":"Optional; default \"code_point\"."},"overlap":{"type":"integer","minimum":0,"maximum":16,"description":"Optional; default 0. Whole sentences repeated from the previous chunk."}}},"output_schema":{"type":"object","additionalProperties":false,"required":["data_version","unit","max_size","overlap","sentence_count","count","chunks"],"properties":{"data_version":{"type":"string","const":"unicode-17.0.0"},"unit":{"type":"string","enum":["code_point","utf16","utf8"]},"max_size":{"type":"integer","minimum":1,"maximum":1048576},"overlap":{"type":"integer","minimum":0,"maximum":16},"sentence_count":{"type":"integer","minimum":0,"maximum":65536},"count":{"type":"integer","minimum":0},"chunks":{"type":"array","items":{"type":"object","additionalProperties":false,"required":["text","start","end","size","first_sentence","sentence_count","oversized"],"properties":{"text":{"type":"string","minLength":1},"start":{"type":"integer","minimum":0},"end":{"type":"integer","minimum":1},"size":{"type":"integer","minimum":1},"first_sentence":{"type":"integer","minimum":0},"sentence_count":{"type":"integer","minimum":1},"oversized":{"type":"boolean"}}}}}},"examples":[{"input":{"text":"One. Two two. Three three three.","max_size":14},"output":{"data_version":"unicode-17.0.0","unit":"code_point","max_size":14,"overlap":0,"sentence_count":3,"count":2,"chunks":[{"text":"One. Two two. ","start":0,"end":14,"size":14,"first_sentence":0,"sentence_count":2,"oversized":false},{"text":"Three three three.","start":14,"end":32,"size":18,"first_sentence":2,"sentence_count":1,"oversized":true}]}},{"input":{"text":"A1. B2. C3. D4.","max_size":8,"unit":"utf8","overlap":1},"output":{"data_version":"unicode-17.0.0","unit":"utf8","max_size":8,"overlap":1,"sentence_count":4,"count":3,"chunks":[{"text":"A1. B2. ","start":0,"end":8,"size":8,"first_sentence":0,"sentence_count":2,"oversized":false},{"text":"B2. C3. ","start":4,"end":12,"size":8,"first_sentence":1,"sentence_count":2,"oversized":false},{"text":"C3. D4.","start":8,"end":15,"size":7,"first_sentence":2,"sentence_count":2,"oversized":false}]}}],"execute_url":"/v1/tools/text-sentences-chunk/versions/1.0.0/execute"}