Class TextChunker

Namespace

JD.AI.Core.Memory

Assembly

JD.AI.Core.dll

Splits text into overlapping chunks suitable for embedding. Supports configurable chunk size and overlap to maintain context across chunk boundaries.

public static class TextChunker

Inheritance

object

TextChunker

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Fields

DefaultMaxChunkChars

Default maximum tokens per chunk (approximate, using char-based estimation).

public const int DefaultMaxChunkChars = 1500

Field Value

int

DefaultOverlapChars

Default overlap between consecutive chunks.

public const int DefaultOverlapChars = 200

Field Value

int

Methods

Chunk(string, int, int)

Splits text into chunks with overlap, respecting paragraph and sentence boundaries.

public static IReadOnlyList<TextChunk> Chunk(string text, int maxChunkChars = 1500, int overlapChars = 200)

Parameters

text string

The text to chunk.

maxChunkChars int

Maximum characters per chunk.

overlapChars int

Number of characters to overlap between chunks.

Returns

IReadOnlyList<TextChunk>

List of text chunks with metadata.