Retrieval pipelines split documents into overlapping chunks so that context isn't lost at boundaries.
Given a list of tokens, a chunk size and an overlap (0 ≤ overlap < size), return chunks where each
chunk starts size - overlap tokens after the previous one. Stop when a chunk reaches the end of the
tokens — don't emit a final chunk that is entirely contained in the previous one.
Return a list of {"start": index, "tokens": [...]}.
Implement chunk_tokens. Input: tokens: list[str], size: int, overlap: int. Returns: list[dict].
def chunk_tokens(tokens: list[str], size: int, overlap: int) -> list[dict]:
return [{"start": 0, "tokens": tokens}]
Run executes the public tests. Submit grades against public and hidden tests.
Retrieval pipelines split documents into overlapping chunks so that context isn't lost at boundaries.
Given a list of tokens, a chunk size and an overlap (0 ≤ overlap < size), return chunks where each
chunk starts size - overlap tokens after the previous one. Stop when a chunk reaches the end of the
tokens — don't emit a final chunk that is entirely contained in the previous one.
Return a list of {"start": index, "tokens": [...]}.
Implement chunk_tokens. Input: tokens: list[str], size: int, overlap: int. Returns: list[dict].
def chunk_tokens(tokens: list[str], size: int, overlap: int) -> list[dict]:
return [{"start": 0, "tokens": tokens}]
Run executes the public tests. Submit grades against public and hidden tests.