Before counting tokens precisely, a pipeline uses a rough tokenizer: lowercase the text, then emit every run of letters/digits as one token and every other non-space character as its own token. Whitespace separates tokens and is not emitted.
Example: "Hello, world!!" → ["hello", ",", "world", "!", "!"].
Implement tokenize. Input: text: str. Returns: list[str].
def tokenize(text: str) -> list[str]:
return text.split()
Run executes the public tests. Submit grades against public and hidden tests.
Before counting tokens precisely, a pipeline uses a rough tokenizer: lowercase the text, then emit every run of letters/digits as one token and every other non-space character as its own token. Whitespace separates tokens and is not emitted.
Example: "Hello, world!!" → ["hello", ",", "world", "!", "!"].
Implement tokenize. Input: text: str. Returns: list[str].
def tokenize(text: str) -> list[str]:
return text.split()
Run executes the public tests. Submit grades against public and hidden tests.