truncate¶
- torchtune.data.truncate(tokens: List[Any], max_seq_len: int, eos_id: Optional[Any] = None) List[Any][source]¶
Truncate a list of tokens to a maximum length. If eos_id is provided, the last token will be replaced with eos_id.
- Parameters:
tokens (List[Any]) – list of tokens to truncate
max_seq_len (int) – maximum length of the list
eos_id (Optional[Any]) – token to replace the last token with. If None, the last token will not be replaced. Default is None.
- Returns:
truncated list of tokens
- Return type:
List[Any]
