truncate

torchtune.data.truncate(tokens: List[Any], max_seq_len: int, eos_id: Optional[Any] = None, truncation_type: str = 'right') → List[Any]

Truncate a list of tokens to a maximum length. If eos_id is provided, the last token will be replaced with eos_id.

Parameters:

• tokens (List[Any]) – list of tokens to truncate

• max_seq_len (int) – maximum length of the list

• eos_id (Optional[Any]) – token to replace the last token with. If None, the last token will not be replaced. Default is None.

• truncation_type (str) – type of truncation to apply, either “left” or “right”. Default is “right”.

Returns:

truncated list of tokens

Return type:

List[Any]

Raises:

ValueError – if truncation_type is not “left” or “right”