Shortcuts

truncate

torchtune.data.truncate(tokens: List[Any], max_seq_len: int, eos_id: Optional[Any] = None) List[Any][source]

Truncate a list of tokens to a maximum length. If eos_id is provided, the last token will be replaced with eos_id.

Parameters:
  • tokens (List[Any]) – list of tokens to truncate

  • max_seq_len (int) – maximum length of the list

  • eos_id (Optional[Any]) – token to replace the last token with. If None, the last token will not be replaced. Default is None.

Returns:

truncated list of tokens

Return type:

List[Any]

Docs

Access comprehensive developer documentation for PyTorch

View Docs

Tutorials

Get in-depth tutorials for beginners and advanced developers

View Tutorials

Resources

Find development resources and get your questions answered

View Resources