Shortcuts

torchtext.functional

to_tensor

torchtext.functional.to_tensor(input: Any, padding_value: Optional[int] = None, dtype: torch.dtype = torch.int64)torch.Tensor[source]

Convert input to torch tensor

Parameters
  • padding_value (Optional[int]) – Pad value to make each input in the batch of length equal to the longest sequence in the batch.

  • dtype (torch.dtype) – torch.dtype of output tensor

  • input (Union[List[int], List[List[int]]]) – Sequence or batch of token ids

Return type

Tensor

Tutorials using to_tensor:

truncate

torchtext.functional.truncate(input: Any, max_seq_len: int)Any[source]

Truncate input sequence or batch

Parameters
  • input (Union[List[Union[str, int]], List[List[Union[str, int]]]]) – Input sequence or batch to be truncated

  • max_seq_len (int) – Maximum length beyond which input is discarded

Returns

Truncated sequence

Return type

Union[List[Union[str, int]], List[List[Union[str, int]]]]

add_token

torchtext.functional.add_token(input: Any, token_id: Any, begin: bool = True)Any[source]

Add token to start or end of sequence

Parameters
  • input (Union[List[Union[str, int]], List[List[Union[str, int]]]]) – Input sequence or batch

  • token_id (Union[str, int]) – token to be added

  • begin (bool, optional) – Whether to insert token at start or end or sequence, defaults to True

Returns

sequence or batch with token_id added to begin or end or input

Return type

Union[List[Union[str, int]], List[List[Union[str, int]]]]

Docs

Access comprehensive developer documentation for PyTorch

View Docs

Tutorials

Get in-depth tutorials for beginners and advanced developers

View Tutorials

Resources

Find development resources and get your questions answered

View Resources