Shortcuts

torchtext.functional

to_tensor

torchtext.functional.to_tensor(input: Any, padding_value: Optional[int] = None, dtype: dtype = torch.int64) Tensor[source]

Convert input to torch tensor

Parameters:
  • padding_value (Optional[int]) – Pad value to make each input in the batch of length equal to the longest sequence in the batch.

  • dtype (torch.dtype) – torch.dtype of output tensor

  • input (Union[List[int], List[List[int]]]) – Sequence or batch of token ids

Return type:

Tensor

Tutorials using to_tensor:
SST-2 Binary text classification with XLM-RoBERTa model

SST-2 Binary text classification with XLM-RoBERTa model

SST-2 Binary text classification with XLM-RoBERTa model

truncate

torchtext.functional.truncate(input: Any, max_seq_len: int) Any[source]

Truncate input sequence or batch

Parameters:
  • input (Union[List[Union[str, int]], List[List[Union[str, int]]]]) – Input sequence or batch to be truncated

  • max_seq_len (int) – Maximum length beyond which input is discarded

Returns:

Truncated sequence

Return type:

Union[List[Union[str, int]], List[List[Union[str, int]]]]

add_token

torchtext.functional.add_token(input: Any, token_id: Any, begin: bool = True) Any[source]

Add token to start or end of sequence

Parameters:
  • input (Union[List[Union[str, int]], List[List[Union[str, int]]]]) – Input sequence or batch

  • token_id (Union[str, int]) – token to be added

  • begin (bool, optional) – Whether to insert token at start or end or sequence, defaults to True

Returns:

sequence or batch with token_id added to begin or end or input

Return type:

Union[List[Union[str, int]], List[List[Union[str, int]]]]

str_to_int

torchtext.functional.str_to_int(input: Any) Any[source]

Convert string tokens to integers (either single sequence or batch).

Parameters:

input (Union[List[str], List[List[str]]]) – Input sequence or batch

Returns:

Sequence or batch of string tokens converted to integers

Return type:

Union[List[int], List[List[int]]]

Docs

Access comprehensive developer documentation for PyTorch

View Docs

Tutorials

Get in-depth tutorials for beginners and advanced developers

View Tutorials

Resources

Find development resources and get your questions answered

View Resources