torchtext.functional

to_tensor

torchtext.functional.to_tensor(input: Any, padding_value: Optional[int] = None, dtype: torch.dtype = torch.int64) → torch.Tensor

Convert input to torch tensor

Parameters

• padding_value (Optional[int]) – Pad value to make each input in the batch of length equal to the longest sequence in the batch.

• dtype (torch.dtype) – torch.dtype of output tensor

• input (Union[List[int], List[List[int]]]) – Sequence or batch of token ids

Return type

Tensor

Tutorials using to_tensor:

SST-2 Binary text classification with XLM-RoBERTa model

truncate

torchtext.functional.truncate(input: Any, max_seq_len: int) → Any

Truncate input sequence or batch

Parameters

• input (Union[List[Union[str, int]], List[List[Union[str, int]]]]) – Input sequence or batch to be truncated

• max_seq_len (int) – Maximum length beyond which input is discarded

Returns

Truncated sequence

Return type

Union[List[Union[str, int]], List[List[Union[str, int]]]]

add_token

torchtext.functional.add_token(input: Any, token_id: Any, begin: bool = True) → Any

Add token to start or end of sequence

Parameters

• input (Union[List[Union[str, int]], List[List[Union[str, int]]]]) – Input sequence or batch

• token_id (Union[str, int]) – token to be added

• begin (bool, optional) – Whether to insert token at start or end or sequence, defaults to True

Returns

sequence or batch with token_id added to begin or end or input

Return type

Union[List[Union[str, int]], List[List[Union[str, int]]]]

str_to_int

torchtext.functional.str_to_int(input: Any) → Any

Convert string tokens to integers (either single sequence or batch).

Parameters

input (Union[List[str], List[List[str]]]) – Input sequence or batch

Returns

Sequence or batch of string tokens converted to integers

Return type

Union[List[int], List[List[int]]]