torcharrow.Column

A torcharrow.Column is a 1-dimension torch.Tensor like data structure containing elements of a single data type. It also supports non-numeric types such as string, list, struct.

Data types

TorchArrow defines the following data types for column, which is in module torcharrow.dtypes (abbreviated as dt in table below):

Data type	dtype
32-bit floating point	dt.float32 or dt.Float32(nullable)
64-bit floating point	dt.float64 or dt.Float64(nullable)
8-bit signed integer	dt.int8 or dt.Int8(nullable)
16-bit signed integer	dt.int16 or dt.Int16(nullable)
32-bit signed integer	dt.int32 or dt.Int32(nullable)
64-bit signed integer	dt.int64 or dt.Int64(nullable)
Boolean	dt.boolean or dt.Boolean(nullable)
String	dt.string or dt.String(nullable)
List	dt.List(item_dtype, nullable)
Struct	dt.Struct(fields, nullable)

Column class reference

class torcharrow.Column

Column.dtype

the data type of a torcharrow.Column

Column.device

the device on which a torcharrow.Column is or will be allocated.

Column.length

Return number of rows including null values

Column.null_count

Return number of null values

Column.head	Return the first n rows.
Column.tail	Return the last n rows.
Column.cast	Cast the Column to the given dtype
Column.is_valid_at	(EXPERIMENTAL API) Return whether data at index i is valid, i.e., non-null
Column.append	Returns column/dataframe with values appended.
Column.isin	Check whether each element in the column is contained in values.
Column.all	Return whether all non-null elements are True
Column.any	Return whether any non-null element is True
Column.map	Maps rows according to input correspondence.
Column.filter	Select rows where predicate is True.
Column.flatmap	Maps rows to list of rows according to input correspondence dtype required if result type != item type.
Column.transform	Like map() but invokes the callable on mini-batches of rows at a time.
Column.fill_null	Fill null values using the specified method.
Column.drop_null	Return a column/frame with rows removed where a row has any or all nulls.
Column.drop_duplicates	(EXPERIMENTAL API) Remove duplicate values from row/frame but keep the first, last, none
Column.to_arrow	Convert self to arrow array
Column.to_tensor	Convert to PyTorch containers (Tensor, PackedList, PackedMap, etc)
Column.to_pylist	Convert to plain Python container (list of scalars or containers)
Column.to_pandas	Convert self to Pandas Series

NumericalColumn class reference

class torcharrow.NumericalColumn

NumericalColumn.abs	Absolute value of each element of the series.
NumericalColumn.ceil	Rounds each value upward to the smallest integral
NumericalColumn.floor	Rounds each value downward to the largest integral value
NumericalColumn.round	Round each value in a data to the given number of decimals.
NumericalColumn.log	Returns a new column with the natural logarithm of the elements
NumericalColumn.describe	Generate descriptive statistics.
NumericalColumn.min	Return the minimum of the non-null values.
NumericalColumn.max	Return the maximum of the non-null values.
NumericalColumn.sum	Return the sum of the non-null values.
NumericalColumn.mean	Return the mean of the non-null values.
NumericalColumn.std	Return the stddev(s) of the data.
NumericalColumn.median	Return the median of the values in the data.

StringColumn class reference

class torcharrow.StringColumn

istring_column.StringMethods.length	Compute the length of each element in the Column.
istring_column.StringMethods.slice	Slice substrings from each element in the Column.
istring_column.StringMethods.split	Split strings around given separator/delimiter.
istring_column.StringMethods.strip	Remove leading and trailing whitespaces.
istring_column.StringMethods.isalpha	Return True if the string is an alphabetic string, False otherwise.
istring_column.StringMethods.isnumeric	Returns True if all the characters are numeric, otherwise False.
istring_column.StringMethods.isalnum	Return True if all characters in the string are alphanumeric (either alphabets or numbers), False otherwise.
istring_column.StringMethods.isdigit	Return True if all characters in the string are numeric, False otherwise.
istring_column.StringMethods.isdecimal	Return True if the string contains only decimal digit (from 0 to 9), False otherwise.
istring_column.StringMethods.isspace	Return True all characters in the string are whitespace, False otherwise.
istring_column.StringMethods.islower	Return True if the non-empty string is in lower case, False otherwise.
istring_column.StringMethods.isupper	Return True if the non-empty string is in upper case, False otherwise.
istring_column.StringMethods.istitle	Return True if each word of the string starts with an upper case letter, False otherwise.
istring_column.StringMethods.lower	Convert strings in the Column to lowercase.
istring_column.StringMethods.upper	Convert strings in the Column to uppercase.
istring_column.StringMethods.startswith	Test if the beginning of each string element matches a pattern.
istring_column.StringMethods.endswith	Test if the end of each string element matches a pattern.
istring_column.StringMethods.count	Count occurrences of pattern in each string of column
istring_column.StringMethods.find	Return lowest indices in each strings in the Column.
istring_column.StringMethods.replace	Replace each occurrence of pattern in the Column.
istring_column.StringMethods.match	Determine if each string matches a regular expression
istring_column.StringMethods.contains	Test for each item if pattern is contained within a string; returns a boolean
istring_column.StringMethods.findall	Find for each item all occurrences of pattern (see re.findall())

ListColumn class reference

class torcharrow.ListColumn

ilist_column.ListMethods.length	Compute the length of each element in the Column.
ilist_column.ListMethods.slice	Slice sublist from each element in the column
ilist_column.ListMethods.vmap	(EXPERIMENTAL API) Vectorizing map.