ts.torch_handler.request_envelope package

Submodules

ts.torch_handler.request_envelope.base module

Base class for all RequestEnvelope.

A request envelope reformats the inputs/outputs of a call to a handler. It translates from formats specific to a model orchestrator like Seldon or KServe to a set of flat Python items, and vice versa.

class ts.torch_handler.request_envelope.base.BaseEnvelope(handle_fn)

Bases: ABC

Interface for all envelopes. Derive from this class, replacing the abstract methods

abstract format_output(data)

handle(data, context)

The Input Requests and Response are handled here.

abstract parse_input(data)

ts.torch_handler.request_envelope.body module

Default class for all handlers. Grabs the data out of the request body.

class ts.torch_handler.request_envelope.body.BodyEnvelope(handle_fn)

Bases: BaseEnvelope

Gets the key “body” from the input data, returns a raw list

format_output(data)

parse_input(data)

ts.torch_handler.request_envelope.json module

Uses JSON formatted inputs/outputs, following the structure outlined in https://www.tensorflow.org/tfx/serving/api_rest

class ts.torch_handler.request_envelope.json.JSONEnvelope(handle_fn)

Bases: BaseEnvelope

Implementation. Captures batches in JSON format, returns also in JSON format.

format_output(data)

parse_input(data)

ts.torch_handler.request_envelope.kserve module

The KServe Envelope is used to handle the KServe Input Request inside Torchserve.

class ts.torch_handler.request_envelope.kserve.KServeEnvelope(handle_fn)

Bases: BaseEnvelope

This function is used to handle the input request specified in kserve format and converts it into a Torchserve readable format.

Parameters:

Format (data - List of Input Request in kserve) –

Returns:

Returns the list of the Input Request in Torchserve Format

Return type:

[list]

format_output(data)

Returns the prediction response and captum explanation response of the input request.

Parameters:

outputs (List) – The outputs arguments is in the form of a list of dictionaries.

Returns:

The response is returned as a list of predictions and explanations

Return type:

(list)

parse_input(data)

ts.torch_handler.request_envelope.kservev2 module

The KServe Envelope is used to handle the KServe Input Request inside Torchserve.

class ts.torch_handler.request_envelope.kservev2.KServev2Envelope(handle_fn)

Bases: BaseEnvelope

Implementation. Captures batches in KServe v2 protocol format, returns also in FServing v2 protocol format.

format_output(data)

Translates Torchserve output KServe v2 response format.

Parameters: data (list): Torchserve response for handler.

Returns: KServe v2 response json. {

“id”: “f0222600-353f-47df-8d9d-c96d96fa894e”, “model_name”: “bert”, “model_version”: “1”, “outputs”: [{

“name”: “input-0”, “shape”: [1], “datatype”: “INT64”, “data”: [2]

}]

}

parse_input(data)

Translates KServe request input to list of data expected by Torchserve.

Parameters: data (json): KServe v2 request input json. {

“inputs”: [{

“name”: “input-0”, “shape”: [37], “datatype”: “INT64”, “data”: [66, 108, 111, 111, 109]

}]

}

Returns: list of data objects. [{ ‘name’: ‘input-0’, ‘shape’: [5], ‘datatype’: ‘INT64’, ‘data’: [66, 108, 111, 111, 109] }]

Module contents