ts package

Subpackages

• ts.metrics package
  • Submodules
  • ts.metrics.dimension module
  • ts.metrics.metric module
  • ts.metrics.metric_collector module
  • ts.metrics.metric_encoder module
  • ts.metrics.metrics_store module
  • ts.metrics.process_memory_metric module
  • ts.metrics.system_metrics module
  • ts.metrics.unit module
  • Module contents
• ts.model_service package
  • Submodules
  • ts.model_service.model_service module
  • Module contents
• ts.protocol package
  • Submodules
  • ts.protocol.otf_message_handler module
  • Module contents
• ts.torch_handler package
  • Subpackages
    • ts.torch_handler.request_envelope package
      • Submodules
      • ts.torch_handler.request_envelope.base module
      • ts.torch_handler.request_envelope.body module
      • ts.torch_handler.request_envelope.json module
      • ts.torch_handler.request_envelope.kserve module
      • ts.torch_handler.request_envelope.kservev2 module
      • Module contents
    • ts.torch_handler.unit_tests package
      • Subpackages
      • Submodules
      • ts.torch_handler.unit_tests.test_base_handler module
      • ts.torch_handler.unit_tests.test_envelopes module
      • ts.torch_handler.unit_tests.test_image_classifier module
      • ts.torch_handler.unit_tests.test_image_segmenter module
      • ts.torch_handler.unit_tests.test_mnist_kf module
      • ts.torch_handler.unit_tests.test_object_detector module
      • Module contents
  • Submodules
  • ts.torch_handler.base_handler module
  • ts.torch_handler.contractions module
  • ts.torch_handler.densenet_handler module
  • ts.torch_handler.image_classifier module
  • ts.torch_handler.image_segmenter module
  • ts.torch_handler.object_detector module
  • ts.torch_handler.text_classifier module
  • ts.torch_handler.text_handler module
  • ts.torch_handler.vision_handler module
  • Module contents
• ts.utils package
  • Submodules
  • ts.utils.timeit_decorator module
  • ts.utils.util module
  • Module contents

Submodules

ts.arg_parser module

This module parses the arguments given through the torchserve command-line. This is used by model-server at runtime.

class ts.arg_parser.ArgParser

Bases: object

Argument parser for torchserve and torchserve-export commands TODO : Add readme url

static extract_args(args=None)

static model_service_worker_args()

ArgParser for backend worker. Takes the socket name and socket type. :return:

static ts_parser()

Argument parser for torchserve start service

ts.context module

Context object of incoming request

class ts.context.Context(model_name, model_dir, manifest, batch_size, gpu, mms_version, limit_max_image_pixels=True, metrics=None, model_yaml_config=None)

Bases: object

Context stores model relevant worker information Some fixed during load times and some set by the service

get_all_request_header(idx: int) → Dict[str, str]

get_request_header(idx: int, key: str) → Optional[str]

get_request_id(idx: int = 0) → Optional[str]

get_response_content_type(idx: int) → Optional[str]

get_response_headers(idx: int) → Dict[str, str]

get_response_status(idx: int) → Tuple[int, str]

get_sequence_id(idx: int) → str

property metrics

property request_processor

set_all_response_status(code: int = 200, phrase: str = '') → None

Set the status code of individual requests :param phrase: :param code: :return:

set_response_content_type(idx: int, value: str) → None

set_response_header(idx, key, value)

set_response_status(code: int = 200, phrase: str = '', idx: int = 0)

Set the status code of individual requests :param phrase: :param idx: The index data in the list(data) that is sent to the handle() method :param code: :return:

property system_properties

class ts.context.RequestProcessor(request_header: dict)

Bases: object

Request processor

add_response_property(key: str, value: str) → None

get_request_properties() → dict

get_request_property(key: str) → Optional[str]

get_response_header(key: str) → Optional[str]

get_response_headers() → dict

get_response_status_code() → int

get_response_status_phrase() → Optional[str]

report_status(code, reason_phrase=None) → None

ts.model_loader module

Model loader.

class ts.model_loader.ModelLoader

Bases: object

Base Model Loader class.

abstract load(model_name: str, model_dir: str, handler: Optional[str] = None, gpu_id: Optional[int] = None, batch_size: Optional[int] = None, envelope: Optional[str] = None, limit_max_image_pixels: Optional[bool] = True)

Load model from file.

Parameters:

• model_name –

• model_dir –

• handler –

• gpu_id –

• batch_size –

• envelope –

• limit_max_image_pixels –

Returns:

Model

class ts.model_loader.ModelLoaderFactory

Bases: object

static get_model_loader()

class ts.model_loader.TsModelLoader

Bases: ModelLoader

TorchServe 1.0 Model Loader

load(model_name: str, model_dir: str, handler: Optional[str] = None, gpu_id: Optional[int] = None, batch_size: Optional[int] = None, envelope: Optional[str] = None, limit_max_image_pixels: Optional[bool] = True, metrics_cache: Optional[MetricsCacheYamlImpl] = None) → Service

Load TorchServe 1.0 model from file.

Parameters:

• model_name –

• model_dir –

• handler –

• gpu_id –

• batch_size –

• envelope –

• limit_max_image_pixels –

• metrics_cache – MetricsCacheYamlImpl object

Returns:

ts.model_server module

File to define the entry point to Model Server

ts.model_server.load_properties(file_path: str) → Dict[str, str]

Read properties file into map.

ts.model_server.start() → None

This is the entry point for model server :return:

ts.model_service_worker module

ModelServiceWorker is the worker that is started by the MMS front-end. Communication message format: binary encoding

class ts.model_service_worker.TorchModelServiceWorker(s_type: Optional[str] = None, s_name: Optional[str] = None, host_addr: Optional[str] = None, port_num: Optional[int] = None, metrics_config: Optional[str] = None, async_comm: Optional[bool] = False)

Bases: object

Backend worker to handle Model Server’s python service code

handle_connection(cl_socket)

Handle socket connection.

Parameters:

cl_socket –

Returns:

handle_connection_async(cl_socket)

Handle socket connection.

Parameters:

cl_socket –

Returns:

load_model(load_model_request)

Expected command {

“command” : “load”, string “modelPath” : “/path/to/model/file”, string “modelName” : “name”, string “gpu” : None if CPU else gpu_id, int “handler” : service handler entry point if provided, string “envelope” : name of wrapper/unwrapper of request data if provided, string “batchSize” : batch size, int “limitMaxImagePixels”: limit pillow image max_image_pixels, bool

}

Parameters:

load_model_request –

Returns:

run_server()

Run the backend worker process and listen on a socket :return:

ts.service module

CustomService class definitions

class ts.service.Service(model_name, model_dir, manifest, entry_point, gpu, batch_size, limit_max_image_pixels=True, metrics_cache=None)

Bases: object

Wrapper for custom entry_point

property context

predict(batch)

PREDICT COMMAND = {

“command”: “predict”, “batch”: [ REQUEST_INPUT ]

} :param batch: list of request :return:

static retrieve_data_for_inference(batch)

REQUEST_INPUT = {

“requestId” : “111-222-3333”, “parameters” : [ PARAMETER ]

}

PARAMETER = {

“name” : parameter name “contentType”: “http-content-types”, “value”: “val1”

}

Parameters:

batch –

Returns:

set_cl_socket(cl_socket)

ts.service.emit_metrics(metrics)

Emit the metrics in the provided Dictionary

Parameters:

• metrics (A dictionary of all) –

• metrics –

• metric_name (when key is) –

• object (value is a metric) –

ts.version module

This is the current version of TorchServe

Module contents

This module does the following: a. Starts model-server. b. Creates end-points based on the configured models. c. Exposes standard “ping” and “api-description” endpoints. d. Waits for servicing inference requests.