torchx.workspace¶
Status: Beta
Workspaces are used to apply local changes on top of existing images so you can execute your code on a remote cluster. This module contains the interfaces used by workspace implementations.
These workspaces are defined as an fsspec
path which the directories and
files under will be used to generate a patch.
Example workspace paths:
file://.
the current working directory
memory://foo-bar/
an in-memory workspace for notebook/programmatic usage
- class torchx.workspace.WorkspaceMixin(*args: object, **kwargs: object)[source]¶
Note: (Prototype) this interface may change without notice!
A mix-in that can be attached to a Scheduler that adds the ability to builds a workspace. A workspace is the local checkout of the codebase/project that builds into an image. The workspace scheduler adds capability to automatically rebuild images or generate diff patches that are applied to the
Role
, allowing the user to make local code changes to the application and having those changes be reflected (either through a new image or an overlaid patch) at runtime without a manual image rebuild. The exact semantics of what the workspace build artifact is, is implementation dependent.- abstract build_workspace_and_update_role(role: Role, workspace: str, cfg: Mapping[str, Optional[Union[str, int, float, bool, List[str], Dict[str, str]]]]) None [source]¶
Builds the specified
workspace
with respect toimg
and updates therole
to reflect the built workspace artifacts. In the simplest case, this method builds a new image and updates the role’s image. Certain (more efficient) implementations build incremental diff patches that overlay on top of the role’s image.Note: this method mutates the passed
role
.
- dryrun_push_images(app: AppDef, cfg: Mapping[str, Optional[Union[str, int, float, bool, List[str], Dict[str, str]]]]) T [source]¶
dryrun_push does a dryrun of the image push and updates the app to have the final values. Only called for remote jobs.
push
must be called before scheduling the job.
- torchx.workspace.walk_workspace(fs: AbstractFileSystem, path: str, ignore_name: str = '.torchxignore') Iterable[Tuple[str, Iterable[str], Mapping[str, Mapping[str, object]]]] [source]¶
walk_workspace walks the filesystem path and applies the ignore rules specified via
ignore_name
. This follows the rules for.dockerignore
. https://docs.docker.com/engine/reference/builder/#dockerignore-file
torchx.workspace.docker_workspace¶
- class torchx.workspace.docker_workspace.DockerWorkspaceMixin(*args: object, docker_client: Optional[DockerClient] = None, **kwargs: object)[source]¶
DockerWorkspaceMixin will build patched docker images from the workspace. These patched images are docker images and can be either used locally via the docker daemon or pushed using the helper methods to a remote repository for remote jobs.
This requires a running docker daemon locally and for remote pushing requires being authenticated to those repositories via
docker login
.If there is a
Dockerfile.torchx
file present in the workspace that will be used instead to build the container.The docker build is provided with some extra build arguments that can be used in the Dockerfile.torchx:
IMAGE: the image string from the first Role in the AppDef
WORKSPACE: the full workspace path
To exclude files from the build context you can use the standard .dockerignore file.
See more:
- build_workspace_and_update_role(role: Role, workspace: str, cfg: Mapping[str, Optional[Union[str, int, float, bool, List[str], Dict[str, str]]]]) None [source]¶
Builds a new docker image using the
role
’s image as the base image and updates therole
’s image with this newly built docker image id- Parameters:
role – the role whose image (a Docker image) is to be used as the base image
workspace – a fsspec path to a directory with contents to be overlaid
- dryrun_push_images(app: AppDef, cfg: Mapping[str, Optional[Union[str, int, float, bool, List[str], Dict[str, str]]]]) Dict[str, Tuple[str, str]] [source]¶
_update_app_images replaces the local Docker images (identified via
sha256:...
) in the providedAppDef
with the remote path that they will be uploaded to and returns a mapping of local to remote names.push
must be called with the returned mapping before launching the job.- Returns:
A dict of [local image name, (remote repo, tag)].
- push_images(images_to_push: Dict[str, Tuple[str, str]]) None [source]¶
_push_images pushes the specified images to the remote container repository with the specified tag. The docker daemon must be authenticated to the remote repository using
docker login
.- Parameters:
images_to_push – A dict of [local image name, (remote repo, tag)].
torchx.workspace.dir_workspace¶
- class torchx.workspace.dir_workspace.DirWorkspaceMixin(*args: object, **kwargs: object)[source]¶
- build_workspace_and_update_role(role: Role, workspace: str, cfg: Mapping[str, Optional[Union[str, int, float, bool, List[str], Dict[str, str]]]]) None [source]¶
Creates a new directory specified by
job_dir
from the workspace. Role image fields will be set to thejob_dir
.Any files listed in the
.torchxignore
folder will be skipped.