Hyperparameter tuning with Ray Tune

Hyperparameter tuning can make the difference between an average model and a highly accurate one. Often simple things like choosing a different learning rate or changing a network layer size can have a dramatic impact on your model performance.

Fortunately, there are tools that help with finding the best combination of parameters. Ray Tune is an industry standard tool for distributed hyperparameter tuning. Ray Tune includes the latest hyperparameter search algorithms, integrates with TensorBoard and other analysis libraries, and natively supports distributed training through Ray’s distributed machine learning engine.

In this tutorial, we will show you how to integrate Ray Tune into your PyTorch training workflow. We will extend this tutorial from the PyTorch documentation for training a CIFAR10 image classifier.

As you will see, we only need to add some slight modifications. In particular, we need to

1. wrap data loading and training in functions,
2. make some network parameters configurable,
3. add checkpointing (optional),
4. and define the search space for the model tuning

To run this tutorial, please make sure the following packages are installed:

• ray[tune]: Distributed hyperparameter tuning library
• torchvision: For the data transformers

Setup / Imports

Let’s start with the imports:

from functools import partial
import numpy as np
import os
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import random_split
import torchvision
import torchvision.transforms as transforms
from ray import tune
from ray.tune import CLIReporter
from ray.tune.schedulers import ASHAScheduler

Most of the imports are needed for building the PyTorch model. Only the last three imports are for Ray Tune.

Data loaders

We wrap the data loaders in their own function and pass a global data directory. This way we can share a data directory between different trials.

def load_data(data_dir="./data"):
    transform = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
    ])

    trainset = torchvision.datasets.CIFAR10(
        root=data_dir, train=True, download=True, transform=transform)

    testset = torchvision.datasets.CIFAR10(
        root=data_dir, train=False, download=True, transform=transform)

    return trainset, testset

Configurable neural network

We can only tune those parameters that are configurable. In this example, we can specify the layer sizes of the fully connected layers:

class Net(nn.Module):
    def __init__(self, l1=120, l2=84):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, l1)
        self.fc2 = nn.Linear(l1, l2)
        self.fc3 = nn.Linear(l2, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

The train function

Now it gets interesting, because we introduce some changes to the example from the PyTorch documentation.

We wrap the training script in a function train_cifar(config, checkpoint_dir=None, data_dir=None). As you can guess, the config parameter will receive the hyperparameters we would like to train with. The checkpoint_dir parameter is used to restore checkpoints. The data_dir specifies the directory where we load and store the data, so multiple runs can share the same data source.

net = Net(config["l1"], config["l2"])

if checkpoint_dir:
    model_state, optimizer_state = torch.load(
        os.path.join(checkpoint_dir, "checkpoint"))
    net.load_state_dict(model_state)
    optimizer.load_state_dict(optimizer_state)

The learning rate of the optimizer is made configurable, too:

optimizer = optim.SGD(net.parameters(), lr=config["lr"], momentum=0.9)

We also split the training data into a training and validation subset. We thus train on 80% of the data and calculate the validation loss on the remaining 20%. The batch sizes with which we iterate through the training and test sets are configurable as well.

Adding (multi) GPU support with DataParallel

Image classification benefits largely from GPUs. Luckily, we can continue to use PyTorch’s abstractions in Ray Tune. Thus, we can wrap our model in nn.DataParallel to support data parallel training on multiple GPUs:

device = "cpu"
if torch.cuda.is_available():
    device = "cuda:0"
    if torch.cuda.device_count() > 1:
        net = nn.DataParallel(net)
net.to(device)

By using a device variable we make sure that training also works when we have no GPUs available. PyTorch requires us to send our data to the GPU memory explicitly, like this:

for i, data in enumerate(trainloader, 0):
    inputs, labels = data
    inputs, labels = inputs.to(device), labels.to(device)

The code now supports training on CPUs, on a single GPU, and on multiple GPUs. Notably, Ray also supports fractional GPUs so we can share GPUs among trials, as long as the model still fits on the GPU memory. We’ll come back to that later.

Communicating with Ray Tune

The most interesting part is the communication with Ray Tune:

with tune.checkpoint_dir(epoch) as checkpoint_dir:
    path = os.path.join(checkpoint_dir, "checkpoint")
    torch.save((net.state_dict(), optimizer.state_dict()), path)

tune.report(loss=(val_loss / val_steps), accuracy=correct / total)

Here we first save a checkpoint and then report some metrics back to Ray Tune. Specifically, we send the validation loss and accuracy back to Ray Tune. Ray Tune can then use these metrics to decide which hyperparameter configuration lead to the best results. These metrics can also be used to stop bad performing trials early in order to avoid wasting resources on those trials.

The checkpoint saving is optional, however, it is necessary if we wanted to use advanced schedulers like Population Based Training. Also, by saving the checkpoint we can later load the trained models and validate them on a test set.

Full training function

The full code example looks like this:

def train_cifar(config, checkpoint_dir=None, data_dir=None):
    net = Net(config["l1"], config["l2"])

    device = "cpu"
    if torch.cuda.is_available():
        device = "cuda:0"
        if torch.cuda.device_count() > 1:
            net = nn.DataParallel(net)
    net.to(device)

    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(net.parameters(), lr=config["lr"], momentum=0.9)

    if checkpoint_dir:
        model_state, optimizer_state = torch.load(
            os.path.join(checkpoint_dir, "checkpoint"))
        net.load_state_dict(model_state)
        optimizer.load_state_dict(optimizer_state)

    trainset, testset = load_data(data_dir)

    test_abs = int(len(trainset) * 0.8)
    train_subset, val_subset = random_split(
        trainset, [test_abs, len(trainset) - test_abs])

    trainloader = torch.utils.data.DataLoader(
        train_subset,
        batch_size=int(config["batch_size"]),
        shuffle=True,
        num_workers=8)
    valloader = torch.utils.data.DataLoader(
        val_subset,
        batch_size=int(config["batch_size"]),
        shuffle=True,
        num_workers=8)

    for epoch in range(10):  # loop over the dataset multiple times
        running_loss = 0.0
        epoch_steps = 0
        for i, data in enumerate(trainloader, 0):
            # get the inputs; data is a list of [inputs, labels]
            inputs, labels = data
            inputs, labels = inputs.to(device), labels.to(device)

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward + backward + optimize
            outputs = net(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            # print statistics
            running_loss += loss.item()
            epoch_steps += 1
            if i % 2000 == 1999:  # print every 2000 mini-batches
                print("[%d, %5d] loss: %.3f" % (epoch + 1, i + 1,
                                                running_loss / epoch_steps))
                running_loss = 0.0

        # Validation loss
        val_loss = 0.0
        val_steps = 0
        total = 0
        correct = 0
        for i, data in enumerate(valloader, 0):
            with torch.no_grad():
                inputs, labels = data
                inputs, labels = inputs.to(device), labels.to(device)

                outputs = net(inputs)
                _, predicted = torch.max(outputs.data, 1)
                total += labels.size(0)
                correct += (predicted == labels).sum().item()

                loss = criterion(outputs, labels)
                val_loss += loss.cpu().numpy()
                val_steps += 1

        with tune.checkpoint_dir(epoch) as checkpoint_dir:
            path = os.path.join(checkpoint_dir, "checkpoint")
            torch.save((net.state_dict(), optimizer.state_dict()), path)

        tune.report(loss=(val_loss / val_steps), accuracy=correct / total)
    print("Finished Training")

As you can see, most of the code is adapted directly from the original example.

Test set accuracy

Commonly the performance of a machine learning model is tested on a hold-out test set with data that has not been used for training the model. We also wrap this in a function:

def test_accuracy(net, device="cpu"):
    trainset, testset = load_data()

    testloader = torch.utils.data.DataLoader(
        testset, batch_size=4, shuffle=False, num_workers=2)

    correct = 0
    total = 0
    with torch.no_grad():
        for data in testloader:
            images, labels = data
            images, labels = images.to(device), labels.to(device)
            outputs = net(images)
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()

    return correct / total

The function also expects a device parameter, so we can do the test set validation on a GPU.

Configuring the search space

Lastly, we need to define Ray Tune’s search space. Here is an example:

config = {
    "l1": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
    "l2": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
    "lr": tune.loguniform(1e-4, 1e-1),
    "batch_size": tune.choice([2, 4, 8, 16])
}

The tune.sample_from() function makes it possible to define your own sample methods to obtain hyperparameters. In this example, the l1 and l2 parameters should be powers of 2 between 4 and 256, so either 4, 8, 16, 32, 64, 128, or 256. The lr (learning rate) should be uniformly sampled between 0.0001 and 0.1. Lastly, the batch size is a choice between 2, 4, 8, and 16.

At each trial, Ray Tune will now randomly sample a combination of parameters from these search spaces. It will then train a number of models in parallel and find the best performing one among these. We also use the ASHAScheduler which will terminate bad performing trials early.

We wrap the train_cifar function with functools.partial to set the constant data_dir parameter. We can also tell Ray Tune what resources should be available for each trial:

gpus_per_trial = 2
# ...
result = tune.run(
    partial(train_cifar, data_dir=data_dir),
    resources_per_trial={"cpu": 8, "gpu": gpus_per_trial},
    config=config,
    num_samples=num_samples,
    scheduler=scheduler,
    progress_reporter=reporter,
    checkpoint_at_end=True)

You can specify the number of CPUs, which are then available e.g. to increase the num_workers of the PyTorch DataLoader instances. The selected number of GPUs are made visible to PyTorch in each trial. Trials do not have access to GPUs that haven’t been requested for them - so you don’t have to care about two trials using the same set of resources.

Here we can also specify fractional GPUs, so something like gpus_per_trial=0.5 is completely valid. The trials will then share GPUs among each other. You just have to make sure that the models still fit in the GPU memory.

After training the models, we will find the best performing one and load the trained network from the checkpoint file. We then obtain the test set accuracy and report everything by printing.

The full main function looks like this:

def main(num_samples=10, max_num_epochs=10, gpus_per_trial=2):
    data_dir = os.path.abspath("./data")
    load_data(data_dir)
    config = {
        "l1": tune.sample_from(lambda _: 2 ** np.random.randint(2, 9)),
        "l2": tune.sample_from(lambda _: 2 ** np.random.randint(2, 9)),
        "lr": tune.loguniform(1e-4, 1e-1),
        "batch_size": tune.choice([2, 4, 8, 16])
    }
    scheduler = ASHAScheduler(
        metric="loss",
        mode="min",
        max_t=max_num_epochs,
        grace_period=1,
        reduction_factor=2)
    reporter = CLIReporter(
        # parameter_columns=["l1", "l2", "lr", "batch_size"],
        metric_columns=["loss", "accuracy", "training_iteration"])
    result = tune.run(
        partial(train_cifar, data_dir=data_dir),
        resources_per_trial={"cpu": 2, "gpu": gpus_per_trial},
        config=config,
        num_samples=num_samples,
        scheduler=scheduler,
        progress_reporter=reporter)

    best_trial = result.get_best_trial("loss", "min", "last")
    print("Best trial config: {}".format(best_trial.config))
    print("Best trial final validation loss: {}".format(
        best_trial.last_result["loss"]))
    print("Best trial final validation accuracy: {}".format(
        best_trial.last_result["accuracy"]))

    best_trained_model = Net(best_trial.config["l1"], best_trial.config["l2"])
    device = "cpu"
    if torch.cuda.is_available():
        device = "cuda:0"
        if gpus_per_trial > 1:
            best_trained_model = nn.DataParallel(best_trained_model)
    best_trained_model.to(device)

    best_checkpoint_dir = best_trial.checkpoint.value
    model_state, optimizer_state = torch.load(os.path.join(
        best_checkpoint_dir, "checkpoint"))
    best_trained_model.load_state_dict(model_state)

    test_acc = test_accuracy(best_trained_model, device)
    print("Best trial test set accuracy: {}".format(test_acc))


if __name__ == "__main__":
    # You can change the number of GPUs per trial here:
    main(num_samples=10, max_num_epochs=10, gpus_per_trial=0)

Out:

Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to /var/lib/jenkins/workspace/beginner_source/data/cifar-10-python.tar.gz
Extracting /var/lib/jenkins/workspace/beginner_source/data/cifar-10-python.tar.gz to /var/lib/jenkins/workspace/beginner_source/data
Files already downloaded and verified
== Status ==
Memory usage on this node: 4.6/240.1 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/1.0 accelerator_type:M60)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (10 PENDING)
+---------------------+----------+-------+--------------+------+------+-------------+
| Trial name          | status   | loc   |   batch_size |   l1 |   l2 |          lr |
|---------------------+----------+-------+--------------+------+------+-------------|
| DEFAULT_29585_00000 | PENDING  |       |            8 |   16 |    4 | 0.0224094   |
| DEFAULT_29585_00001 | PENDING  |       |            4 |    8 |   16 | 0.0156001   |
| DEFAULT_29585_00002 | PENDING  |       |            2 |  256 |    4 | 0.000325006 |
| DEFAULT_29585_00003 | PENDING  |       |           16 |  128 |  256 | 0.00576339  |
| DEFAULT_29585_00004 | PENDING  |       |            2 |  128 |  256 | 0.000801667 |
| DEFAULT_29585_00005 | PENDING  |       |           16 |   16 |   64 | 0.000935427 |
| DEFAULT_29585_00006 | PENDING  |       |           16 |    4 |  256 | 0.0350703   |
| DEFAULT_29585_00007 | PENDING  |       |            8 |  128 |  256 | 0.00049783  |
| DEFAULT_29585_00008 | PENDING  |       |            2 |    4 |  256 | 0.000257535 |
| DEFAULT_29585_00009 | PENDING  |       |            2 |   16 |  256 | 0.00103426  |
+---------------------+----------+-------+--------------+------+------+-------------+


(pid=1464) Files already downloaded and verified
(pid=1470) Files already downloaded and verified
(pid=1469) Files already downloaded and verified
(pid=1443) Files already downloaded and verified
(pid=1471) Files already downloaded and verified
(pid=1475) Files already downloaded and verified
(pid=1479) Files already downloaded and verified
(pid=1482) Files already downloaded and verified
(pid=1422) Files already downloaded and verified
(pid=1474) Files already downloaded and verified
(pid=1464) Files already downloaded and verified
(pid=1470) Files already downloaded and verified
(pid=1469) Files already downloaded and verified
(pid=1443) Files already downloaded and verified
(pid=1471) Files already downloaded and verified
(pid=1475) Files already downloaded and verified
(pid=1479) Files already downloaded and verified
(pid=1482) Files already downloaded and verified
(pid=1422) Files already downloaded and verified
(pid=1474) Files already downloaded and verified
(pid=1470) [1,  2000] loss: 2.305
(pid=1464) [1,  2000] loss: 2.314
(pid=1474) [1,  2000] loss: 2.154
(pid=1482) [1,  2000] loss: 2.259
(pid=1469) [1,  2000] loss: 2.269
(pid=1471) [1,  2000] loss: 2.263
(pid=1475) [1,  2000] loss: 2.295
(pid=1443) [1,  2000] loss: 2.180
(pid=1479) [1,  2000] loss: 2.236
(pid=1422) [1,  2000] loss: 1.762
(pid=1470) [1,  4000] loss: 1.098
(pid=1474) [1,  4000] loss: 0.916
(pid=1464) [1,  4000] loss: 1.114
(pid=1482) [1,  4000] loss: 1.000
Result for DEFAULT_29585_00006:
  accuracy: 0.1463
  date: 2021-07-28_21-24-44
  done: false
  experiment_id: 0b71c9f87a174eec9cd929ab59f282e0
  hostname: 2264d6af0a6a
  iterations_since_restore: 1
  loss: 2.2123412368774416
  node_ip: 172.17.0.2
  pid: 1443
  should_checkpoint: true
  time_since_restore: 27.045654773712158
  time_this_iter_s: 27.045654773712158
  time_total_s: 27.045654773712158
  timestamp: 1627507484
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '29585_00006'

== Status ==
Memory usage on this node: 10.0/240.1 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -2.2123412368774416
Resources requested: 20.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/1.0 accelerator_type:M60)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (10 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00000 | RUNNING  |                 |            8 |   16 |    4 | 0.0224094   |         |            |                      |
| DEFAULT_29585_00001 | RUNNING  |                 |            4 |    8 |   16 | 0.0156001   |         |            |                      |
| DEFAULT_29585_00002 | RUNNING  |                 |            2 |  256 |    4 | 0.000325006 |         |            |                      |
| DEFAULT_29585_00003 | RUNNING  |                 |           16 |  128 |  256 | 0.00576339  |         |            |                      |
| DEFAULT_29585_00004 | RUNNING  |                 |            2 |  128 |  256 | 0.000801667 |         |            |                      |
| DEFAULT_29585_00005 | RUNNING  |                 |           16 |   16 |   64 | 0.000935427 |         |            |                      |
| DEFAULT_29585_00006 | RUNNING  | 172.17.0.2:1443 |           16 |    4 |  256 | 0.0350703   | 2.21234 |     0.1463 |                    1 |
| DEFAULT_29585_00007 | RUNNING  |                 |            8 |  128 |  256 | 0.00049783  |         |            |                      |
| DEFAULT_29585_00008 | RUNNING  |                 |            2 |    4 |  256 | 0.000257535 |         |            |                      |
| DEFAULT_29585_00009 | RUNNING  |                 |            2 |   16 |  256 | 0.00103426  |         |            |                      |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_29585_00005:
  accuracy: 0.27
  date: 2021-07-28_21-24-45
  done: false
  experiment_id: e9270fc4d818458b9a9ccc4ac3084736
  hostname: 2264d6af0a6a
  iterations_since_restore: 1
  loss: 1.977501169204712
  node_ip: 172.17.0.2
  pid: 1479
  should_checkpoint: true
  time_since_restore: 27.610695838928223
  time_this_iter_s: 27.610695838928223
  time_total_s: 27.610695838928223
  timestamp: 1627507485
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '29585_00005'

Result for DEFAULT_29585_00003:
  accuracy: 0.461
  date: 2021-07-28_21-24-45
  done: false
  experiment_id: e50b2c4b9e054ba9bd23f17f4b53ae05
  hostname: 2264d6af0a6a
  iterations_since_restore: 1
  loss: 1.4939670184135436
  node_ip: 172.17.0.2
  pid: 1422
  should_checkpoint: true
  time_since_restore: 28.191472053527832
  time_this_iter_s: 28.191472053527832
  time_total_s: 28.191472053527832
  timestamp: 1627507485
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '29585_00003'

(pid=1469) [1,  4000] loss: 1.154
(pid=1471) [1,  4000] loss: 1.142
(pid=1475) [1,  4000] loss: 1.032
(pid=1470) [1,  6000] loss: 0.670
(pid=1474) [1,  6000] loss: 0.573
(pid=1464) [1,  6000] loss: 0.684
(pid=1482) [1,  6000] loss: 0.592
Result for DEFAULT_29585_00000:
  accuracy: 0.0988
  date: 2021-07-28_21-24-58
  done: true
  experiment_id: 01ea87dac406416f90ba006cb4584907
  hostname: 2264d6af0a6a
  iterations_since_restore: 1
  loss: 2.30682048740387
  node_ip: 172.17.0.2
  pid: 1469
  should_checkpoint: true
  time_since_restore: 40.58424234390259
  time_this_iter_s: 40.58424234390259
  time_total_s: 40.58424234390259
  timestamp: 1627507498
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '29585_00000'

== Status ==
Memory usage on this node: 10.1/240.1 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -2.0949212030410767
Resources requested: 20.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_0_dd2b559219a27a74ecf8cc41a405d759, 0.0/2.0 CPU_group_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_dd2b559219a27a74ecf8cc41a405d759, 0.0/2.0 CPU_group_79fc53ec1198c2fe95cd3008fdb7c606, 0.0/2.0 CPU_group_e0cc112d61123b1cab4fc0b65c303c9d, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_0_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_0_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_e0cc112d61123b1cab4fc0b65c303c9d, 0.0/2.0 CPU_group_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_79fc53ec1198c2fe95cd3008fdb7c606, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (10 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00000 | RUNNING  | 172.17.0.2:1469 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | RUNNING  |                 |            4 |    8 |   16 | 0.0156001   |         |            |                      |
| DEFAULT_29585_00002 | RUNNING  |                 |            2 |  256 |    4 | 0.000325006 |         |            |                      |
| DEFAULT_29585_00003 | RUNNING  | 172.17.0.2:1422 |           16 |  128 |  256 | 0.00576339  | 1.49397 |     0.461  |                    1 |
| DEFAULT_29585_00004 | RUNNING  |                 |            2 |  128 |  256 | 0.000801667 |         |            |                      |
| DEFAULT_29585_00005 | RUNNING  | 172.17.0.2:1479 |           16 |   16 |   64 | 0.000935427 | 1.9775  |     0.27   |                    1 |
| DEFAULT_29585_00006 | RUNNING  | 172.17.0.2:1443 |           16 |    4 |  256 | 0.0350703   | 2.21234 |     0.1463 |                    1 |
| DEFAULT_29585_00007 | RUNNING  |                 |            8 |  128 |  256 | 0.00049783  |         |            |                      |
| DEFAULT_29585_00008 | RUNNING  |                 |            2 |    4 |  256 | 0.000257535 |         |            |                      |
| DEFAULT_29585_00009 | RUNNING  |                 |            2 |   16 |  256 | 0.00103426  |         |            |                      |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_29585_00007:
  accuracy: 0.303
  date: 2021-07-28_21-25-00
  done: false
  experiment_id: fb2273b6a16e44f0bfefb6762fdb492f
  hostname: 2264d6af0a6a
  iterations_since_restore: 1
  loss: 1.877325047969818
  node_ip: 172.17.0.2
  pid: 1475
  should_checkpoint: true
  time_since_restore: 42.38650846481323
  time_this_iter_s: 42.38650846481323
  time_total_s: 42.38650846481323
  timestamp: 1627507500
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '29585_00007'

(pid=1443) [2,  2000] loss: 2.289
(pid=1471) [1,  6000] loss: 0.772
(pid=1479) [2,  2000] loss: 1.772
(pid=1470) [1,  8000] loss: 0.480
(pid=1422) [2,  2000] loss: 1.383
(pid=1474) [1,  8000] loss: 0.421
(pid=1464) [1,  8000] loss: 0.483
(pid=1482) [1,  8000] loss: 0.418
Result for DEFAULT_29585_00006:
  accuracy: 0.1058
  date: 2021-07-28_21-25-07
  done: false
  experiment_id: 0b71c9f87a174eec9cd929ab59f282e0
  hostname: 2264d6af0a6a
  iterations_since_restore: 2
  loss: 2.3056208351135252
  node_ip: 172.17.0.2
  pid: 1443
  should_checkpoint: true
  time_since_restore: 49.38667154312134
  time_this_iter_s: 22.34101676940918
  time_total_s: 49.38667154312134
  timestamp: 1627507507
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: '29585_00006'

== Status ==
Memory usage on this node: 9.5/240.1 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -2.3056208351135252 | Iter 1.000: -1.977501169204712
Resources requested: 18.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_dd2b559219a27a74ecf8cc41a405d759, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_79fc53ec1198c2fe95cd3008fdb7c606, 0.0/2.0 CPU_group_0_e0cc112d61123b1cab4fc0b65c303c9d, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_0_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_0_79fc53ec1198c2fe95cd3008fdb7c606, 0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_dd2b559219a27a74ecf8cc41a405d759, 0.0/2.0 CPU_group_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_0_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_0_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_e0cc112d61123b1cab4fc0b65c303c9d, 0.0/2.0 CPU_group_0_5adfccbfb6de97e60775c7d4f2e73ce2)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (9 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00001 | RUNNING    |                 |            4 |    8 |   16 | 0.0156001   |         |            |                      |
| DEFAULT_29585_00002 | RUNNING    |                 |            2 |  256 |    4 | 0.000325006 |         |            |                      |
| DEFAULT_29585_00003 | RUNNING    | 172.17.0.2:1422 |           16 |  128 |  256 | 0.00576339  | 1.49397 |     0.461  |                    1 |
| DEFAULT_29585_00004 | RUNNING    |                 |            2 |  128 |  256 | 0.000801667 |         |            |                      |
| DEFAULT_29585_00005 | RUNNING    | 172.17.0.2:1479 |           16 |   16 |   64 | 0.000935427 | 1.9775  |     0.27   |                    1 |
| DEFAULT_29585_00006 | RUNNING    | 172.17.0.2:1443 |           16 |    4 |  256 | 0.0350703   | 2.30562 |     0.1058 |                    2 |
| DEFAULT_29585_00007 | RUNNING    | 172.17.0.2:1475 |            8 |  128 |  256 | 0.00049783  | 1.87733 |     0.303  |                    1 |
| DEFAULT_29585_00008 | RUNNING    |                 |            2 |    4 |  256 | 0.000257535 |         |            |                      |
| DEFAULT_29585_00009 | RUNNING    |                 |            2 |   16 |  256 | 0.00103426  |         |            |                      |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_29585_00005:
  accuracy: 0.4131
  date: 2021-07-28_21-25-07
  done: false
  experiment_id: e9270fc4d818458b9a9ccc4ac3084736
  hostname: 2264d6af0a6a
  iterations_since_restore: 2
  loss: 1.6063717344284059
  node_ip: 172.17.0.2
  pid: 1479
  should_checkpoint: true
  time_since_restore: 49.626917362213135
  time_this_iter_s: 22.016221523284912
  time_total_s: 49.626917362213135
  timestamp: 1627507507
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: '29585_00005'

Result for DEFAULT_29585_00003:
  accuracy: 0.5318
  date: 2021-07-28_21-25-08
  done: false
  experiment_id: e50b2c4b9e054ba9bd23f17f4b53ae05
  hostname: 2264d6af0a6a
  iterations_since_restore: 2
  loss: 1.3067960282325746
  node_ip: 172.17.0.2
  pid: 1422
  should_checkpoint: true
  time_since_restore: 51.129855155944824
  time_this_iter_s: 22.938383102416992
  time_total_s: 51.129855155944824
  timestamp: 1627507508
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: '29585_00003'

(pid=1470) [1, 10000] loss: 0.371
(pid=1474) [1, 10000] loss: 0.322
(pid=1475) [2,  2000] loss: 1.779
(pid=1471) [1,  8000] loss: 0.578
(pid=1482) [1, 10000] loss: 0.318
(pid=1464) [1, 10000] loss: 0.376
(pid=1470) [1, 12000] loss: 0.295
(pid=1474) [1, 12000] loss: 0.272
(pid=1443) [3,  2000] loss: 2.303
(pid=1479) [3,  2000] loss: 1.536
(pid=1422) [3,  2000] loss: 1.245
(pid=1464) [1, 12000] loss: 0.301
(pid=1482) [1, 12000] loss: 0.261
(pid=1471) [1, 10000] loss: 0.462
(pid=1475) [2,  4000] loss: 0.827
Result for DEFAULT_29585_00006:
  accuracy: 0.0961
  date: 2021-07-28_21-25-28
  done: false
  experiment_id: 0b71c9f87a174eec9cd929ab59f282e0
  hostname: 2264d6af0a6a
  iterations_since_restore: 3
  loss: 2.3101513275146486
  node_ip: 172.17.0.2
  pid: 1443
  should_checkpoint: true
  time_since_restore: 70.83084154129028
  time_this_iter_s: 21.444169998168945
  time_total_s: 70.83084154129028
  timestamp: 1627507528
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: '29585_00006'

== Status ==
Memory usage on this node: 9.5/240.1 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.6063717344284059 | Iter 1.000: -1.977501169204712
Resources requested: 18.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_0_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_0_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_dd2b559219a27a74ecf8cc41a405d759, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_0_e0cc112d61123b1cab4fc0b65c303c9d, 0.0/2.0 CPU_group_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_0_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_0_dd2b559219a27a74ecf8cc41a405d759, 0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_e0cc112d61123b1cab4fc0b65c303c9d)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (9 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00001 | RUNNING    |                 |            4 |    8 |   16 | 0.0156001   |         |            |                      |
| DEFAULT_29585_00002 | RUNNING    |                 |            2 |  256 |    4 | 0.000325006 |         |            |                      |
| DEFAULT_29585_00003 | RUNNING    | 172.17.0.2:1422 |           16 |  128 |  256 | 0.00576339  | 1.3068  |     0.5318 |                    2 |
| DEFAULT_29585_00004 | RUNNING    |                 |            2 |  128 |  256 | 0.000801667 |         |            |                      |
| DEFAULT_29585_00005 | RUNNING    | 172.17.0.2:1479 |           16 |   16 |   64 | 0.000935427 | 1.60637 |     0.4131 |                    2 |
| DEFAULT_29585_00006 | RUNNING    | 172.17.0.2:1443 |           16 |    4 |  256 | 0.0350703   | 2.31015 |     0.0961 |                    3 |
| DEFAULT_29585_00007 | RUNNING    | 172.17.0.2:1475 |            8 |  128 |  256 | 0.00049783  | 1.87733 |     0.303  |                    1 |
| DEFAULT_29585_00008 | RUNNING    |                 |            2 |    4 |  256 | 0.000257535 |         |            |                      |
| DEFAULT_29585_00009 | RUNNING    |                 |            2 |   16 |  256 | 0.00103426  |         |            |                      |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_29585_00005:
  accuracy: 0.4628
  date: 2021-07-28_21-25-29
  done: false
  experiment_id: e9270fc4d818458b9a9ccc4ac3084736
  hostname: 2264d6af0a6a
  iterations_since_restore: 3
  loss: 1.4674815274238586
  node_ip: 172.17.0.2
  pid: 1479
  should_checkpoint: true
  time_since_restore: 71.27776837348938
  time_this_iter_s: 21.650851011276245
  time_total_s: 71.27776837348938
  timestamp: 1627507529
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: '29585_00005'

(pid=1470) [1, 14000] loss: 0.242
(pid=1474) [1, 14000] loss: 0.219
Result for DEFAULT_29585_00003:
  accuracy: 0.5547
  date: 2021-07-28_21-25-31
  done: false
  experiment_id: e50b2c4b9e054ba9bd23f17f4b53ae05
  hostname: 2264d6af0a6a
  iterations_since_restore: 3
  loss: 1.255931022453308
  node_ip: 172.17.0.2
  pid: 1422
  should_checkpoint: true
  time_since_restore: 73.5317735671997
  time_this_iter_s: 22.401918411254883
  time_total_s: 73.5317735671997
  timestamp: 1627507531
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: '29585_00003'

Result for DEFAULT_29585_00001:
  accuracy: 0.1015
  date: 2021-07-28_21-25-32
  done: true
  experiment_id: fa9781c03c0145b3a7b3ddc91e63d0d5
  hostname: 2264d6af0a6a
  iterations_since_restore: 1
  loss: 2.3142357900619506
  node_ip: 172.17.0.2
  pid: 1471
  should_checkpoint: true
  time_since_restore: 75.13051223754883
  time_this_iter_s: 75.13051223754883
  time_total_s: 75.13051223754883
  timestamp: 1627507532
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '29585_00001'

(pid=1482) [1, 14000] loss: 0.220
(pid=1464) [1, 14000] loss: 0.258
Result for DEFAULT_29585_00007:
  accuracy: 0.3926
  date: 2021-07-28_21-25-36
  done: true
  experiment_id: fb2273b6a16e44f0bfefb6762fdb492f
  hostname: 2264d6af0a6a
  iterations_since_restore: 2
  loss: 1.6618205209732055
  node_ip: 172.17.0.2
  pid: 1475
  should_checkpoint: true
  time_since_restore: 78.599449634552
  time_this_iter_s: 36.21294116973877
  time_total_s: 78.599449634552
  timestamp: 1627507536
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: '29585_00007'

== Status ==
Memory usage on this node: 8.8/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.6340961277008057 | Iter 1.000: -2.0949212030410767
Resources requested: 16.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_0_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_1d1e09e32907f60e3fca3a44589f66ad, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_e0cc112d61123b1cab4fc0b65c303c9d, 0.0/2.0 CPU_group_0_dd2b559219a27a74ecf8cc41a405d759, 0.0/2.0 CPU_group_0_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_dd2b559219a27a74ecf8cc41a405d759, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_e0cc112d61123b1cab4fc0b65c303c9d, 0.0/2.0 CPU_group_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_0_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_0_5adfccbfb6de97e60775c7d4f2e73ce2)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (8 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00002 | RUNNING    |                 |            2 |  256 |    4 | 0.000325006 |         |            |                      |
| DEFAULT_29585_00003 | RUNNING    | 172.17.0.2:1422 |           16 |  128 |  256 | 0.00576339  | 1.25593 |     0.5547 |                    3 |
| DEFAULT_29585_00004 | RUNNING    |                 |            2 |  128 |  256 | 0.000801667 |         |            |                      |
| DEFAULT_29585_00005 | RUNNING    | 172.17.0.2:1479 |           16 |   16 |   64 | 0.000935427 | 1.46748 |     0.4628 |                    3 |
| DEFAULT_29585_00006 | RUNNING    | 172.17.0.2:1443 |           16 |    4 |  256 | 0.0350703   | 2.31015 |     0.0961 |                    3 |
| DEFAULT_29585_00007 | RUNNING    | 172.17.0.2:1475 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
| DEFAULT_29585_00008 | RUNNING    |                 |            2 |    4 |  256 | 0.000257535 |         |            |                      |
| DEFAULT_29585_00009 | RUNNING    |                 |            2 |   16 |  256 | 0.00103426  |         |            |                      |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1470) [1, 16000] loss: 0.202
(pid=1474) [1, 16000] loss: 0.193
(pid=1443) [4,  2000] loss: 2.307
(pid=1479) [4,  2000] loss: 1.424
(pid=1482) [1, 16000] loss: 0.186
(pid=1464) [1, 16000] loss: 0.219
(pid=1422) [4,  2000] loss: 1.150
(pid=1470) [1, 18000] loss: 0.180
(pid=1474) [1, 18000] loss: 0.172
Result for DEFAULT_29585_00006:
  accuracy: 0.0961
  date: 2021-07-28_21-25-48
  done: false
  experiment_id: 0b71c9f87a174eec9cd929ab59f282e0
  hostname: 2264d6af0a6a
  iterations_since_restore: 4
  loss: 2.3074669498443603
  node_ip: 172.17.0.2
  pid: 1443
  should_checkpoint: true
  time_since_restore: 91.24203896522522
  time_this_iter_s: 20.411197423934937
  time_total_s: 91.24203896522522
  timestamp: 1627507548
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: '29585_00006'

== Status ==
Memory usage on this node: 8.2/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -2.3074669498443603 | Iter 2.000: -1.6340961277008057 | Iter 1.000: -2.0949212030410767
Resources requested: 14.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_0_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_1d1e09e32907f60e3fca3a44589f66ad, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_e0cc112d61123b1cab4fc0b65c303c9d, 0.0/2.0 CPU_group_0_dd2b559219a27a74ecf8cc41a405d759, 0.0/2.0 CPU_group_0_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_dd2b559219a27a74ecf8cc41a405d759, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_e0cc112d61123b1cab4fc0b65c303c9d, 0.0/2.0 CPU_group_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_0_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_0_5adfccbfb6de97e60775c7d4f2e73ce2)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00002 | RUNNING    |                 |            2 |  256 |    4 | 0.000325006 |         |            |                      |
| DEFAULT_29585_00003 | RUNNING    | 172.17.0.2:1422 |           16 |  128 |  256 | 0.00576339  | 1.25593 |     0.5547 |                    3 |
| DEFAULT_29585_00004 | RUNNING    |                 |            2 |  128 |  256 | 0.000801667 |         |            |                      |
| DEFAULT_29585_00005 | RUNNING    | 172.17.0.2:1479 |           16 |   16 |   64 | 0.000935427 | 1.46748 |     0.4628 |                    3 |
| DEFAULT_29585_00006 | RUNNING    | 172.17.0.2:1443 |           16 |    4 |  256 | 0.0350703   | 2.30747 |     0.0961 |                    4 |
| DEFAULT_29585_00008 | RUNNING    |                 |            2 |    4 |  256 | 0.000257535 |         |            |                      |
| DEFAULT_29585_00009 | RUNNING    |                 |            2 |   16 |  256 | 0.00103426  |         |            |                      |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_29585_00005:
  accuracy: 0.4992
  date: 2021-07-28_21-25-49
  done: false
  experiment_id: e9270fc4d818458b9a9ccc4ac3084736
  hostname: 2264d6af0a6a
  iterations_since_restore: 4
  loss: 1.3946209341049194
  node_ip: 172.17.0.2
  pid: 1479
  should_checkpoint: true
  time_since_restore: 91.93775177001953
  time_this_iter_s: 20.65998339653015
  time_total_s: 91.93775177001953
  timestamp: 1627507549
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: '29585_00005'

Result for DEFAULT_29585_00003:
  accuracy: 0.5629
  date: 2021-07-28_21-25-52
  done: false
  experiment_id: e50b2c4b9e054ba9bd23f17f4b53ae05
  hostname: 2264d6af0a6a
  iterations_since_restore: 4
  loss: 1.2356455825805663
  node_ip: 172.17.0.2
  pid: 1422
  should_checkpoint: true
  time_since_restore: 94.41662836074829
  time_this_iter_s: 20.884854793548584
  time_total_s: 94.41662836074829
  timestamp: 1627507552
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: '29585_00003'

(pid=1482) [1, 18000] loss: 0.163
(pid=1464) [1, 18000] loss: 0.193
(pid=1470) [1, 20000] loss: 0.158
(pid=1474) [1, 20000] loss: 0.149
(pid=1482) [1, 20000] loss: 0.142
(pid=1464) [1, 20000] loss: 0.173
(pid=1443) [5,  2000] loss: 2.307
(pid=1479) [5,  2000] loss: 1.345
(pid=1422) [5,  2000] loss: 1.073
Result for DEFAULT_29585_00008:
  accuracy: 0.4328
  date: 2021-07-28_21-26-08
  done: false
  experiment_id: 5f0894680a004816aa2aa41fc07e39d3
  hostname: 2264d6af0a6a
  iterations_since_restore: 1
  loss: 1.5323000041723251
  node_ip: 172.17.0.2
  pid: 1470
  should_checkpoint: true
  time_since_restore: 111.01790618896484
  time_this_iter_s: 111.01790618896484
  time_total_s: 111.01790618896484
  timestamp: 1627507568
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '29585_00008'

== Status ==
Memory usage on this node: 8.2/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.3946209341049194 | Iter 2.000: -1.6340961277008057 | Iter 1.000: -1.977501169204712
Resources requested: 14.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_0_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_0_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_0_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_1d1e09e32907f60e3fca3a44589f66ad)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00002 | RUNNING    |                 |            2 |  256 |    4 | 0.000325006 |         |            |                      |
| DEFAULT_29585_00003 | RUNNING    | 172.17.0.2:1422 |           16 |  128 |  256 | 0.00576339  | 1.23565 |     0.5629 |                    4 |
| DEFAULT_29585_00004 | RUNNING    |                 |            2 |  128 |  256 | 0.000801667 |         |            |                      |
| DEFAULT_29585_00005 | RUNNING    | 172.17.0.2:1479 |           16 |   16 |   64 | 0.000935427 | 1.39462 |     0.4992 |                    4 |
| DEFAULT_29585_00006 | RUNNING    | 172.17.0.2:1443 |           16 |    4 |  256 | 0.0350703   | 2.30747 |     0.0961 |                    4 |
| DEFAULT_29585_00008 | RUNNING    | 172.17.0.2:1470 |            2 |    4 |  256 | 0.000257535 | 1.5323  |     0.4328 |                    1 |
| DEFAULT_29585_00009 | RUNNING    |                 |            2 |   16 |  256 | 0.00103426  |         |            |                      |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_29585_00009:
  accuracy: 0.4346
  date: 2021-07-28_21-26-09
  done: false
  experiment_id: ea64a6e7d70a4ad8871632d3f7b5d378
  hostname: 2264d6af0a6a
  iterations_since_restore: 1
  loss: 1.5396203134074806
  node_ip: 172.17.0.2
  pid: 1474
  should_checkpoint: true
  time_since_restore: 111.7411732673645
  time_this_iter_s: 111.7411732673645
  time_total_s: 111.7411732673645
  timestamp: 1627507569
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '29585_00009'

Result for DEFAULT_29585_00006:
  accuracy: 0.0996
  date: 2021-07-28_21-26-09
  done: false
  experiment_id: 0b71c9f87a174eec9cd929ab59f282e0
  hostname: 2264d6af0a6a
  iterations_since_restore: 5
  loss: 2.306953881072998
  node_ip: 172.17.0.2
  pid: 1443
  should_checkpoint: true
  time_since_restore: 112.00499701499939
  time_this_iter_s: 20.76295804977417
  time_total_s: 112.00499701499939
  timestamp: 1627507569
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: '29585_00006'

Result for DEFAULT_29585_00005:
  accuracy: 0.5209
  date: 2021-07-28_21-26-10
  done: false
  experiment_id: e9270fc4d818458b9a9ccc4ac3084736
  hostname: 2264d6af0a6a
  iterations_since_restore: 5
  loss: 1.3264849575996398
  node_ip: 172.17.0.2
  pid: 1479
  should_checkpoint: true
  time_since_restore: 112.31328916549683
  time_this_iter_s: 20.375537395477295
  time_total_s: 112.31328916549683
  timestamp: 1627507570
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: '29585_00005'

Result for DEFAULT_29585_00003:
  accuracy: 0.5829
  date: 2021-07-28_21-26-13
  done: false
  experiment_id: e50b2c4b9e054ba9bd23f17f4b53ae05
  hostname: 2264d6af0a6a
  iterations_since_restore: 5
  loss: 1.2015536951065064
  node_ip: 172.17.0.2
  pid: 1422
  should_checkpoint: true
  time_since_restore: 115.83827805519104
  time_this_iter_s: 21.42164969444275
  time_total_s: 115.83827805519104
  timestamp: 1627507573
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: '29585_00003'

Result for DEFAULT_29585_00004:
  accuracy: 0.4866
  date: 2021-07-28_21-26-15
  done: false
  experiment_id: 3cff2c4dcd814197bcabbb3e417d8353
  hostname: 2264d6af0a6a
  iterations_since_restore: 1
  loss: 1.4023548944108188
  node_ip: 172.17.0.2
  pid: 1482
  should_checkpoint: true
  time_since_restore: 117.99768018722534
  time_this_iter_s: 117.99768018722534
  time_total_s: 117.99768018722534
  timestamp: 1627507575
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '29585_00004'

== Status ==
Memory usage on this node: 8.2/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.3946209341049194 | Iter 2.000: -1.6340961277008057 | Iter 1.000: -1.877325047969818
Resources requested: 14.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_0_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_1d1e09e32907f60e3fca3a44589f66ad, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_0_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00002 | RUNNING    |                 |            2 |  256 |    4 | 0.000325006 |         |            |                      |
| DEFAULT_29585_00003 | RUNNING    | 172.17.0.2:1422 |           16 |  128 |  256 | 0.00576339  | 1.20155 |     0.5829 |                    5 |
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.40235 |     0.4866 |                    1 |
| DEFAULT_29585_00005 | RUNNING    | 172.17.0.2:1479 |           16 |   16 |   64 | 0.000935427 | 1.32648 |     0.5209 |                    5 |
| DEFAULT_29585_00006 | RUNNING    | 172.17.0.2:1443 |           16 |    4 |  256 | 0.0350703   | 2.30695 |     0.0996 |                    5 |
| DEFAULT_29585_00008 | RUNNING    | 172.17.0.2:1470 |            2 |    4 |  256 | 0.000257535 | 1.5323  |     0.4328 |                    1 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.53962 |     0.4346 |                    1 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_29585_00002:
  accuracy: 0.3723
  date: 2021-07-28_21-26-16
  done: false
  experiment_id: 4ee5737261e54e37b0b08cfdcd74ddf4
  hostname: 2264d6af0a6a
  iterations_since_restore: 1
  loss: 1.6857840451717376
  node_ip: 172.17.0.2
  pid: 1464
  should_checkpoint: true
  time_since_restore: 118.81487035751343
  time_this_iter_s: 118.81487035751343
  time_total_s: 118.81487035751343
  timestamp: 1627507576
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '29585_00002'

(pid=1470) [2,  2000] loss: 1.570
(pid=1474) [2,  2000] loss: 1.495
(pid=1443) [6,  2000] loss: 2.308
(pid=1479) [6,  2000] loss: 1.269
(pid=1482) [2,  2000] loss: 1.397
(pid=1464) [2,  2000] loss: 1.691
(pid=1470) [2,  4000] loss: 0.766
(pid=1474) [2,  4000] loss: 0.722
(pid=1422) [6,  2000] loss: 1.011
Result for DEFAULT_29585_00006:
  accuracy: 0.0985
  date: 2021-07-28_21-26-30
  done: false
  experiment_id: 0b71c9f87a174eec9cd929ab59f282e0
  hostname: 2264d6af0a6a
  iterations_since_restore: 6
  loss: 2.3054874824523925
  node_ip: 172.17.0.2
  pid: 1443
  should_checkpoint: true
  time_since_restore: 132.31310272216797
  time_this_iter_s: 20.30810570716858
  time_total_s: 132.31310272216797
  timestamp: 1627507590
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: '29585_00006'

== Status ==
Memory usage on this node: 8.3/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.3946209341049194 | Iter 2.000: -1.6340961277008057 | Iter 1.000: -1.7815545465707778
Resources requested: 14.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_0_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_1d1e09e32907f60e3fca3a44589f66ad, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_0_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00002 | RUNNING    | 172.17.0.2:1464 |            2 |  256 |    4 | 0.000325006 | 1.68578 |     0.3723 |                    1 |
| DEFAULT_29585_00003 | RUNNING    | 172.17.0.2:1422 |           16 |  128 |  256 | 0.00576339  | 1.20155 |     0.5829 |                    5 |
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.40235 |     0.4866 |                    1 |
| DEFAULT_29585_00005 | RUNNING    | 172.17.0.2:1479 |           16 |   16 |   64 | 0.000935427 | 1.32648 |     0.5209 |                    5 |
| DEFAULT_29585_00006 | RUNNING    | 172.17.0.2:1443 |           16 |    4 |  256 | 0.0350703   | 2.30549 |     0.0985 |                    6 |
| DEFAULT_29585_00008 | RUNNING    | 172.17.0.2:1470 |            2 |    4 |  256 | 0.000257535 | 1.5323  |     0.4328 |                    1 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.53962 |     0.4346 |                    1 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_29585_00005:
  accuracy: 0.5438
  date: 2021-07-28_21-26-30
  done: false
  experiment_id: e9270fc4d818458b9a9ccc4ac3084736
  hostname: 2264d6af0a6a
  iterations_since_restore: 6
  loss: 1.2748261854171752
  node_ip: 172.17.0.2
  pid: 1479
  should_checkpoint: true
  time_since_restore: 132.53567552566528
  time_this_iter_s: 20.222386360168457
  time_total_s: 132.53567552566528
  timestamp: 1627507590
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: '29585_00005'

(pid=1482) [2,  4000] loss: 0.675
(pid=1470) [2,  6000] loss: 0.499
Result for DEFAULT_29585_00003:
  accuracy: 0.5933
  date: 2021-07-28_21-26-34
  done: false
  experiment_id: e50b2c4b9e054ba9bd23f17f4b53ae05
  hostname: 2264d6af0a6a
  iterations_since_restore: 6
  loss: 1.155671979522705
  node_ip: 172.17.0.2
  pid: 1422
  should_checkpoint: true
  time_since_restore: 136.9269895553589
  time_this_iter_s: 21.088711500167847
  time_total_s: 136.9269895553589
  timestamp: 1627507594
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: '29585_00003'

(pid=1474) [2,  6000] loss: 0.483
(pid=1464) [2,  4000] loss: 0.837
(pid=1470) [2,  8000] loss: 0.371
(pid=1482) [2,  6000] loss: 0.450
(pid=1474) [2,  8000] loss: 0.358
(pid=1443) [7,  2000] loss: 2.308
(pid=1464) [2,  6000] loss: 0.544
(pid=1479) [7,  2000] loss: 1.213
(pid=1422) [7,  2000] loss: 0.959
Result for DEFAULT_29585_00006:
  accuracy: 0.0961
  date: 2021-07-28_21-26-50
  done: false
  experiment_id: 0b71c9f87a174eec9cd929ab59f282e0
  hostname: 2264d6af0a6a
  iterations_since_restore: 7
  loss: 2.3085534187316896
  node_ip: 172.17.0.2
  pid: 1443
  should_checkpoint: true
  time_since_restore: 152.33232474327087
  time_this_iter_s: 20.019222021102905
  time_total_s: 152.33232474327087
  timestamp: 1627507610
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: '29585_00006'

== Status ==
Memory usage on this node: 8.3/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.3946209341049194 | Iter 2.000: -1.6340961277008057 | Iter 1.000: -1.7815545465707778
Resources requested: 14.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_0_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_1d1e09e32907f60e3fca3a44589f66ad, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_0_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00002 | RUNNING    | 172.17.0.2:1464 |            2 |  256 |    4 | 0.000325006 | 1.68578 |     0.3723 |                    1 |
| DEFAULT_29585_00003 | RUNNING    | 172.17.0.2:1422 |           16 |  128 |  256 | 0.00576339  | 1.15567 |     0.5933 |                    6 |
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.40235 |     0.4866 |                    1 |
| DEFAULT_29585_00005 | RUNNING    | 172.17.0.2:1479 |           16 |   16 |   64 | 0.000935427 | 1.27483 |     0.5438 |                    6 |
| DEFAULT_29585_00006 | RUNNING    | 172.17.0.2:1443 |           16 |    4 |  256 | 0.0350703   | 2.30855 |     0.0961 |                    7 |
| DEFAULT_29585_00008 | RUNNING    | 172.17.0.2:1470 |            2 |    4 |  256 | 0.000257535 | 1.5323  |     0.4328 |                    1 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.53962 |     0.4346 |                    1 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_29585_00005:
  accuracy: 0.5555
  date: 2021-07-28_21-26-50
  done: false
  experiment_id: e9270fc4d818458b9a9ccc4ac3084736
  hostname: 2264d6af0a6a
  iterations_since_restore: 7
  loss: 1.241218276643753
  node_ip: 172.17.0.2
  pid: 1479
  should_checkpoint: true
  time_since_restore: 152.65327906608582
  time_this_iter_s: 20.117603540420532
  time_total_s: 152.65327906608582
  timestamp: 1627507610
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: '29585_00005'

(pid=1470) [2, 10000] loss: 0.294
(pid=1474) [2, 10000] loss: 0.286
(pid=1482) [2,  8000] loss: 0.330
(pid=1464) [2,  8000] loss: 0.403
Result for DEFAULT_29585_00003:
  accuracy: 0.6014
  date: 2021-07-28_21-26-55
  done: false
  experiment_id: e50b2c4b9e054ba9bd23f17f4b53ae05
  hostname: 2264d6af0a6a
  iterations_since_restore: 7
  loss: 1.1640394621372223
  node_ip: 172.17.0.2
  pid: 1422
  should_checkpoint: true
  time_since_restore: 157.7347710132599
  time_this_iter_s: 20.807781457901
  time_total_s: 157.7347710132599
  timestamp: 1627507615
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: '29585_00003'

== Status ==
Memory usage on this node: 8.3/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.3946209341049194 | Iter 2.000: -1.6340961277008057 | Iter 1.000: -1.7815545465707778
Resources requested: 14.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_0_1d1e09e32907f60e3fca3a44589f66ad)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00002 | RUNNING    | 172.17.0.2:1464 |            2 |  256 |    4 | 0.000325006 | 1.68578 |     0.3723 |                    1 |
| DEFAULT_29585_00003 | RUNNING    | 172.17.0.2:1422 |           16 |  128 |  256 | 0.00576339  | 1.16404 |     0.6014 |                    7 |
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.40235 |     0.4866 |                    1 |
| DEFAULT_29585_00005 | RUNNING    | 172.17.0.2:1479 |           16 |   16 |   64 | 0.000935427 | 1.24122 |     0.5555 |                    7 |
| DEFAULT_29585_00006 | RUNNING    | 172.17.0.2:1443 |           16 |    4 |  256 | 0.0350703   | 2.30855 |     0.0961 |                    7 |
| DEFAULT_29585_00008 | RUNNING    | 172.17.0.2:1470 |            2 |    4 |  256 | 0.000257535 | 1.5323  |     0.4328 |                    1 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.53962 |     0.4346 |                    1 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1470) [2, 12000] loss: 0.243
(pid=1474) [2, 12000] loss: 0.240
(pid=1482) [2, 10000] loss: 0.262
(pid=1464) [2, 10000] loss: 0.320
(pid=1443) [8,  2000] loss: 2.307
(pid=1479) [8,  2000] loss: 1.168
(pid=1470) [2, 14000] loss: 0.206
(pid=1474) [2, 14000] loss: 0.202
(pid=1422) [8,  2000] loss: 0.919
Result for DEFAULT_29585_00006:
  accuracy: 0.1055
  date: 2021-07-28_21-27-10
  done: false
  experiment_id: 0b71c9f87a174eec9cd929ab59f282e0
  hostname: 2264d6af0a6a
  iterations_since_restore: 8
  loss: 2.3053534370422364
  node_ip: 172.17.0.2
  pid: 1443
  should_checkpoint: true
  time_since_restore: 172.3896210193634
  time_this_iter_s: 20.05729627609253
  time_total_s: 172.3896210193634
  timestamp: 1627507630
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: '29585_00006'

== Status ==
Memory usage on this node: 8.3/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -2.3053534370422364 | Iter 4.000: -1.3946209341049194 | Iter 2.000: -1.6340961277008057 | Iter 1.000: -1.7815545465707778
Resources requested: 14.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_0_1d1e09e32907f60e3fca3a44589f66ad)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00002 | RUNNING    | 172.17.0.2:1464 |            2 |  256 |    4 | 0.000325006 | 1.68578 |     0.3723 |                    1 |
| DEFAULT_29585_00003 | RUNNING    | 172.17.0.2:1422 |           16 |  128 |  256 | 0.00576339  | 1.16404 |     0.6014 |                    7 |
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.40235 |     0.4866 |                    1 |
| DEFAULT_29585_00005 | RUNNING    | 172.17.0.2:1479 |           16 |   16 |   64 | 0.000935427 | 1.24122 |     0.5555 |                    7 |
| DEFAULT_29585_00006 | RUNNING    | 172.17.0.2:1443 |           16 |    4 |  256 | 0.0350703   | 2.30535 |     0.1055 |                    8 |
| DEFAULT_29585_00008 | RUNNING    | 172.17.0.2:1470 |            2 |    4 |  256 | 0.000257535 | 1.5323  |     0.4328 |                    1 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.53962 |     0.4346 |                    1 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1482) [2, 12000] loss: 0.218
Result for DEFAULT_29585_00005:
  accuracy: 0.5717
  date: 2021-07-28_21-27-10
  done: false
  experiment_id: e9270fc4d818458b9a9ccc4ac3084736
  hostname: 2264d6af0a6a
  iterations_since_restore: 8
  loss: 1.214192063331604
  node_ip: 172.17.0.2
  pid: 1479
  should_checkpoint: true
  time_since_restore: 172.7827067375183
  time_this_iter_s: 20.129427671432495
  time_total_s: 172.7827067375183
  timestamp: 1627507630
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: '29585_00005'

(pid=1464) [2, 12000] loss: 0.263
(pid=1470) [2, 16000] loss: 0.182
Result for DEFAULT_29585_00003:
  accuracy: 0.6041
  date: 2021-07-28_21-27-16
  done: false
  experiment_id: e50b2c4b9e054ba9bd23f17f4b53ae05
  hostname: 2264d6af0a6a
  iterations_since_restore: 8
  loss: 1.1856979976177215
  node_ip: 172.17.0.2
  pid: 1422
  should_checkpoint: true
  time_since_restore: 178.51454830169678
  time_this_iter_s: 20.77977728843689
  time_total_s: 178.51454830169678
  timestamp: 1627507636
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: '29585_00003'

== Status ==
Memory usage on this node: 8.3/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -1.214192063331604 | Iter 4.000: -1.3946209341049194 | Iter 2.000: -1.6340961277008057 | Iter 1.000: -1.7815545465707778
Resources requested: 14.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_0_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00002 | RUNNING    | 172.17.0.2:1464 |            2 |  256 |    4 | 0.000325006 | 1.68578 |     0.3723 |                    1 |
| DEFAULT_29585_00003 | RUNNING    | 172.17.0.2:1422 |           16 |  128 |  256 | 0.00576339  | 1.1857  |     0.6041 |                    8 |
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.40235 |     0.4866 |                    1 |
| DEFAULT_29585_00005 | RUNNING    | 172.17.0.2:1479 |           16 |   16 |   64 | 0.000935427 | 1.21419 |     0.5717 |                    8 |
| DEFAULT_29585_00006 | RUNNING    | 172.17.0.2:1443 |           16 |    4 |  256 | 0.0350703   | 2.30535 |     0.1055 |                    8 |
| DEFAULT_29585_00008 | RUNNING    | 172.17.0.2:1470 |            2 |    4 |  256 | 0.000257535 | 1.5323  |     0.4328 |                    1 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.53962 |     0.4346 |                    1 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1474) [2, 16000] loss: 0.177
(pid=1482) [2, 14000] loss: 0.184
(pid=1464) [2, 14000] loss: 0.226
(pid=1443) [9,  2000] loss: 2.308
(pid=1470) [2, 18000] loss: 0.160
(pid=1479) [9,  2000] loss: 1.147
(pid=1474) [2, 18000] loss: 0.155
(pid=1482) [2, 16000] loss: 0.163
(pid=1464) [2, 16000] loss: 0.196
Result for DEFAULT_29585_00006:
  accuracy: 0.0963
  date: 2021-07-28_21-27-30
  done: false
  experiment_id: 0b71c9f87a174eec9cd929ab59f282e0
  hostname: 2264d6af0a6a
  iterations_since_restore: 9
  loss: 2.3037846210479738
  node_ip: 172.17.0.2
  pid: 1443
  should_checkpoint: true
  time_since_restore: 192.45000195503235
  time_this_iter_s: 20.060380935668945
  time_total_s: 192.45000195503235
  timestamp: 1627507650
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: '29585_00006'

== Status ==
Memory usage on this node: 8.3/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -1.214192063331604 | Iter 4.000: -1.3946209341049194 | Iter 2.000: -1.6340961277008057 | Iter 1.000: -1.7815545465707778
Resources requested: 14.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_0_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00002 | RUNNING    | 172.17.0.2:1464 |            2 |  256 |    4 | 0.000325006 | 1.68578 |     0.3723 |                    1 |
| DEFAULT_29585_00003 | RUNNING    | 172.17.0.2:1422 |           16 |  128 |  256 | 0.00576339  | 1.1857  |     0.6041 |                    8 |
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.40235 |     0.4866 |                    1 |
| DEFAULT_29585_00005 | RUNNING    | 172.17.0.2:1479 |           16 |   16 |   64 | 0.000935427 | 1.21419 |     0.5717 |                    8 |
| DEFAULT_29585_00006 | RUNNING    | 172.17.0.2:1443 |           16 |    4 |  256 | 0.0350703   | 2.30378 |     0.0963 |                    9 |
| DEFAULT_29585_00008 | RUNNING    | 172.17.0.2:1470 |            2 |    4 |  256 | 0.000257535 | 1.5323  |     0.4328 |                    1 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.53962 |     0.4346 |                    1 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1422) [9,  2000] loss: 0.881
Result for DEFAULT_29585_00005:
  accuracy: 0.5742
  date: 2021-07-28_21-27-30
  done: false
  experiment_id: e9270fc4d818458b9a9ccc4ac3084736
  hostname: 2264d6af0a6a
  iterations_since_restore: 9
  loss: 1.2065563109874726
  node_ip: 172.17.0.2
  pid: 1479
  should_checkpoint: true
  time_since_restore: 193.01840925216675
  time_this_iter_s: 20.235702514648438
  time_total_s: 193.01840925216675
  timestamp: 1627507650
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: '29585_00005'

(pid=1470) [2, 20000] loss: 0.145
(pid=1474) [2, 20000] loss: 0.142
Result for DEFAULT_29585_00003:
  accuracy: 0.6037
  date: 2021-07-28_21-27-37
  done: false
  experiment_id: e50b2c4b9e054ba9bd23f17f4b53ae05
  hostname: 2264d6af0a6a
  iterations_since_restore: 9
  loss: 1.2358031953811646
  node_ip: 172.17.0.2
  pid: 1422
  should_checkpoint: true
  time_since_restore: 199.46260905265808
  time_this_iter_s: 20.948060750961304
  time_total_s: 199.46260905265808
  timestamp: 1627507657
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: '29585_00003'

== Status ==
Memory usage on this node: 8.2/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -1.214192063331604 | Iter 4.000: -1.3946209341049194 | Iter 2.000: -1.6340961277008057 | Iter 1.000: -1.7815545465707778
Resources requested: 14.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_0_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_1d1e09e32907f60e3fca3a44589f66ad)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00002 | RUNNING    | 172.17.0.2:1464 |            2 |  256 |    4 | 0.000325006 | 1.68578 |     0.3723 |                    1 |
| DEFAULT_29585_00003 | RUNNING    | 172.17.0.2:1422 |           16 |  128 |  256 | 0.00576339  | 1.2358  |     0.6037 |                    9 |
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.40235 |     0.4866 |                    1 |
| DEFAULT_29585_00005 | RUNNING    | 172.17.0.2:1479 |           16 |   16 |   64 | 0.000935427 | 1.20656 |     0.5742 |                    9 |
| DEFAULT_29585_00006 | RUNNING    | 172.17.0.2:1443 |           16 |    4 |  256 | 0.0350703   | 2.30378 |     0.0963 |                    9 |
| DEFAULT_29585_00008 | RUNNING    | 172.17.0.2:1470 |            2 |    4 |  256 | 0.000257535 | 1.5323  |     0.4328 |                    1 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.53962 |     0.4346 |                    1 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1482) [2, 18000] loss: 0.143
(pid=1464) [2, 18000] loss: 0.169
(pid=1443) [10,  2000] loss: 2.308
(pid=1479) [10,  2000] loss: 1.118
Result for DEFAULT_29585_00008:
  accuracy: 0.4596
  date: 2021-07-28_21-27-46
  done: false
  experiment_id: 5f0894680a004816aa2aa41fc07e39d3
  hostname: 2264d6af0a6a
  iterations_since_restore: 2
  loss: 1.4904517725475133
  node_ip: 172.17.0.2
  pid: 1470
  should_checkpoint: true
  time_since_restore: 208.33406853675842
  time_this_iter_s: 97.31616234779358
  time_total_s: 208.33406853675842
  timestamp: 1627507666
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: '29585_00008'

== Status ==
Memory usage on this node: 8.3/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -1.214192063331604 | Iter 4.000: -1.3946209341049194 | Iter 2.000: -1.6063717344284059 | Iter 1.000: -1.7815545465707778
Resources requested: 14.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_0_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_1d1e09e32907f60e3fca3a44589f66ad)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00002 | RUNNING    | 172.17.0.2:1464 |            2 |  256 |    4 | 0.000325006 | 1.68578 |     0.3723 |                    1 |
| DEFAULT_29585_00003 | RUNNING    | 172.17.0.2:1422 |           16 |  128 |  256 | 0.00576339  | 1.2358  |     0.6037 |                    9 |
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.40235 |     0.4866 |                    1 |
| DEFAULT_29585_00005 | RUNNING    | 172.17.0.2:1479 |           16 |   16 |   64 | 0.000935427 | 1.20656 |     0.5742 |                    9 |
| DEFAULT_29585_00006 | RUNNING    | 172.17.0.2:1443 |           16 |    4 |  256 | 0.0350703   | 2.30378 |     0.0963 |                    9 |
| DEFAULT_29585_00008 | RUNNING    | 172.17.0.2:1470 |            2 |    4 |  256 | 0.000257535 | 1.49045 |     0.4596 |                    2 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.53962 |     0.4346 |                    1 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1482) [2, 20000] loss: 0.125
Result for DEFAULT_29585_00009:
  accuracy: 0.482
  date: 2021-07-28_21-27-46
  done: false
  experiment_id: ea64a6e7d70a4ad8871632d3f7b5d378
  hostname: 2264d6af0a6a
  iterations_since_restore: 2
  loss: 1.4378724331121893
  node_ip: 172.17.0.2
  pid: 1474
  should_checkpoint: true
  time_since_restore: 208.90379214286804
  time_this_iter_s: 97.16261887550354
  time_total_s: 208.90379214286804
  timestamp: 1627507666
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: '29585_00009'

(pid=1464) [2, 20000] loss: 0.151
Result for DEFAULT_29585_00006:
  accuracy: 0.0961
  date: 2021-07-28_21-27-50
  done: true
  experiment_id: 0b71c9f87a174eec9cd929ab59f282e0
  hostname: 2264d6af0a6a
  iterations_since_restore: 10
  loss: 2.3160431800842285
  node_ip: 172.17.0.2
  pid: 1443
  should_checkpoint: true
  time_since_restore: 213.04612159729004
  time_this_iter_s: 20.59611964225769
  time_total_s: 213.04612159729004
  timestamp: 1627507670
  timesteps_since_restore: 0
  training_iteration: 10
  trial_id: '29585_00006'

Result for DEFAULT_29585_00005:
  accuracy: 0.5843
  date: 2021-07-28_21-27-51
  done: true
  experiment_id: e9270fc4d818458b9a9ccc4ac3084736
  hostname: 2264d6af0a6a
  iterations_since_restore: 10
  loss: 1.190362274312973
  node_ip: 172.17.0.2
  pid: 1479
  should_checkpoint: true
  time_since_restore: 213.69842147827148
  time_this_iter_s: 20.680012226104736
  time_total_s: 213.69842147827148
  timestamp: 1627507671
  timesteps_since_restore: 0
  training_iteration: 10
  trial_id: '29585_00005'

== Status ==
Memory usage on this node: 7.6/240.1 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -1.214192063331604 | Iter 4.000: -1.3946209341049194 | Iter 2.000: -1.5484117534879596 | Iter 1.000: -1.7815545465707778
Resources requested: 12.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_0_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_0_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_0_1d1e09e32907f60e3fca3a44589f66ad)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (6 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00002 | RUNNING    | 172.17.0.2:1464 |            2 |  256 |    4 | 0.000325006 | 1.68578 |     0.3723 |                    1 |
| DEFAULT_29585_00003 | RUNNING    | 172.17.0.2:1422 |           16 |  128 |  256 | 0.00576339  | 1.2358  |     0.6037 |                    9 |
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.40235 |     0.4866 |                    1 |
| DEFAULT_29585_00005 | RUNNING    | 172.17.0.2:1479 |           16 |   16 |   64 | 0.000935427 | 1.19036 |     0.5843 |                   10 |
| DEFAULT_29585_00008 | RUNNING    | 172.17.0.2:1470 |            2 |    4 |  256 | 0.000257535 | 1.49045 |     0.4596 |                    2 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.43787 |     0.482  |                    2 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00006 | TERMINATED |                 |           16 |    4 |  256 | 0.0350703   | 2.31604 |     0.0961 |                   10 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1422) [10,  2000] loss: 0.857
(pid=1470) [3,  2000] loss: 1.385
(pid=1474) [3,  2000] loss: 1.374
Result for DEFAULT_29585_00003:
  accuracy: 0.5904
  date: 2021-07-28_21-27-57
  done: true
  experiment_id: e50b2c4b9e054ba9bd23f17f4b53ae05
  hostname: 2264d6af0a6a
  iterations_since_restore: 10
  loss: 1.2767406641483308
  node_ip: 172.17.0.2
  pid: 1422
  should_checkpoint: true
  time_since_restore: 220.16486930847168
  time_this_iter_s: 20.7022602558136
  time_total_s: 220.16486930847168
  timestamp: 1627507677
  timesteps_since_restore: 0
  training_iteration: 10
  trial_id: '29585_00003'

== Status ==
Memory usage on this node: 7.0/240.1 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.214192063331604 | Iter 4.000: -1.3946209341049194 | Iter 2.000: -1.5484117534879596 | Iter 1.000: -1.7815545465707778
Resources requested: 10.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_0_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_5adfccbfb6de97e60775c7d4f2e73ce2, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_dcb7bcd64281ed5b972dd72af63be9a9, 0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_0_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_0_1d1e09e32907f60e3fca3a44589f66ad)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (5 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00002 | RUNNING    | 172.17.0.2:1464 |            2 |  256 |    4 | 0.000325006 | 1.68578 |     0.3723 |                    1 |
| DEFAULT_29585_00003 | RUNNING    | 172.17.0.2:1422 |           16 |  128 |  256 | 0.00576339  | 1.27674 |     0.5904 |                   10 |
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.40235 |     0.4866 |                    1 |
| DEFAULT_29585_00008 | RUNNING    | 172.17.0.2:1470 |            2 |    4 |  256 | 0.000257535 | 1.49045 |     0.4596 |                    2 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.43787 |     0.482  |                    2 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00005 | TERMINATED |                 |           16 |   16 |   64 | 0.000935427 | 1.19036 |     0.5843 |                   10 |
| DEFAULT_29585_00006 | TERMINATED |                 |           16 |    4 |  256 | 0.0350703   | 2.31604 |     0.0961 |                   10 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_29585_00004:
  accuracy: 0.5586
  date: 2021-07-28_21-27-59
  done: false
  experiment_id: 3cff2c4dcd814197bcabbb3e417d8353
  hostname: 2264d6af0a6a
  iterations_since_restore: 2
  loss: 1.2445484913449734
  node_ip: 172.17.0.2
  pid: 1482
  should_checkpoint: true
  time_since_restore: 221.4720482826233
  time_this_iter_s: 103.47436809539795
  time_total_s: 221.4720482826233
  timestamp: 1627507679
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: '29585_00004'

Result for DEFAULT_29585_00002:
  accuracy: 0.4458
  date: 2021-07-28_21-28-01
  done: true
  experiment_id: 4ee5737261e54e37b0b08cfdcd74ddf4
  hostname: 2264d6af0a6a
  iterations_since_restore: 2
  loss: 1.5131160246104003
  node_ip: 172.17.0.2
  pid: 1464
  should_checkpoint: true
  time_since_restore: 223.30014181137085
  time_this_iter_s: 104.48527145385742
  time_total_s: 223.30014181137085
  timestamp: 1627507681
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: '29585_00002'

(pid=1470) [3,  4000] loss: 0.694
(pid=1474) [3,  4000] loss: 0.685
(pid=1482) [3,  2000] loss: 1.205
(pid=1470) [3,  6000] loss: 0.477
(pid=1474) [3,  6000] loss: 0.454
(pid=1482) [3,  4000] loss: 0.587
(pid=1470) [3,  8000] loss: 0.358
(pid=1474) [3,  8000] loss: 0.341
(pid=1482) [3,  6000] loss: 0.398
(pid=1470) [3, 10000] loss: 0.285
(pid=1474) [3, 10000] loss: 0.276
(pid=1482) [3,  8000] loss: 0.293
(pid=1470) [3, 12000] loss: 0.233
(pid=1474) [3, 12000] loss: 0.231
(pid=1470) [3, 14000] loss: 0.191
(pid=1482) [3, 10000] loss: 0.240
(pid=1474) [3, 14000] loss: 0.193
(pid=1470) [3, 16000] loss: 0.175
(pid=1474) [3, 16000] loss: 0.173
(pid=1482) [3, 12000] loss: 0.194
(pid=1470) [3, 18000] loss: 0.154
(pid=1474) [3, 18000] loss: 0.155
(pid=1482) [3, 14000] loss: 0.168
(pid=1470) [3, 20000] loss: 0.136
(pid=1474) [3, 20000] loss: 0.137
(pid=1482) [3, 16000] loss: 0.146
Result for DEFAULT_29585_00008:
  accuracy: 0.4876
  date: 2021-07-28_21-29-10
  done: false
  experiment_id: 5f0894680a004816aa2aa41fc07e39d3
  hostname: 2264d6af0a6a
  iterations_since_restore: 3
  loss: 1.41938562374115
  node_ip: 172.17.0.2
  pid: 1470
  should_checkpoint: true
  time_since_restore: 292.88119411468506
  time_this_iter_s: 84.54712557792664
  time_total_s: 292.88119411468506
  timestamp: 1627507750
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: '29585_00008'

== Status ==
Memory usage on this node: 5.8/240.1 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.214192063331604 | Iter 4.000: -1.3946209341049194 | Iter 2.000: -1.5017838985789567 | Iter 1.000: -1.7815545465707778
Resources requested: 6.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_0_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_1d1e09e32907f60e3fca3a44589f66ad, 0.0/2.0 CPU_group_f5588f93d1e2bc1a6aea3263c01a9f3c, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.24455 |     0.5586 |                    2 |
| DEFAULT_29585_00008 | RUNNING    | 172.17.0.2:1470 |            2 |    4 |  256 | 0.000257535 | 1.41939 |     0.4876 |                    3 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.43787 |     0.482  |                    2 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00002 | TERMINATED |                 |            2 |  256 |    4 | 0.000325006 | 1.51312 |     0.4458 |                    2 |
| DEFAULT_29585_00003 | TERMINATED |                 |           16 |  128 |  256 | 0.00576339  | 1.27674 |     0.5904 |                   10 |
| DEFAULT_29585_00005 | TERMINATED |                 |           16 |   16 |   64 | 0.000935427 | 1.19036 |     0.5843 |                   10 |
| DEFAULT_29585_00006 | TERMINATED |                 |           16 |    4 |  256 | 0.0350703   | 2.31604 |     0.0961 |                   10 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1482) [3, 18000] loss: 0.133
Result for DEFAULT_29585_00009:
  accuracy: 0.4844
  date: 2021-07-28_21-29-11
  done: false
  experiment_id: ea64a6e7d70a4ad8871632d3f7b5d378
  hostname: 2264d6af0a6a
  iterations_since_restore: 3
  loss: 1.4389056988980389
  node_ip: 172.17.0.2
  pid: 1474
  should_checkpoint: true
  time_since_restore: 293.4887282848358
  time_this_iter_s: 84.58493614196777
  time_total_s: 293.4887282848358
  timestamp: 1627507751
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: '29585_00009'

(pid=1470) [4,  2000] loss: 1.375
(pid=1474) [4,  2000] loss: 1.288
(pid=1482) [3, 20000] loss: 0.118
(pid=1470) [4,  4000] loss: 0.674
(pid=1474) [4,  4000] loss: 0.669
Result for DEFAULT_29585_00004:
  accuracy: 0.5815
  date: 2021-07-28_21-29-30
  done: false
  experiment_id: 3cff2c4dcd814197bcabbb3e417d8353
  hostname: 2264d6af0a6a
  iterations_since_restore: 3
  loss: 1.193133587324986
  node_ip: 172.17.0.2
  pid: 1482
  should_checkpoint: true
  time_since_restore: 312.6266505718231
  time_this_iter_s: 91.15460228919983
  time_total_s: 312.6266505718231
  timestamp: 1627507770
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: '29585_00004'

== Status ==
Memory usage on this node: 5.8/240.1 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.214192063331604 | Iter 4.000: -1.3946209341049194 | Iter 2.000: -1.5017838985789567 | Iter 1.000: -1.7815545465707778
Resources requested: 6.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.19313 |     0.5815 |                    3 |
| DEFAULT_29585_00008 | RUNNING    | 172.17.0.2:1470 |            2 |    4 |  256 | 0.000257535 | 1.41939 |     0.4876 |                    3 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.43891 |     0.4844 |                    3 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00002 | TERMINATED |                 |            2 |  256 |    4 | 0.000325006 | 1.51312 |     0.4458 |                    2 |
| DEFAULT_29585_00003 | TERMINATED |                 |           16 |  128 |  256 | 0.00576339  | 1.27674 |     0.5904 |                   10 |
| DEFAULT_29585_00005 | TERMINATED |                 |           16 |   16 |   64 | 0.000935427 | 1.19036 |     0.5843 |                   10 |
| DEFAULT_29585_00006 | TERMINATED |                 |           16 |    4 |  256 | 0.0350703   | 2.31604 |     0.0961 |                   10 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1470) [4,  6000] loss: 0.452
(pid=1474) [4,  6000] loss: 0.436
(pid=1482) [4,  2000] loss: 1.062
(pid=1470) [4,  8000] loss: 0.336
(pid=1474) [4,  8000] loss: 0.335
(pid=1482) [4,  4000] loss: 0.543
(pid=1470) [4, 10000] loss: 0.264
(pid=1474) [4, 10000] loss: 0.263
(pid=1470) [4, 12000] loss: 0.223
(pid=1482) [4,  6000] loss: 0.358
(pid=1474) [4, 12000] loss: 0.224
(pid=1470) [4, 14000] loss: 0.197
(pid=1474) [4, 14000] loss: 0.196
(pid=1482) [4,  8000] loss: 0.270
(pid=1470) [4, 16000] loss: 0.169
(pid=1474) [4, 16000] loss: 0.169
(pid=1482) [4, 10000] loss: 0.225
(pid=1470) [4, 18000] loss: 0.151
(pid=1474) [4, 18000] loss: 0.152
(pid=1482) [4, 12000] loss: 0.181
(pid=1470) [4, 20000] loss: 0.134
(pid=1474) [4, 20000] loss: 0.138
(pid=1482) [4, 14000] loss: 0.155
Result for DEFAULT_29585_00008:
  accuracy: 0.464
  date: 2021-07-28_21-30-33
  done: true
  experiment_id: 5f0894680a004816aa2aa41fc07e39d3
  hostname: 2264d6af0a6a
  iterations_since_restore: 4
  loss: 1.4191759492989628
  node_ip: 172.17.0.2
  pid: 1470
  should_checkpoint: true
  time_since_restore: 375.7807364463806
  time_this_iter_s: 82.89954233169556
  time_total_s: 375.7807364463806
  timestamp: 1627507833
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: '29585_00008'

== Status ==
Memory usage on this node: 5.8/240.1 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.214192063331604 | Iter 4.000: -1.4068984417019412 | Iter 2.000: -1.5017838985789567 | Iter 1.000: -1.7815545465707778
Resources requested: 6.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/1.0 accelerator_type:M60)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.19313 |     0.5815 |                    3 |
| DEFAULT_29585_00008 | RUNNING    | 172.17.0.2:1470 |            2 |    4 |  256 | 0.000257535 | 1.41918 |     0.464  |                    4 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.43891 |     0.4844 |                    3 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00002 | TERMINATED |                 |            2 |  256 |    4 | 0.000325006 | 1.51312 |     0.4458 |                    2 |
| DEFAULT_29585_00003 | TERMINATED |                 |           16 |  128 |  256 | 0.00576339  | 1.27674 |     0.5904 |                   10 |
| DEFAULT_29585_00005 | TERMINATED |                 |           16 |   16 |   64 | 0.000935427 | 1.19036 |     0.5843 |                   10 |
| DEFAULT_29585_00006 | TERMINATED |                 |           16 |    4 |  256 | 0.0350703   | 2.31604 |     0.0961 |                   10 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_29585_00009:
  accuracy: 0.5092
  date: 2021-07-28_21-30-34
  done: false
  experiment_id: ea64a6e7d70a4ad8871632d3f7b5d378
  hostname: 2264d6af0a6a
  iterations_since_restore: 4
  loss: 1.3442489511128515
  node_ip: 172.17.0.2
  pid: 1474
  should_checkpoint: true
  time_since_restore: 376.79525995254517
  time_this_iter_s: 83.30653166770935
  time_total_s: 376.79525995254517
  timestamp: 1627507834
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: '29585_00009'

(pid=1482) [4, 16000] loss: 0.139
(pid=1474) [5,  2000] loss: 1.269
(pid=1482) [4, 18000] loss: 0.122
(pid=1474) [5,  4000] loss: 0.640
(pid=1482) [4, 20000] loss: 0.111
(pid=1474) [5,  6000] loss: 0.443
Result for DEFAULT_29585_00004:
  accuracy: 0.5996
  date: 2021-07-28_21-31-01
  done: false
  experiment_id: 3cff2c4dcd814197bcabbb3e417d8353
  hostname: 2264d6af0a6a
  iterations_since_restore: 4
  loss: 1.145064449703123
  node_ip: 172.17.0.2
  pid: 1482
  should_checkpoint: true
  time_since_restore: 404.00305557250977
  time_this_iter_s: 91.37640500068665
  time_total_s: 404.00305557250977
  timestamp: 1627507861
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: '29585_00004'

== Status ==
Memory usage on this node: 5.2/240.1 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.214192063331604 | Iter 4.000: -1.3694349426088854 | Iter 2.000: -1.5017838985789567 | Iter 1.000: -1.7815545465707778
Resources requested: 4.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_0_ba47e49b59caf612e7cbbd0fa21ebfbe, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_ba47e49b59caf612e7cbbd0fa21ebfbe)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.14506 |     0.5996 |                    4 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.34425 |     0.5092 |                    4 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00002 | TERMINATED |                 |            2 |  256 |    4 | 0.000325006 | 1.51312 |     0.4458 |                    2 |
| DEFAULT_29585_00003 | TERMINATED |                 |           16 |  128 |  256 | 0.00576339  | 1.27674 |     0.5904 |                   10 |
| DEFAULT_29585_00005 | TERMINATED |                 |           16 |   16 |   64 | 0.000935427 | 1.19036 |     0.5843 |                   10 |
| DEFAULT_29585_00006 | TERMINATED |                 |           16 |    4 |  256 | 0.0350703   | 2.31604 |     0.0961 |                   10 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
| DEFAULT_29585_00008 | TERMINATED |                 |            2 |    4 |  256 | 0.000257535 | 1.41918 |     0.464  |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1474) [5,  8000] loss: 0.327
(pid=1482) [5,  2000] loss: 0.992
(pid=1474) [5, 10000] loss: 0.259
(pid=1482) [5,  4000] loss: 0.505
(pid=1474) [5, 12000] loss: 0.217
(pid=1474) [5, 14000] loss: 0.185
(pid=1482) [5,  6000] loss: 0.347
(pid=1474) [5, 16000] loss: 0.164
(pid=1482) [5,  8000] loss: 0.248
(pid=1474) [5, 18000] loss: 0.145
(pid=1482) [5, 10000] loss: 0.200
(pid=1474) [5, 20000] loss: 0.135
(pid=1482) [5, 12000] loss: 0.174
Result for DEFAULT_29585_00009:
  accuracy: 0.5213
  date: 2021-07-28_21-31-55
  done: false
  experiment_id: ea64a6e7d70a4ad8871632d3f7b5d378
  hostname: 2264d6af0a6a
  iterations_since_restore: 5
  loss: 1.3313996821369976
  node_ip: 172.17.0.2
  pid: 1474
  should_checkpoint: true
  time_since_restore: 458.1253709793091
  time_this_iter_s: 81.33011102676392
  time_total_s: 458.1253709793091
  timestamp: 1627507915
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: '29585_00009'

== Status ==
Memory usage on this node: 5.2/240.1 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.214192063331604 | Iter 4.000: -1.3694349426088854 | Iter 2.000: -1.5017838985789567 | Iter 1.000: -1.7815545465707778
Resources requested: 4.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/1.0 accelerator_type:M60)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.14506 |     0.5996 |                    4 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.3314  |     0.5213 |                    5 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00002 | TERMINATED |                 |            2 |  256 |    4 | 0.000325006 | 1.51312 |     0.4458 |                    2 |
| DEFAULT_29585_00003 | TERMINATED |                 |           16 |  128 |  256 | 0.00576339  | 1.27674 |     0.5904 |                   10 |
| DEFAULT_29585_00005 | TERMINATED |                 |           16 |   16 |   64 | 0.000935427 | 1.19036 |     0.5843 |                   10 |
| DEFAULT_29585_00006 | TERMINATED |                 |           16 |    4 |  256 | 0.0350703   | 2.31604 |     0.0961 |                   10 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
| DEFAULT_29585_00008 | TERMINATED |                 |            2 |    4 |  256 | 0.000257535 | 1.41918 |     0.464  |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1482) [5, 14000] loss: 0.146
(pid=1474) [6,  2000] loss: 1.249
(pid=1482) [5, 16000] loss: 0.129
(pid=1474) [6,  4000] loss: 0.647
(pid=1482) [5, 18000] loss: 0.119
(pid=1474) [6,  6000] loss: 0.432
(pid=1482) [5, 20000] loss: 0.110
(pid=1474) [6,  8000] loss: 0.322
(pid=1474) [6, 10000] loss: 0.256
Result for DEFAULT_29585_00004:
  accuracy: 0.5934
  date: 2021-07-28_21-32-31
  done: false
  experiment_id: 3cff2c4dcd814197bcabbb3e417d8353
  hostname: 2264d6af0a6a
  iterations_since_restore: 5
  loss: 1.1614596049055907
  node_ip: 172.17.0.2
  pid: 1482
  should_checkpoint: true
  time_since_restore: 493.54671239852905
  time_this_iter_s: 89.54365682601929
  time_total_s: 493.54671239852905
  timestamp: 1627507951
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: '29585_00004'

== Status ==
Memory usage on this node: 5.2/240.1 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.214192063331604 | Iter 4.000: -1.3694349426088854 | Iter 2.000: -1.5017838985789567 | Iter 1.000: -1.7815545465707778
Resources requested: 4.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.16146 |     0.5934 |                    5 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.3314  |     0.5213 |                    5 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00002 | TERMINATED |                 |            2 |  256 |    4 | 0.000325006 | 1.51312 |     0.4458 |                    2 |
| DEFAULT_29585_00003 | TERMINATED |                 |           16 |  128 |  256 | 0.00576339  | 1.27674 |     0.5904 |                   10 |
| DEFAULT_29585_00005 | TERMINATED |                 |           16 |   16 |   64 | 0.000935427 | 1.19036 |     0.5843 |                   10 |
| DEFAULT_29585_00006 | TERMINATED |                 |           16 |    4 |  256 | 0.0350703   | 2.31604 |     0.0961 |                   10 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
| DEFAULT_29585_00008 | TERMINATED |                 |            2 |    4 |  256 | 0.000257535 | 1.41918 |     0.464  |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1474) [6, 12000] loss: 0.215
(pid=1482) [6,  2000] loss: 0.934
(pid=1474) [6, 14000] loss: 0.187
(pid=1482) [6,  4000] loss: 0.478
(pid=1474) [6, 16000] loss: 0.163
(pid=1482) [6,  6000] loss: 0.322
(pid=1474) [6, 18000] loss: 0.144
(pid=1482) [6,  8000] loss: 0.241
(pid=1474) [6, 20000] loss: 0.129
(pid=1482) [6, 10000] loss: 0.199
Result for DEFAULT_29585_00009:
  accuracy: 0.534
  date: 2021-07-28_21-33-17
  done: false
  experiment_id: ea64a6e7d70a4ad8871632d3f7b5d378
  hostname: 2264d6af0a6a
  iterations_since_restore: 6
  loss: 1.340734103111946
  node_ip: 172.17.0.2
  pid: 1474
  should_checkpoint: true
  time_since_restore: 539.5027532577515
  time_this_iter_s: 81.37738227844238
  time_total_s: 539.5027532577515
  timestamp: 1627507997
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: '29585_00009'

== Status ==
Memory usage on this node: 5.3/240.1 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.214192063331604 | Iter 4.000: -1.3694349426088854 | Iter 2.000: -1.5017838985789567 | Iter 1.000: -1.7815545465707778
Resources requested: 4.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/1.0 accelerator_type:M60)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.16146 |     0.5934 |                    5 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.34073 |     0.534  |                    6 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00002 | TERMINATED |                 |            2 |  256 |    4 | 0.000325006 | 1.51312 |     0.4458 |                    2 |
| DEFAULT_29585_00003 | TERMINATED |                 |           16 |  128 |  256 | 0.00576339  | 1.27674 |     0.5904 |                   10 |
| DEFAULT_29585_00005 | TERMINATED |                 |           16 |   16 |   64 | 0.000935427 | 1.19036 |     0.5843 |                   10 |
| DEFAULT_29585_00006 | TERMINATED |                 |           16 |    4 |  256 | 0.0350703   | 2.31604 |     0.0961 |                   10 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
| DEFAULT_29585_00008 | TERMINATED |                 |            2 |    4 |  256 | 0.000257535 | 1.41918 |     0.464  |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1482) [6, 12000] loss: 0.163
(pid=1474) [7,  2000] loss: 1.247
(pid=1482) [6, 14000] loss: 0.140
(pid=1474) [7,  4000] loss: 0.628
(pid=1482) [6, 16000] loss: 0.126
(pid=1474) [7,  6000] loss: 0.415
(pid=1482) [6, 18000] loss: 0.112
(pid=1474) [7,  8000] loss: 0.311
(pid=1482) [6, 20000] loss: 0.101
(pid=1474) [7, 10000] loss: 0.251
(pid=1474) [7, 12000] loss: 0.215
Result for DEFAULT_29585_00004:
  accuracy: 0.588
  date: 2021-07-28_21-34-00
  done: false
  experiment_id: 3cff2c4dcd814197bcabbb3e417d8353
  hostname: 2264d6af0a6a
  iterations_since_restore: 6
  loss: 1.224446568779156
  node_ip: 172.17.0.2
  pid: 1482
  should_checkpoint: true
  time_since_restore: 582.2956564426422
  time_this_iter_s: 88.74894404411316
  time_total_s: 582.2956564426422
  timestamp: 1627508040
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: '29585_00004'

== Status ==
Memory usage on this node: 5.2/240.1 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.214192063331604 | Iter 4.000: -1.3694349426088854 | Iter 2.000: -1.5017838985789567 | Iter 1.000: -1.7815545465707778
Resources requested: 4.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.22445 |     0.588  |                    6 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.34073 |     0.534  |                    6 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00002 | TERMINATED |                 |            2 |  256 |    4 | 0.000325006 | 1.51312 |     0.4458 |                    2 |
| DEFAULT_29585_00003 | TERMINATED |                 |           16 |  128 |  256 | 0.00576339  | 1.27674 |     0.5904 |                   10 |
| DEFAULT_29585_00005 | TERMINATED |                 |           16 |   16 |   64 | 0.000935427 | 1.19036 |     0.5843 |                   10 |
| DEFAULT_29585_00006 | TERMINATED |                 |           16 |    4 |  256 | 0.0350703   | 2.31604 |     0.0961 |                   10 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
| DEFAULT_29585_00008 | TERMINATED |                 |            2 |    4 |  256 | 0.000257535 | 1.41918 |     0.464  |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1474) [7, 14000] loss: 0.182
(pid=1482) [7,  2000] loss: 0.903
(pid=1474) [7, 16000] loss: 0.164
(pid=1482) [7,  4000] loss: 0.461
(pid=1474) [7, 18000] loss: 0.142
(pid=1482) [7,  6000] loss: 0.310
(pid=1474) [7, 20000] loss: 0.130
(pid=1482) [7,  8000] loss: 0.227
Result for DEFAULT_29585_00009:
  accuracy: 0.5231
  date: 2021-07-28_21-34-38
  done: false
  experiment_id: ea64a6e7d70a4ad8871632d3f7b5d378
  hostname: 2264d6af0a6a
  iterations_since_restore: 7
  loss: 1.3981339836945756
  node_ip: 172.17.0.2
  pid: 1474
  should_checkpoint: true
  time_since_restore: 621.1267395019531
  time_this_iter_s: 81.62398624420166
  time_total_s: 621.1267395019531
  timestamp: 1627508078
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: '29585_00009'

== Status ==
Memory usage on this node: 5.3/240.1 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.214192063331604 | Iter 4.000: -1.3694349426088854 | Iter 2.000: -1.5017838985789567 | Iter 1.000: -1.7815545465707778
Resources requested: 4.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/1.0 accelerator_type:M60)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.22445 |     0.588  |                    6 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.39813 |     0.5231 |                    7 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00002 | TERMINATED |                 |            2 |  256 |    4 | 0.000325006 | 1.51312 |     0.4458 |                    2 |
| DEFAULT_29585_00003 | TERMINATED |                 |           16 |  128 |  256 | 0.00576339  | 1.27674 |     0.5904 |                   10 |
| DEFAULT_29585_00005 | TERMINATED |                 |           16 |   16 |   64 | 0.000935427 | 1.19036 |     0.5843 |                   10 |
| DEFAULT_29585_00006 | TERMINATED |                 |           16 |    4 |  256 | 0.0350703   | 2.31604 |     0.0961 |                   10 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
| DEFAULT_29585_00008 | TERMINATED |                 |            2 |    4 |  256 | 0.000257535 | 1.41918 |     0.464  |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1482) [7, 10000] loss: 0.184
(pid=1474) [8,  2000] loss: 1.229
(pid=1482) [7, 12000] loss: 0.162
(pid=1474) [8,  4000] loss: 0.625
(pid=1482) [7, 14000] loss: 0.136
(pid=1474) [8,  6000] loss: 0.417
(pid=1482) [7, 16000] loss: 0.122
(pid=1474) [8,  8000] loss: 0.321
(pid=1482) [7, 18000] loss: 0.103
(pid=1474) [8, 10000] loss: 0.246
(pid=1482) [7, 20000] loss: 0.097
(pid=1474) [8, 12000] loss: 0.211
Result for DEFAULT_29585_00004:
  accuracy: 0.6075
  date: 2021-07-28_21-35-28
  done: false
  experiment_id: 3cff2c4dcd814197bcabbb3e417d8353
  hostname: 2264d6af0a6a
  iterations_since_restore: 7
  loss: 1.2095530263678642
  node_ip: 172.17.0.2
  pid: 1482
  should_checkpoint: true
  time_since_restore: 670.9358050823212
  time_this_iter_s: 88.64014863967896
  time_total_s: 670.9358050823212
  timestamp: 1627508128
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: '29585_00004'

== Status ==
Memory usage on this node: 5.2/240.1 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.214192063331604 | Iter 4.000: -1.3694349426088854 | Iter 2.000: -1.5017838985789567 | Iter 1.000: -1.7815545465707778
Resources requested: 4.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/1.0 accelerator_type:M60)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.20955 |     0.6075 |                    7 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.39813 |     0.5231 |                    7 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00002 | TERMINATED |                 |            2 |  256 |    4 | 0.000325006 | 1.51312 |     0.4458 |                    2 |
| DEFAULT_29585_00003 | TERMINATED |                 |           16 |  128 |  256 | 0.00576339  | 1.27674 |     0.5904 |                   10 |
| DEFAULT_29585_00005 | TERMINATED |                 |           16 |   16 |   64 | 0.000935427 | 1.19036 |     0.5843 |                   10 |
| DEFAULT_29585_00006 | TERMINATED |                 |           16 |    4 |  256 | 0.0350703   | 2.31604 |     0.0961 |                   10 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
| DEFAULT_29585_00008 | TERMINATED |                 |            2 |    4 |  256 | 0.000257535 | 1.41918 |     0.464  |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1474) [8, 14000] loss: 0.181
(pid=1474) [8, 16000] loss: 0.156
(pid=1482) [8,  2000] loss: 0.864
(pid=1474) [8, 18000] loss: 0.144
(pid=1482) [8,  4000] loss: 0.438
(pid=1474) [8, 20000] loss: 0.127
(pid=1482) [8,  6000] loss: 0.297
(pid=1482) [8,  8000] loss: 0.228
Result for DEFAULT_29585_00009:
  accuracy: 0.5335
  date: 2021-07-28_21-36-00
  done: true
  experiment_id: ea64a6e7d70a4ad8871632d3f7b5d378
  hostname: 2264d6af0a6a
  iterations_since_restore: 8
  loss: 1.3428828423896804
  node_ip: 172.17.0.2
  pid: 1474
  should_checkpoint: true
  time_since_restore: 703.0817213058472
  time_this_iter_s: 81.95498180389404
  time_total_s: 703.0817213058472
  timestamp: 1627508160
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: '29585_00009'

== Status ==
Memory usage on this node: 5.2/240.1 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2785374528606421 | Iter 4.000: -1.3694349426088854 | Iter 2.000: -1.5017838985789567 | Iter 1.000: -1.7815545465707778
Resources requested: 4.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.20955 |     0.6075 |                    7 |
| DEFAULT_29585_00009 | RUNNING    | 172.17.0.2:1474 |            2 |   16 |  256 | 0.00103426  | 1.34288 |     0.5335 |                    8 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00002 | TERMINATED |                 |            2 |  256 |    4 | 0.000325006 | 1.51312 |     0.4458 |                    2 |
| DEFAULT_29585_00003 | TERMINATED |                 |           16 |  128 |  256 | 0.00576339  | 1.27674 |     0.5904 |                   10 |
| DEFAULT_29585_00005 | TERMINATED |                 |           16 |   16 |   64 | 0.000935427 | 1.19036 |     0.5843 |                   10 |
| DEFAULT_29585_00006 | TERMINATED |                 |           16 |    4 |  256 | 0.0350703   | 2.31604 |     0.0961 |                   10 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
| DEFAULT_29585_00008 | TERMINATED |                 |            2 |    4 |  256 | 0.000257535 | 1.41918 |     0.464  |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1482) [8, 10000] loss: 0.172
(pid=1482) [8, 12000] loss: 0.155
(pid=1482) [8, 14000] loss: 0.135
(pid=1482) [8, 16000] loss: 0.119
(pid=1482) [8, 18000] loss: 0.104
(pid=1482) [8, 20000] loss: 0.091
Result for DEFAULT_29585_00004:
  accuracy: 0.5993
  date: 2021-07-28_21-36-56
  done: false
  experiment_id: 3cff2c4dcd814197bcabbb3e417d8353
  hostname: 2264d6af0a6a
  iterations_since_restore: 8
  loss: 1.2500360607438807
  node_ip: 172.17.0.2
  pid: 1482
  should_checkpoint: true
  time_since_restore: 758.2536313533783
  time_this_iter_s: 87.31782627105713
  time_total_s: 758.2536313533783
  timestamp: 1627508216
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: '29585_00004'

== Status ==
Memory usage on this node: 4.6/240.1 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2500360607438807 | Iter 4.000: -1.3694349426088854 | Iter 2.000: -1.5017838985789567 | Iter 1.000: -1.7815545465707778
Resources requested: 2.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/2.0 CPU_group_959bb1c4a8e2227fd6edaeb48ccf9d2e, 0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.25004 |     0.5993 |                    8 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00002 | TERMINATED |                 |            2 |  256 |    4 | 0.000325006 | 1.51312 |     0.4458 |                    2 |
| DEFAULT_29585_00003 | TERMINATED |                 |           16 |  128 |  256 | 0.00576339  | 1.27674 |     0.5904 |                   10 |
| DEFAULT_29585_00005 | TERMINATED |                 |           16 |   16 |   64 | 0.000935427 | 1.19036 |     0.5843 |                   10 |
| DEFAULT_29585_00006 | TERMINATED |                 |           16 |    4 |  256 | 0.0350703   | 2.31604 |     0.0961 |                   10 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
| DEFAULT_29585_00008 | TERMINATED |                 |            2 |    4 |  256 | 0.000257535 | 1.41918 |     0.464  |                    4 |
| DEFAULT_29585_00009 | TERMINATED |                 |            2 |   16 |  256 | 0.00103426  | 1.34288 |     0.5335 |                    8 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1482) [9,  2000] loss: 0.781
(pid=1482) [9,  4000] loss: 0.417
(pid=1482) [9,  6000] loss: 0.293
(pid=1482) [9,  8000] loss: 0.215
(pid=1482) [9, 10000] loss: 0.180
(pid=1482) [9, 12000] loss: 0.150
(pid=1482) [9, 14000] loss: 0.123
(pid=1482) [9, 16000] loss: 0.114
(pid=1482) [9, 18000] loss: 0.101
(pid=1482) [9, 20000] loss: 0.094
Result for DEFAULT_29585_00004:
  accuracy: 0.5991
  date: 2021-07-28_21-38-22
  done: false
  experiment_id: 3cff2c4dcd814197bcabbb3e417d8353
  hostname: 2264d6af0a6a
  iterations_since_restore: 9
  loss: 1.2725142115146015
  node_ip: 172.17.0.2
  pid: 1482
  should_checkpoint: true
  time_since_restore: 844.3728473186493
  time_this_iter_s: 86.119215965271
  time_total_s: 844.3728473186493
  timestamp: 1627508302
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: '29585_00004'

== Status ==
Memory usage on this node: 4.6/240.1 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2500360607438807 | Iter 4.000: -1.3694349426088854 | Iter 2.000: -1.5017838985789567 | Iter 1.000: -1.7815545465707778
Resources requested: 2.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.27251 |     0.5991 |                    9 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00002 | TERMINATED |                 |            2 |  256 |    4 | 0.000325006 | 1.51312 |     0.4458 |                    2 |
| DEFAULT_29585_00003 | TERMINATED |                 |           16 |  128 |  256 | 0.00576339  | 1.27674 |     0.5904 |                   10 |
| DEFAULT_29585_00005 | TERMINATED |                 |           16 |   16 |   64 | 0.000935427 | 1.19036 |     0.5843 |                   10 |
| DEFAULT_29585_00006 | TERMINATED |                 |           16 |    4 |  256 | 0.0350703   | 2.31604 |     0.0961 |                   10 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
| DEFAULT_29585_00008 | TERMINATED |                 |            2 |    4 |  256 | 0.000257535 | 1.41918 |     0.464  |                    4 |
| DEFAULT_29585_00009 | TERMINATED |                 |            2 |   16 |  256 | 0.00103426  | 1.34288 |     0.5335 |                    8 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1482) [10,  2000] loss: 0.770
(pid=1482) [10,  4000] loss: 0.396
(pid=1482) [10,  6000] loss: 0.270
(pid=1482) [10,  8000] loss: 0.211
(pid=1482) [10, 10000] loss: 0.178
(pid=1482) [10, 12000] loss: 0.145
(pid=1482) [10, 14000] loss: 0.130
(pid=1482) [10, 16000] loss: 0.110
(pid=1482) [10, 18000] loss: 0.100
(pid=1482) [10, 20000] loss: 0.086
Result for DEFAULT_29585_00004:
  accuracy: 0.595
  date: 2021-07-28_21-39-49
  done: true
  experiment_id: 3cff2c4dcd814197bcabbb3e417d8353
  hostname: 2264d6af0a6a
  iterations_since_restore: 10
  loss: 1.2961692139043703
  node_ip: 172.17.0.2
  pid: 1482
  should_checkpoint: true
  time_since_restore: 931.2560675144196
  time_this_iter_s: 86.88322019577026
  time_total_s: 931.2560675144196
  timestamp: 1627508389
  timesteps_since_restore: 0
  training_iteration: 10
  trial_id: '29585_00004'

== Status ==
Memory usage on this node: 4.6/240.1 GiB
Using AsyncHyperBand: num_stopped=10
Bracket: Iter 8.000: -1.2500360607438807 | Iter 4.000: -1.3694349426088854 | Iter 2.000: -1.5017838985789567 | Iter 1.000: -1.7815545465707778
Resources requested: 2.0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/1.0 accelerator_type:M60, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00004 | RUNNING    | 172.17.0.2:1482 |            2 |  128 |  256 | 0.000801667 | 1.29617 |     0.595  |                   10 |
| DEFAULT_29585_00000 | TERMINATED |                 |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |                 |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00002 | TERMINATED |                 |            2 |  256 |    4 | 0.000325006 | 1.51312 |     0.4458 |                    2 |
| DEFAULT_29585_00003 | TERMINATED |                 |           16 |  128 |  256 | 0.00576339  | 1.27674 |     0.5904 |                   10 |
| DEFAULT_29585_00005 | TERMINATED |                 |           16 |   16 |   64 | 0.000935427 | 1.19036 |     0.5843 |                   10 |
| DEFAULT_29585_00006 | TERMINATED |                 |           16 |    4 |  256 | 0.0350703   | 2.31604 |     0.0961 |                   10 |
| DEFAULT_29585_00007 | TERMINATED |                 |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
| DEFAULT_29585_00008 | TERMINATED |                 |            2 |    4 |  256 | 0.000257535 | 1.41918 |     0.464  |                    4 |
| DEFAULT_29585_00009 | TERMINATED |                 |            2 |   16 |  256 | 0.00103426  | 1.34288 |     0.5335 |                    8 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


== Status ==
Memory usage on this node: 4.2/240.1 GiB
Using AsyncHyperBand: num_stopped=10
Bracket: Iter 8.000: -1.2500360607438807 | Iter 4.000: -1.3694349426088854 | Iter 2.000: -1.5017838985789567 | Iter 1.000: -1.7815545465707778
Resources requested: 0/32 CPUs, 0/2 GPUs, 0.0/220.17 GiB heap, 0.0/9.31 GiB objects (0.0/2.0 CPU_group_0_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/2.0 CPU_group_5bd4860c3f6b6e184fa899b6eac89e77, 0.0/1.0 accelerator_type:M60)
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2021-07-28_21-24-15
Number of trials: 10/10 (10 TERMINATED)
+---------------------+------------+-------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc   |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_29585_00000 | TERMINATED |       |            8 |   16 |    4 | 0.0224094   | 2.30682 |     0.0988 |                    1 |
| DEFAULT_29585_00001 | TERMINATED |       |            4 |    8 |   16 | 0.0156001   | 2.31424 |     0.1015 |                    1 |
| DEFAULT_29585_00002 | TERMINATED |       |            2 |  256 |    4 | 0.000325006 | 1.51312 |     0.4458 |                    2 |
| DEFAULT_29585_00003 | TERMINATED |       |           16 |  128 |  256 | 0.00576339  | 1.27674 |     0.5904 |                   10 |
| DEFAULT_29585_00004 | TERMINATED |       |            2 |  128 |  256 | 0.000801667 | 1.29617 |     0.595  |                   10 |
| DEFAULT_29585_00005 | TERMINATED |       |           16 |   16 |   64 | 0.000935427 | 1.19036 |     0.5843 |                   10 |
| DEFAULT_29585_00006 | TERMINATED |       |           16 |    4 |  256 | 0.0350703   | 2.31604 |     0.0961 |                   10 |
| DEFAULT_29585_00007 | TERMINATED |       |            8 |  128 |  256 | 0.00049783  | 1.66182 |     0.3926 |                    2 |
| DEFAULT_29585_00008 | TERMINATED |       |            2 |    4 |  256 | 0.000257535 | 1.41918 |     0.464  |                    4 |
| DEFAULT_29585_00009 | TERMINATED |       |            2 |   16 |  256 | 0.00103426  | 1.34288 |     0.5335 |                    8 |
+---------------------+------------+-------+--------------+------+------+-------------+---------+------------+----------------------+


Best trial config: {'l1': 16, 'l2': 64, 'lr': 0.0009354271097155562, 'batch_size': 16}
Best trial final validation loss: 1.190362274312973
Best trial final validation accuracy: 0.5843
Files already downloaded and verified
Files already downloaded and verified
Best trial test set accuracy: 0.593

If you run the code, an example output could look like this:

Number of trials: 10 (10 TERMINATED)
+-----+------+------+-------------+--------------+---------+------------+--------------------+
| ... |   l1 |   l2 |          lr |   batch_size |    loss |   accuracy | training_iteration |
|-----+------+------+-------------+--------------+---------+------------+--------------------|
| ... |   64 |    4 | 0.00011629  |            2 | 1.87273 |     0.244  |                  2 |
| ... |   32 |   64 | 0.000339763 |            8 | 1.23603 |     0.567  |                  8 |
| ... |    8 |   16 | 0.00276249  |           16 | 1.1815  |     0.5836 |                 10 |
| ... |    4 |   64 | 0.000648721 |            4 | 1.31131 |     0.5224 |                  8 |
| ... |   32 |   16 | 0.000340753 |            8 | 1.26454 |     0.5444 |                  8 |
| ... |    8 |    4 | 0.000699775 |            8 | 1.99594 |     0.1983 |                  2 |
| ... |  256 |    8 | 0.0839654   |           16 | 2.3119  |     0.0993 |                  1 |
| ... |   16 |  128 | 0.0758154   |           16 | 2.33575 |     0.1327 |                  1 |
| ... |   16 |    8 | 0.0763312   |           16 | 2.31129 |     0.1042 |                  4 |
| ... |  128 |   16 | 0.000124903 |            4 | 2.26917 |     0.1945 |                  1 |
+-----+------+------+-------------+--------------+---------+------------+--------------------+


Best trial config: {'l1': 8, 'l2': 16, 'lr': 0.00276249, 'batch_size': 16, 'data_dir': '...'}
Best trial final validation loss: 1.181501
Best trial final validation accuracy: 0.5836
Best trial test set accuracy: 0.5806

Most trials have been stopped early in order to avoid wasting resources. The best performing trial achieved a validation accuracy of about 58%, which could be confirmed on the test set.

So that’s it! You can now tune the parameters of your PyTorch models.