Hyperparameter tuning with Ray Tune

Hyperparameter tuning can make the difference between an average model and a highly accurate one. Often simple things like choosing a different learning rate or changing a network layer size can have a dramatic impact on your model performance.

Fortunately, there are tools that help with finding the best combination of parameters. Ray Tune is an industry standard tool for distributed hyperparameter tuning. Ray Tune includes the latest hyperparameter search algorithms, integrates with TensorBoard and other analysis libraries, and natively supports distributed training through Ray’s distributed machine learning engine.

In this tutorial, we will show you how to integrate Ray Tune into your PyTorch training workflow. We will extend this tutorial from the PyTorch documentation for training a CIFAR10 image classifier.

As you will see, we only need to add some slight modifications. In particular, we need to

1. wrap data loading and training in functions,
2. make some network parameters configurable,
3. add checkpointing (optional),
4. and define the search space for the model tuning

To run this tutorial, please make sure the following packages are installed:

• ray[tune]: Distributed hyperparameter tuning library
• torchvision: For the data transformers

Setup / Imports

Let’s start with the imports:

from functools import partial
import numpy as np
import os
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import random_split
import torchvision
import torchvision.transforms as transforms
from ray import tune
from ray.tune import CLIReporter
from ray.tune.schedulers import ASHAScheduler

Most of the imports are needed for building the PyTorch model. Only the last three imports are for Ray Tune.

Data loaders

We wrap the data loaders in their own function and pass a global data directory. This way we can share a data directory between different trials.

def load_data(data_dir="./data"):
    transform = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
    ])

    trainset = torchvision.datasets.CIFAR10(
        root=data_dir, train=True, download=True, transform=transform)

    testset = torchvision.datasets.CIFAR10(
        root=data_dir, train=False, download=True, transform=transform)

    return trainset, testset

Configurable neural network

We can only tune those parameters that are configurable. In this example, we can specify the layer sizes of the fully connected layers:

class Net(nn.Module):
    def __init__(self, l1=120, l2=84):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, l1)
        self.fc2 = nn.Linear(l1, l2)
        self.fc3 = nn.Linear(l2, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

The train function

Now it gets interesting, because we introduce some changes to the example from the PyTorch documentation.

We wrap the training script in a function train_cifar(config, checkpoint_dir=None, data_dir=None). As you can guess, the config parameter will receive the hyperparameters we would like to train with. The checkpoint_dir parameter is used to restore checkpoints. The data_dir specifies the directory where we load and store the data, so multiple runs can share the same data source.

net = Net(config["l1"], config["l2"])

if checkpoint_dir:
    model_state, optimizer_state = torch.load(
        os.path.join(checkpoint_dir, "checkpoint"))
    net.load_state_dict(model_state)
    optimizer.load_state_dict(optimizer_state)

The learning rate of the optimizer is made configurable, too:

optimizer = optim.SGD(net.parameters(), lr=config["lr"], momentum=0.9)

We also split the training data into a training and validation subset. We thus train on 80% of the data and calculate the validation loss on the remaining 20%. The batch sizes with which we iterate through the training and test sets are configurable as well.

Adding (multi) GPU support with DataParallel

Image classification benefits largely from GPUs. Luckily, we can continue to use PyTorch’s abstractions in Ray Tune. Thus, we can wrap our model in nn.DataParallel to support data parallel training on multiple GPUs:

device = "cpu"
if torch.cuda.is_available():
    device = "cuda:0"
    if torch.cuda.device_count() > 1:
        net = nn.DataParallel(net)
net.to(device)

By using a device variable we make sure that training also works when we have no GPUs available. PyTorch requires us to send our data to the GPU memory explicitly, like this:

for i, data in enumerate(trainloader, 0):
    inputs, labels = data
    inputs, labels = inputs.to(device), labels.to(device)

The code now supports training on CPUs, on a single GPU, and on multiple GPUs. Notably, Ray also supports fractional GPUs so we can share GPUs among trials, as long as the model still fits on the GPU memory. We’ll come back to that later.

Communicating with Ray Tune

The most interesting part is the communication with Ray Tune:

with tune.checkpoint_dir(epoch) as checkpoint_dir:
    path = os.path.join(checkpoint_dir, "checkpoint")
    torch.save((net.state_dict(), optimizer.state_dict()), path)

tune.report(loss=(val_loss / val_steps), accuracy=correct / total)

Here we first save a checkpoint and then report some metrics back to Ray Tune. Specifically, we send the validation loss and accuracy back to Ray Tune. Ray Tune can then use these metrics to decide which hyperparameter configuration lead to the best results. These metrics can also be used to stop bad performing trials early in order to avoid wasting resources on those trials.

The checkpoint saving is optional, however, it is necessary if we wanted to use advanced schedulers like Population Based Training. Also, by saving the checkpoint we can later load the trained models and validate them on a test set.

Full training function

The full code example looks like this:

def train_cifar(config, checkpoint_dir=None, data_dir=None):
    net = Net(config["l1"], config["l2"])

    device = "cpu"
    if torch.cuda.is_available():
        device = "cuda:0"
        if torch.cuda.device_count() > 1:
            net = nn.DataParallel(net)
    net.to(device)

    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(net.parameters(), lr=config["lr"], momentum=0.9)

    if checkpoint_dir:
        model_state, optimizer_state = torch.load(
            os.path.join(checkpoint_dir, "checkpoint"))
        net.load_state_dict(model_state)
        optimizer.load_state_dict(optimizer_state)

    trainset, testset = load_data(data_dir)

    test_abs = int(len(trainset) * 0.8)
    train_subset, val_subset = random_split(
        trainset, [test_abs, len(trainset) - test_abs])

    trainloader = torch.utils.data.DataLoader(
        train_subset,
        batch_size=int(config["batch_size"]),
        shuffle=True,
        num_workers=8)
    valloader = torch.utils.data.DataLoader(
        val_subset,
        batch_size=int(config["batch_size"]),
        shuffle=True,
        num_workers=8)

    for epoch in range(10):  # loop over the dataset multiple times
        running_loss = 0.0
        epoch_steps = 0
        for i, data in enumerate(trainloader, 0):
            # get the inputs; data is a list of [inputs, labels]
            inputs, labels = data
            inputs, labels = inputs.to(device), labels.to(device)

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward + backward + optimize
            outputs = net(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            # print statistics
            running_loss += loss.item()
            epoch_steps += 1
            if i % 2000 == 1999:  # print every 2000 mini-batches
                print("[%d, %5d] loss: %.3f" % (epoch + 1, i + 1,
                                                running_loss / epoch_steps))
                running_loss = 0.0

        # Validation loss
        val_loss = 0.0
        val_steps = 0
        total = 0
        correct = 0
        for i, data in enumerate(valloader, 0):
            with torch.no_grad():
                inputs, labels = data
                inputs, labels = inputs.to(device), labels.to(device)

                outputs = net(inputs)
                _, predicted = torch.max(outputs.data, 1)
                total += labels.size(0)
                correct += (predicted == labels).sum().item()

                loss = criterion(outputs, labels)
                val_loss += loss.cpu().numpy()
                val_steps += 1

        with tune.checkpoint_dir(epoch) as checkpoint_dir:
            path = os.path.join(checkpoint_dir, "checkpoint")
            torch.save((net.state_dict(), optimizer.state_dict()), path)

        tune.report(loss=(val_loss / val_steps), accuracy=correct / total)
    print("Finished Training")

As you can see, most of the code is adapted directly from the original example.

Test set accuracy

Commonly the performance of a machine learning model is tested on a hold-out test set with data that has not been used for training the model. We also wrap this in a function:

def test_accuracy(net, device="cpu"):
    trainset, testset = load_data()

    testloader = torch.utils.data.DataLoader(
        testset, batch_size=4, shuffle=False, num_workers=2)

    correct = 0
    total = 0
    with torch.no_grad():
        for data in testloader:
            images, labels = data
            images, labels = images.to(device), labels.to(device)
            outputs = net(images)
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()

    return correct / total

The function also expects a device parameter, so we can do the test set validation on a GPU.

Configuring the search space

Lastly, we need to define Ray Tune’s search space. Here is an example:

config = {
    "l1": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
    "l2": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
    "lr": tune.loguniform(1e-4, 1e-1),
    "batch_size": tune.choice([2, 4, 8, 16])
}

The tune.sample_from() function makes it possible to define your own sample methods to obtain hyperparameters. In this example, the l1 and l2 parameters should be powers of 2 between 4 and 256, so either 4, 8, 16, 32, 64, 128, or 256. The lr (learning rate) should be uniformly sampled between 0.0001 and 0.1. Lastly, the batch size is a choice between 2, 4, 8, and 16.

At each trial, Ray Tune will now randomly sample a combination of parameters from these search spaces. It will then train a number of models in parallel and find the best performing one among these. We also use the ASHAScheduler which will terminate bad performing trials early.

We wrap the train_cifar function with functools.partial to set the constant data_dir parameter. We can also tell Ray Tune what resources should be available for each trial:

gpus_per_trial = 2
# ...
result = tune.run(
    partial(train_cifar, data_dir=data_dir),
    resources_per_trial={"cpu": 8, "gpu": gpus_per_trial},
    config=config,
    num_samples=num_samples,
    scheduler=scheduler,
    progress_reporter=reporter,
    checkpoint_at_end=True)

You can specify the number of CPUs, which are then available e.g. to increase the num_workers of the PyTorch DataLoader instances. The selected number of GPUs are made visible to PyTorch in each trial. Trials do not have access to GPUs that haven’t been requested for them - so you don’t have to care about two trials using the same set of resources.

Here we can also specify fractional GPUs, so something like gpus_per_trial=0.5 is completely valid. The trials will then share GPUs among each other. You just have to make sure that the models still fit in the GPU memory.

After training the models, we will find the best performing one and load the trained network from the checkpoint file. We then obtain the test set accuracy and report everything by printing.

The full main function looks like this:

def main(num_samples=10, max_num_epochs=10, gpus_per_trial=2):
    data_dir = os.path.abspath("./data")
    load_data(data_dir)
    config = {
        "l1": tune.sample_from(lambda _: 2 ** np.random.randint(2, 9)),
        "l2": tune.sample_from(lambda _: 2 ** np.random.randint(2, 9)),
        "lr": tune.loguniform(1e-4, 1e-1),
        "batch_size": tune.choice([2, 4, 8, 16])
    }
    scheduler = ASHAScheduler(
        metric="loss",
        mode="min",
        max_t=max_num_epochs,
        grace_period=1,
        reduction_factor=2)
    reporter = CLIReporter(
        # parameter_columns=["l1", "l2", "lr", "batch_size"],
        metric_columns=["loss", "accuracy", "training_iteration"])
    result = tune.run(
        partial(train_cifar, data_dir=data_dir),
        resources_per_trial={"cpu": 2, "gpu": gpus_per_trial},
        config=config,
        num_samples=num_samples,
        scheduler=scheduler,
        progress_reporter=reporter)

    best_trial = result.get_best_trial("loss", "min", "last")
    print("Best trial config: {}".format(best_trial.config))
    print("Best trial final validation loss: {}".format(
        best_trial.last_result["loss"]))
    print("Best trial final validation accuracy: {}".format(
        best_trial.last_result["accuracy"]))

    best_trained_model = Net(best_trial.config["l1"], best_trial.config["l2"])
    device = "cpu"
    if torch.cuda.is_available():
        device = "cuda:0"
        if gpus_per_trial > 1:
            best_trained_model = nn.DataParallel(best_trained_model)
    best_trained_model.to(device)

    best_checkpoint_dir = best_trial.checkpoint.value
    model_state, optimizer_state = torch.load(os.path.join(
        best_checkpoint_dir, "checkpoint"))
    best_trained_model.load_state_dict(model_state)

    test_acc = test_accuracy(best_trained_model, device)
    print("Best trial test set accuracy: {}".format(test_acc))


if __name__ == "__main__":
    # You can change the number of GPUs per trial here:
    main(num_samples=10, max_num_epochs=10, gpus_per_trial=0)

Out:

Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to /var/lib/jenkins/workspace/beginner_source/data/cifar-10-python.tar.gz
Extracting /var/lib/jenkins/workspace/beginner_source/data/cifar-10-python.tar.gz to /var/lib/jenkins/workspace/beginner_source/data
Files already downloaded and verified
== Status ==
Memory usage on this node: 4.1/240.1 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 2/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 1/10 (1 RUNNING)
+---------------------+----------+-------+--------------+------+------+-----------+
| Trial name          | status   | loc   |   batch_size |   l1 |   l2 |        lr |
|---------------------+----------+-------+--------------+------+------+-----------|
| DEFAULT_36af6_00000 | RUNNING  |       |           16 |  128 |   16 | 0.0845979 |
+---------------------+----------+-------+--------------+------+------+-----------+


(pid=1363) Files already downloaded and verified
(pid=1360) Files already downloaded and verified
(pid=1352) Files already downloaded and verified
(pid=1411) Files already downloaded and verified
(pid=1365) Files already downloaded and verified
(pid=1366) Files already downloaded and verified
(pid=1394) Files already downloaded and verified
(pid=1359) Files already downloaded and verified
(pid=1361) Files already downloaded and verified
(pid=1364) Files already downloaded and verified
(pid=1363) Files already downloaded and verified
(pid=1360) Files already downloaded and verified
(pid=1352) Files already downloaded and verified
(pid=1411) Files already downloaded and verified
(pid=1365) Files already downloaded and verified
(pid=1366) Files already downloaded and verified
(pid=1394) Files already downloaded and verified
(pid=1359) Files already downloaded and verified
(pid=1361) Files already downloaded and verified
(pid=1364) Files already downloaded and verified
(pid=1394) [1,  2000] loss: 2.299
(pid=1361) [1,  2000] loss: 2.304
(pid=1411) [1,  2000] loss: 2.246
(pid=1364) [1,  2000] loss: 2.345
(pid=1366) [1,  2000] loss: 2.147
(pid=1359) [1,  2000] loss: 2.299
(pid=1365) [1,  2000] loss: 2.308
(pid=1360) [1,  2000] loss: 1.943
(pid=1352) [1,  2000] loss: 2.159
(pid=1363) [1,  2000] loss: 2.311
(pid=1394) [1,  4000] loss: 1.075
(pid=1361) [1,  4000] loss: 1.144
Result for DEFAULT_36af6_00008:
  accuracy: 0.1021
  date: 2020-11-17_21-45-58
  done: false
  experiment_id: a423f1e11a884e81b5f4f5a3eb92e205
  experiment_tag: 8_batch_size=16,l1=4,l2=64,lr=0.0001142
  hostname: 8283a11465db
  iterations_since_restore: 1
  loss: 2.303758712387085
  node_ip: 172.17.0.2
  pid: 1365
  should_checkpoint: true
  time_since_restore: 26.65889072418213
  time_this_iter_s: 26.65889072418213
  time_total_s: 26.65889072418213
  timestamp: 1605649558
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 36af6_00008

== Status ==
Memory usage on this node: 9.6/240.1 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -2.303758712387085
Resources requested: 20/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (10 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00000 | RUNNING  |                 |           16 |  128 |   16 | 0.0845979   |         |            |                      |
| DEFAULT_36af6_00001 | RUNNING  |                 |           16 |   32 |  256 | 0.0302299   |         |            |                      |
| DEFAULT_36af6_00002 | RUNNING  |                 |            4 |    8 |   64 | 0.00155654  |         |            |                      |
| DEFAULT_36af6_00003 | RUNNING  |                 |            2 |    4 |    4 | 0.000653041 |         |            |                      |
| DEFAULT_36af6_00004 | RUNNING  |                 |           16 |    8 |   64 | 0.00135388  |         |            |                      |
| DEFAULT_36af6_00005 | RUNNING  |                 |            4 |  256 |  128 | 0.000942874 |         |            |                      |
| DEFAULT_36af6_00006 | RUNNING  |                 |            4 |    4 |    8 | 0.0779381   |         |            |                      |
| DEFAULT_36af6_00007 | RUNNING  |                 |            2 |   32 |   32 | 0.000245868 |         |            |                      |
| DEFAULT_36af6_00008 | RUNNING  | 172.17.0.2:1365 |           16 |    4 |   64 | 0.0001142   | 2.30376 |     0.1021 |                    1 |
| DEFAULT_36af6_00009 | RUNNING  |                 |            8 |    4 |  256 | 0.000333676 |         |            |                      |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1411) [1,  4000] loss: 0.940
Result for DEFAULT_36af6_00001:
  accuracy: 0.2857
  date: 2020-11-17_21-45-59
  done: false
  experiment_id: 8e656487739948289f62de48efea3e7e
  experiment_tag: 1_batch_size=16,l1=32,l2=256,lr=0.03023
  hostname: 8283a11465db
  iterations_since_restore: 1
  loss: 1.885552494430542
  node_ip: 172.17.0.2
  pid: 1360
  should_checkpoint: true
  time_since_restore: 27.055211067199707
  time_this_iter_s: 27.055211067199707
  time_total_s: 27.055211067199707
  timestamp: 1605649559
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 36af6_00001

Result for DEFAULT_36af6_00004:
  accuracy: 0.379
  date: 2020-11-17_21-45-59
  done: false
  experiment_id: d4fb53be5e144dd8b008748bb2810fb5
  experiment_tag: 4_batch_size=16,l1=8,l2=64,lr=0.0013539
  hostname: 8283a11465db
  iterations_since_restore: 1
  loss: 1.6682969062805175
  node_ip: 172.17.0.2
  pid: 1352
  should_checkpoint: true
  time_since_restore: 27.37540078163147
  time_this_iter_s: 27.37540078163147
  time_total_s: 27.37540078163147
  timestamp: 1605649559
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 36af6_00004

(pid=1364) [1,  4000] loss: 1.170
Result for DEFAULT_36af6_00000:
  accuracy: 0.1022
  date: 2020-11-17_21-46-00
  done: true
  experiment_id: e806d5b3d9984ed2bea6d673ecc89e95
  experiment_tag: 0_batch_size=16,l1=128,l2=16,lr=0.084598
  hostname: 8283a11465db
  iterations_since_restore: 1
  loss: 2.305225065612793
  node_ip: 172.17.0.2
  pid: 1363
  should_checkpoint: true
  time_since_restore: 28.876919984817505
  time_this_iter_s: 28.876919984817505
  time_total_s: 28.876919984817505
  timestamp: 1605649560
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 36af6_00000

(pid=1366) [1,  4000] loss: 0.928
(pid=1359) [1,  4000] loss: 1.058
(pid=1394) [1,  6000] loss: 0.694
(pid=1361) [1,  6000] loss: 0.736
(pid=1411) [1,  6000] loss: 0.576
(pid=1364) [1,  6000] loss: 0.783
(pid=1366) [1,  6000] loss: 0.564
Result for DEFAULT_36af6_00009:
  accuracy: 0.2984
  date: 2020-11-17_21-46-14
  done: false
  experiment_id: c505a3e22c7745a98d825bf5a7d81b9d
  experiment_tag: 9_batch_size=8,l1=4,l2=256,lr=0.00033368
  hostname: 8283a11465db
  iterations_since_restore: 1
  loss: 1.8571716680526733
  node_ip: 172.17.0.2
  pid: 1359
  should_checkpoint: true
  time_since_restore: 41.92972016334534
  time_this_iter_s: 41.92972016334534
  time_total_s: 41.92972016334534
  timestamp: 1605649574
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 36af6_00009

== Status ==
Memory usage on this node: 9.0/240.1 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -1.885552494430542
Resources requested: 18/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (9 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00001 | RUNNING    | 172.17.0.2:1360 |           16 |   32 |  256 | 0.0302299   | 1.88555 |     0.2857 |                    1 |
| DEFAULT_36af6_00002 | RUNNING    |                 |            4 |    8 |   64 | 0.00155654  |         |            |                      |
| DEFAULT_36af6_00003 | RUNNING    |                 |            2 |    4 |    4 | 0.000653041 |         |            |                      |
| DEFAULT_36af6_00004 | RUNNING    | 172.17.0.2:1352 |           16 |    8 |   64 | 0.00135388  | 1.6683  |     0.379  |                    1 |
| DEFAULT_36af6_00005 | RUNNING    |                 |            4 |  256 |  128 | 0.000942874 |         |            |                      |
| DEFAULT_36af6_00006 | RUNNING    |                 |            4 |    4 |    8 | 0.0779381   |         |            |                      |
| DEFAULT_36af6_00007 | RUNNING    |                 |            2 |   32 |   32 | 0.000245868 |         |            |                      |
| DEFAULT_36af6_00008 | RUNNING    | 172.17.0.2:1365 |           16 |    4 |   64 | 0.0001142   | 2.30376 |     0.1021 |                    1 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.85717 |     0.2984 |                    1 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1365) [2,  2000] loss: 2.303
(pid=1360) [2,  2000] loss: 1.863
(pid=1352) [2,  2000] loss: 1.592
(pid=1394) [1,  8000] loss: 0.504
(pid=1361) [1,  8000] loss: 0.513
(pid=1411) [1,  8000] loss: 0.403
Result for DEFAULT_36af6_00008:
  accuracy: 0.1021
  date: 2020-11-17_21-46-21
  done: false
  experiment_id: a423f1e11a884e81b5f4f5a3eb92e205
  experiment_tag: 8_batch_size=16,l1=4,l2=64,lr=0.0001142
  hostname: 8283a11465db
  iterations_since_restore: 2
  loss: 2.3028156017303467
  node_ip: 172.17.0.2
  pid: 1365
  should_checkpoint: true
  time_since_restore: 49.90864372253418
  time_this_iter_s: 23.24975299835205
  time_total_s: 49.90864372253418
  timestamp: 1605649581
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: 36af6_00008

== Status ==
Memory usage on this node: 9.0/240.1 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -2.3028156017303467 | Iter 1.000: -1.885552494430542
Resources requested: 18/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (9 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00001 | RUNNING    | 172.17.0.2:1360 |           16 |   32 |  256 | 0.0302299   | 1.88555 |     0.2857 |                    1 |
| DEFAULT_36af6_00002 | RUNNING    |                 |            4 |    8 |   64 | 0.00155654  |         |            |                      |
| DEFAULT_36af6_00003 | RUNNING    |                 |            2 |    4 |    4 | 0.000653041 |         |            |                      |
| DEFAULT_36af6_00004 | RUNNING    | 172.17.0.2:1352 |           16 |    8 |   64 | 0.00135388  | 1.6683  |     0.379  |                    1 |
| DEFAULT_36af6_00005 | RUNNING    |                 |            4 |  256 |  128 | 0.000942874 |         |            |                      |
| DEFAULT_36af6_00006 | RUNNING    |                 |            4 |    4 |    8 | 0.0779381   |         |            |                      |
| DEFAULT_36af6_00007 | RUNNING    |                 |            2 |   32 |   32 | 0.000245868 |         |            |                      |
| DEFAULT_36af6_00008 | RUNNING    | 172.17.0.2:1365 |           16 |    4 |   64 | 0.0001142   | 2.30282 |     0.1021 |                    2 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.85717 |     0.2984 |                    1 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_36af6_00001:
  accuracy: 0.3186
  date: 2020-11-17_21-46-22
  done: false
  experiment_id: 8e656487739948289f62de48efea3e7e
  experiment_tag: 1_batch_size=16,l1=32,l2=256,lr=0.03023
  hostname: 8283a11465db
  iterations_since_restore: 2
  loss: 1.8584628509521484
  node_ip: 172.17.0.2
  pid: 1360
  should_checkpoint: true
  time_since_restore: 50.11302042007446
  time_this_iter_s: 23.057809352874756
  time_total_s: 50.11302042007446
  timestamp: 1605649582
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: 36af6_00001

Result for DEFAULT_36af6_00004:
  accuracy: 0.4548
  date: 2020-11-17_21-46-23
  done: false
  experiment_id: d4fb53be5e144dd8b008748bb2810fb5
  experiment_tag: 4_batch_size=16,l1=8,l2=64,lr=0.0013539
  hostname: 8283a11465db
  iterations_since_restore: 2
  loss: 1.4895487229347228
  node_ip: 172.17.0.2
  pid: 1352
  should_checkpoint: true
  time_since_restore: 51.03372144699097
  time_this_iter_s: 23.658320665359497
  time_total_s: 51.03372144699097
  timestamp: 1605649583
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: 36af6_00004

(pid=1364) [1,  8000] loss: 0.587
(pid=1366) [1,  8000] loss: 0.400
(pid=1359) [2,  2000] loss: 1.792
(pid=1394) [1, 10000] loss: 0.391
(pid=1361) [1, 10000] loss: 0.394
(pid=1411) [1, 10000] loss: 0.311
(pid=1364) [1, 10000] loss: 0.469
(pid=1365) [3,  2000] loss: 2.303
(pid=1360) [3,  2000] loss: 1.895
(pid=1366) [1, 10000] loss: 0.309
(pid=1394) [1, 12000] loss: 0.319
(pid=1352) [3,  2000] loss: 1.420
(pid=1359) [2,  4000] loss: 0.833
(pid=1361) [1, 12000] loss: 0.319
Result for DEFAULT_36af6_00002:
  accuracy: 0.4337
  date: 2020-11-17_21-46-41
  done: false
  experiment_id: 73eebd293f93477d8354d22cef5833df
  experiment_tag: 2_batch_size=4,l1=8,l2=64,lr=0.0015565
  hostname: 8283a11465db
  iterations_since_restore: 1
  loss: 1.4994180078983308
  node_ip: 172.17.0.2
  pid: 1411
  should_checkpoint: true
  time_since_restore: 69.09181833267212
  time_this_iter_s: 69.09181833267212
  time_total_s: 69.09181833267212
  timestamp: 1605649601
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 36af6_00002

== Status ==
Memory usage on this node: 9.0/240.1 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.8584628509521484 | Iter 1.000: -1.8713620812416076
Resources requested: 18/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (9 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00001 | RUNNING    | 172.17.0.2:1360 |           16 |   32 |  256 | 0.0302299   | 1.85846 |     0.3186 |                    2 |
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.49942 |     0.4337 |                    1 |
| DEFAULT_36af6_00003 | RUNNING    |                 |            2 |    4 |    4 | 0.000653041 |         |            |                      |
| DEFAULT_36af6_00004 | RUNNING    | 172.17.0.2:1352 |           16 |    8 |   64 | 0.00135388  | 1.48955 |     0.4548 |                    2 |
| DEFAULT_36af6_00005 | RUNNING    |                 |            4 |  256 |  128 | 0.000942874 |         |            |                      |
| DEFAULT_36af6_00006 | RUNNING    |                 |            4 |    4 |    8 | 0.0779381   |         |            |                      |
| DEFAULT_36af6_00007 | RUNNING    |                 |            2 |   32 |   32 | 0.000245868 |         |            |                      |
| DEFAULT_36af6_00008 | RUNNING    | 172.17.0.2:1365 |           16 |    4 |   64 | 0.0001142   | 2.30282 |     0.1021 |                    2 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.85717 |     0.2984 |                    1 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_36af6_00006:
  accuracy: 0.0989
  date: 2020-11-17_21-46-42
  done: true
  experiment_id: 210025965f8442108cb1785c47c6d390
  experiment_tag: 6_batch_size=4,l1=4,l2=8,lr=0.077938
  hostname: 8283a11465db
  iterations_since_restore: 1
  loss: 2.3352626296043395
  node_ip: 172.17.0.2
  pid: 1364
  should_checkpoint: true
  time_since_restore: 70.21091294288635
  time_this_iter_s: 70.21091294288635
  time_total_s: 70.21091294288635
  timestamp: 1605649602
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 36af6_00006

Result for DEFAULT_36af6_00008:
  accuracy: 0.0974
  date: 2020-11-17_21-46-45
  done: false
  experiment_id: a423f1e11a884e81b5f4f5a3eb92e205
  experiment_tag: 8_batch_size=16,l1=4,l2=64,lr=0.0001142
  hostname: 8283a11465db
  iterations_since_restore: 3
  loss: 2.30270735168457
  node_ip: 172.17.0.2
  pid: 1365
  should_checkpoint: true
  time_since_restore: 73.16434097290039
  time_this_iter_s: 23.25569725036621
  time_total_s: 73.16434097290039
  timestamp: 1605649605
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: 36af6_00008

Result for DEFAULT_36af6_00001:
  accuracy: 0.305
  date: 2020-11-17_21-46-45
  done: false
  experiment_id: 8e656487739948289f62de48efea3e7e
  experiment_tag: 1_batch_size=16,l1=32,l2=256,lr=0.03023
  hostname: 8283a11465db
  iterations_since_restore: 3
  loss: 1.8816396200180054
  node_ip: 172.17.0.2
  pid: 1360
  should_checkpoint: true
  time_since_restore: 73.37080597877502
  time_this_iter_s: 23.25778555870056
  time_total_s: 73.37080597877502
  timestamp: 1605649605
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: 36af6_00001

Result for DEFAULT_36af6_00005:
  accuracy: 0.415
  date: 2020-11-17_21-46-46
  done: false
  experiment_id: 90fceda1a64e4151a4e2229c7baf1924
  experiment_tag: 5_batch_size=4,l1=256,l2=128,lr=0.00094287
  hostname: 8283a11465db
  iterations_since_restore: 1
  loss: 1.5815452996611594
  node_ip: 172.17.0.2
  pid: 1366
  should_checkpoint: true
  time_since_restore: 74.480295419693
  time_this_iter_s: 74.480295419693
  time_total_s: 74.480295419693
  timestamp: 1605649606
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 36af6_00005

== Status ==
Memory usage on this node: 8.2/240.1 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.8584628509521484 | Iter 1.000: -1.8713620812416076
Resources requested: 16/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (8 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00001 | RUNNING    | 172.17.0.2:1360 |           16 |   32 |  256 | 0.0302299   | 1.88164 |     0.305  |                    3 |
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.49942 |     0.4337 |                    1 |
| DEFAULT_36af6_00003 | RUNNING    |                 |            2 |    4 |    4 | 0.000653041 |         |            |                      |
| DEFAULT_36af6_00004 | RUNNING    | 172.17.0.2:1352 |           16 |    8 |   64 | 0.00135388  | 1.48955 |     0.4548 |                    2 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.58155 |     0.415  |                    1 |
| DEFAULT_36af6_00007 | RUNNING    |                 |            2 |   32 |   32 | 0.000245868 |         |            |                      |
| DEFAULT_36af6_00008 | RUNNING    | 172.17.0.2:1365 |           16 |    4 |   64 | 0.0001142   | 2.30271 |     0.0974 |                    3 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.85717 |     0.2984 |                    1 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_36af6_00004:
  accuracy: 0.5151
  date: 2020-11-17_21-46-46
  done: false
  experiment_id: d4fb53be5e144dd8b008748bb2810fb5
  experiment_tag: 4_batch_size=16,l1=8,l2=64,lr=0.0013539
  hostname: 8283a11465db
  iterations_since_restore: 3
  loss: 1.3414747207641602
  node_ip: 172.17.0.2
  pid: 1352
  should_checkpoint: true
  time_since_restore: 74.57542896270752
  time_this_iter_s: 23.541707515716553
  time_total_s: 74.57542896270752
  timestamp: 1605649606
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: 36af6_00004

(pid=1394) [1, 14000] loss: 0.264
Result for DEFAULT_36af6_00009:
  accuracy: 0.3728
  date: 2020-11-17_21-46-51
  done: false
  experiment_id: c505a3e22c7745a98d825bf5a7d81b9d
  experiment_tag: 9_batch_size=8,l1=4,l2=256,lr=0.00033368
  hostname: 8283a11465db
  iterations_since_restore: 2
  loss: 1.6582274404525756
  node_ip: 172.17.0.2
  pid: 1359
  should_checkpoint: true
  time_since_restore: 79.08877611160278
  time_this_iter_s: 37.159055948257446
  time_total_s: 79.08877611160278
  timestamp: 1605649611
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: 36af6_00009

(pid=1361) [1, 14000] loss: 0.262
(pid=1411) [2,  2000] loss: 1.510
(pid=1366) [2,  2000] loss: 1.479
(pid=1394) [1, 16000] loss: 0.222
(pid=1365) [4,  2000] loss: 2.303
(pid=1360) [4,  2000] loss: 1.919
(pid=1361) [1, 16000] loss: 0.221
(pid=1352) [4,  2000] loss: 1.327
(pid=1411) [2,  4000] loss: 0.740
(pid=1359) [3,  2000] loss: 1.594
Result for DEFAULT_36af6_00001:
  accuracy: 0.2858
  date: 2020-11-17_21-47-07
  done: false
  experiment_id: 8e656487739948289f62de48efea3e7e
  experiment_tag: 1_batch_size=16,l1=32,l2=256,lr=0.03023
  hostname: 8283a11465db
  iterations_since_restore: 4
  loss: 1.9270733840942382
  node_ip: 172.17.0.2
  pid: 1360
  should_checkpoint: true
  time_since_restore: 95.71826243400574
  time_this_iter_s: 22.347456455230713
  time_total_s: 95.71826243400574
  timestamp: 1605649627
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: 36af6_00001

== Status ==
Memory usage on this node: 8.2/240.1 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: None | Iter 4.000: -1.9270733840942382 | Iter 2.000: -1.758345145702362 | Iter 1.000: -1.8713620812416076
Resources requested: 16/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (8 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00001 | RUNNING    | 172.17.0.2:1360 |           16 |   32 |  256 | 0.0302299   | 1.92707 |     0.2858 |                    4 |
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.49942 |     0.4337 |                    1 |
| DEFAULT_36af6_00003 | RUNNING    |                 |            2 |    4 |    4 | 0.000653041 |         |            |                      |
| DEFAULT_36af6_00004 | RUNNING    | 172.17.0.2:1352 |           16 |    8 |   64 | 0.00135388  | 1.34147 |     0.5151 |                    3 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.58155 |     0.415  |                    1 |
| DEFAULT_36af6_00007 | RUNNING    |                 |            2 |   32 |   32 | 0.000245868 |         |            |                      |
| DEFAULT_36af6_00008 | RUNNING    | 172.17.0.2:1365 |           16 |    4 |   64 | 0.0001142   | 2.30271 |     0.0974 |                    3 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.65823 |     0.3728 |                    2 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_36af6_00008:
  accuracy: 0.0974
  date: 2020-11-17_21-47-07
  done: true
  experiment_id: a423f1e11a884e81b5f4f5a3eb92e205
  experiment_tag: 8_batch_size=16,l1=4,l2=64,lr=0.0001142
  hostname: 8283a11465db
  iterations_since_restore: 4
  loss: 2.3027472324371336
  node_ip: 172.17.0.2
  pid: 1365
  should_checkpoint: true
  time_since_restore: 95.63674426078796
  time_this_iter_s: 22.472403287887573
  time_total_s: 95.63674426078796
  timestamp: 1605649627
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: 36af6_00008

Result for DEFAULT_36af6_00004:
  accuracy: 0.5405
  date: 2020-11-17_21-47-09
  done: false
  experiment_id: d4fb53be5e144dd8b008748bb2810fb5
  experiment_tag: 4_batch_size=16,l1=8,l2=64,lr=0.0013539
  hostname: 8283a11465db
  iterations_since_restore: 4
  loss: 1.279247055053711
  node_ip: 172.17.0.2
  pid: 1352
  should_checkpoint: true
  time_since_restore: 97.17229580879211
  time_this_iter_s: 22.596866846084595
  time_total_s: 97.17229580879211
  timestamp: 1605649629
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: 36af6_00004

(pid=1394) [1, 18000] loss: 0.191
(pid=1366) [2,  4000] loss: 0.722
(pid=1361) [1, 18000] loss: 0.191
(pid=1411) [2,  6000] loss: 0.485
(pid=1359) [3,  4000] loss: 0.782
(pid=1394) [1, 20000] loss: 0.170
(pid=1361) [1, 20000] loss: 0.169
(pid=1366) [2,  6000] loss: 0.469
(pid=1360) [5,  2000] loss: 1.970
(pid=1352) [5,  2000] loss: 1.270
(pid=1411) [2,  8000] loss: 0.358
Result for DEFAULT_36af6_00009:
  accuracy: 0.399
  date: 2020-11-17_21-47-26
  done: false
  experiment_id: c505a3e22c7745a98d825bf5a7d81b9d
  experiment_tag: 9_batch_size=8,l1=4,l2=256,lr=0.00033368
  hostname: 8283a11465db
  iterations_since_restore: 3
  loss: 1.548436265516281
  node_ip: 172.17.0.2
  pid: 1359
  should_checkpoint: true
  time_since_restore: 114.82469058036804
  time_this_iter_s: 35.73591446876526
  time_total_s: 114.82469058036804
  timestamp: 1605649646
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: 36af6_00009

== Status ==
Memory usage on this node: 7.7/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.9270733840942382 | Iter 2.000: -1.758345145702362 | Iter 1.000: -1.8713620812416076
Resources requested: 14/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00001 | RUNNING    | 172.17.0.2:1360 |           16 |   32 |  256 | 0.0302299   | 1.92707 |     0.2858 |                    4 |
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.49942 |     0.4337 |                    1 |
| DEFAULT_36af6_00003 | RUNNING    |                 |            2 |    4 |    4 | 0.000653041 |         |            |                      |
| DEFAULT_36af6_00004 | RUNNING    | 172.17.0.2:1352 |           16 |    8 |   64 | 0.00135388  | 1.27925 |     0.5405 |                    4 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.58155 |     0.415  |                    1 |
| DEFAULT_36af6_00007 | RUNNING    |                 |            2 |   32 |   32 | 0.000245868 |         |            |                      |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.54844 |     0.399  |                    3 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_36af6_00001:
  accuracy: 0.2449
  date: 2020-11-17_21-47-29
  done: false
  experiment_id: 8e656487739948289f62de48efea3e7e
  experiment_tag: 1_batch_size=16,l1=32,l2=256,lr=0.03023
  hostname: 8283a11465db
  iterations_since_restore: 5
  loss: 2.041396103477478
  node_ip: 172.17.0.2
  pid: 1360
  should_checkpoint: true
  time_since_restore: 117.50055432319641
  time_this_iter_s: 21.782291889190674
  time_total_s: 117.50055432319641
  timestamp: 1605649649
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: 36af6_00001

Result for DEFAULT_36af6_00004:
  accuracy: 0.5483
  date: 2020-11-17_21-47-31
  done: false
  experiment_id: d4fb53be5e144dd8b008748bb2810fb5
  experiment_tag: 4_batch_size=16,l1=8,l2=64,lr=0.0013539
  hostname: 8283a11465db
  iterations_since_restore: 5
  loss: 1.2623056431770325
  node_ip: 172.17.0.2
  pid: 1352
  should_checkpoint: true
  time_since_restore: 119.15276765823364
  time_this_iter_s: 21.98047184944153
  time_total_s: 119.15276765823364
  timestamp: 1605649651
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: 36af6_00004

(pid=1366) [2,  8000] loss: 0.341
Result for DEFAULT_36af6_00003:
  accuracy: 0.3863
  date: 2020-11-17_21-47-33
  done: false
  experiment_id: e59bc8a0908b442eb6d30667cb56539b
  experiment_tag: 3_batch_size=2,l1=4,l2=4,lr=0.00065304
  hostname: 8283a11465db
  iterations_since_restore: 1
  loss: 1.6439415543258191
  node_ip: 172.17.0.2
  pid: 1394
  should_checkpoint: true
  time_since_restore: 121.09425663948059
  time_this_iter_s: 121.09425663948059
  time_total_s: 121.09425663948059
  timestamp: 1605649653
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 36af6_00003

== Status ==
Memory usage on this node: 7.8/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.9270733840942382 | Iter 2.000: -1.758345145702362 | Iter 1.000: -1.8571716680526733
Resources requested: 14/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00001 | RUNNING    | 172.17.0.2:1360 |           16 |   32 |  256 | 0.0302299   | 2.0414  |     0.2449 |                    5 |
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.49942 |     0.4337 |                    1 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.64394 |     0.3863 |                    1 |
| DEFAULT_36af6_00004 | RUNNING    | 172.17.0.2:1352 |           16 |    8 |   64 | 0.00135388  | 1.26231 |     0.5483 |                    5 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.58155 |     0.415  |                    1 |
| DEFAULT_36af6_00007 | RUNNING    |                 |            2 |   32 |   32 | 0.000245868 |         |            |                      |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.54844 |     0.399  |                    3 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_36af6_00007:
  accuracy: 0.3981
  date: 2020-11-17_21-47-34
  done: false
  experiment_id: 982797bde02f40e4a9744d76174e8913
  experiment_tag: 7_batch_size=2,l1=32,l2=32,lr=0.00024587
  hostname: 8283a11465db
  iterations_since_restore: 1
  loss: 1.6323244907677175
  node_ip: 172.17.0.2
  pid: 1361
  should_checkpoint: true
  time_since_restore: 122.78179550170898
  time_this_iter_s: 122.78179550170898
  time_total_s: 122.78179550170898
  timestamp: 1605649654
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 36af6_00007

(pid=1411) [2, 10000] loss: 0.284
(pid=1359) [4,  2000] loss: 1.531
Result for DEFAULT_36af6_00002:
  accuracy: 0.5038
  date: 2020-11-17_21-47-42
  done: false
  experiment_id: 73eebd293f93477d8354d22cef5833df
  experiment_tag: 2_batch_size=4,l1=8,l2=64,lr=0.0015565
  hostname: 8283a11465db
  iterations_since_restore: 2
  loss: 1.378735035699606
  node_ip: 172.17.0.2
  pid: 1411
  should_checkpoint: true
  time_since_restore: 130.33464217185974
  time_this_iter_s: 61.24282383918762
  time_total_s: 130.33464217185974
  timestamp: 1605649662
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: 36af6_00002

== Status ==
Memory usage on this node: 7.8/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.9270733840942382 | Iter 2.000: -1.6582274404525756 | Iter 1.000: -1.7627342871665954
Resources requested: 14/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00001 | RUNNING    | 172.17.0.2:1360 |           16 |   32 |  256 | 0.0302299   | 2.0414  |     0.2449 |                    5 |
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.37874 |     0.5038 |                    2 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.64394 |     0.3863 |                    1 |
| DEFAULT_36af6_00004 | RUNNING    | 172.17.0.2:1352 |           16 |    8 |   64 | 0.00135388  | 1.26231 |     0.5483 |                    5 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.58155 |     0.415  |                    1 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.63232 |     0.3981 |                    1 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.54844 |     0.399  |                    3 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1394) [2,  2000] loss: 1.698
(pid=1366) [2, 10000] loss: 0.268
(pid=1360) [6,  2000] loss: 2.001
(pid=1361) [2,  2000] loss: 1.610
(pid=1352) [6,  2000] loss: 1.213
Result for DEFAULT_36af6_00001:
  accuracy: 0.2561
  date: 2020-11-17_21-47-51
  done: false
  experiment_id: 8e656487739948289f62de48efea3e7e
  experiment_tag: 1_batch_size=16,l1=32,l2=256,lr=0.03023
  hostname: 8283a11465db
  iterations_since_restore: 6
  loss: 2.0380940685272217
  node_ip: 172.17.0.2
  pid: 1360
  should_checkpoint: true
  time_since_restore: 139.5811927318573
  time_this_iter_s: 22.08063840866089
  time_total_s: 139.5811927318573
  timestamp: 1605649671
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: 36af6_00001

== Status ==
Memory usage on this node: 7.8/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.9270733840942382 | Iter 2.000: -1.6582274404525756 | Iter 1.000: -1.7627342871665954
Resources requested: 14/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00001 | RUNNING    | 172.17.0.2:1360 |           16 |   32 |  256 | 0.0302299   | 2.03809 |     0.2561 |                    6 |
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.37874 |     0.5038 |                    2 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.64394 |     0.3863 |                    1 |
| DEFAULT_36af6_00004 | RUNNING    | 172.17.0.2:1352 |           16 |    8 |   64 | 0.00135388  | 1.26231 |     0.5483 |                    5 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.58155 |     0.415  |                    1 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.63232 |     0.3981 |                    1 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.54844 |     0.399  |                    3 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1359) [4,  4000] loss: 0.754
Result for DEFAULT_36af6_00005:
  accuracy: 0.5127
  date: 2020-11-17_21-47-51
  done: false
  experiment_id: 90fceda1a64e4151a4e2229c7baf1924
  experiment_tag: 5_batch_size=4,l1=256,l2=128,lr=0.00094287
  hostname: 8283a11465db
  iterations_since_restore: 2
  loss: 1.3479058542191982
  node_ip: 172.17.0.2
  pid: 1366
  should_checkpoint: true
  time_since_restore: 139.74493980407715
  time_this_iter_s: 65.26464438438416
  time_total_s: 139.74493980407715
  timestamp: 1605649671
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: 36af6_00005

(pid=1394) [2,  4000] loss: 0.824
Result for DEFAULT_36af6_00004:
  accuracy: 0.5688
  date: 2020-11-17_21-47-52
  done: false
  experiment_id: d4fb53be5e144dd8b008748bb2810fb5
  experiment_tag: 4_batch_size=16,l1=8,l2=64,lr=0.0013539
  hostname: 8283a11465db
  iterations_since_restore: 6
  loss: 1.213833190059662
  node_ip: 172.17.0.2
  pid: 1352
  should_checkpoint: true
  time_since_restore: 140.74087166786194
  time_this_iter_s: 21.588104009628296
  time_total_s: 140.74087166786194
  timestamp: 1605649672
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: 36af6_00004

(pid=1411) [3,  2000] loss: 1.391
(pid=1361) [2,  4000] loss: 0.803
Result for DEFAULT_36af6_00009:
  accuracy: 0.4367
  date: 2020-11-17_21-48-02
  done: false
  experiment_id: c505a3e22c7745a98d825bf5a7d81b9d
  experiment_tag: 9_batch_size=8,l1=4,l2=256,lr=0.00033368
  hostname: 8283a11465db
  iterations_since_restore: 4
  loss: 1.4843962660312653
  node_ip: 172.17.0.2
  pid: 1359
  should_checkpoint: true
  time_since_restore: 150.0106806755066
  time_this_iter_s: 35.18599009513855
  time_total_s: 150.0106806755066
  timestamp: 1605649682
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: 36af6_00009
  (pid=1394) [2,  6000] loss: 0.537

== Status ==
Memory usage on this node: 7.8/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.7057348250627518 | Iter 2.000: -1.5738880816936494 | Iter 1.000: -1.7627342871665954
Resources requested: 14/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00001 | RUNNING    | 172.17.0.2:1360 |           16 |   32 |  256 | 0.0302299   | 2.03809 |     0.2561 |                    6 |
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.37874 |     0.5038 |                    2 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.64394 |     0.3863 |                    1 |
| DEFAULT_36af6_00004 | RUNNING    | 172.17.0.2:1352 |           16 |    8 |   64 | 0.00135388  | 1.21383 |     0.5688 |                    6 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.34791 |     0.5127 |                    2 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.63232 |     0.3981 |                    1 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.4844  |     0.4367 |                    4 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1366) [3,  2000] loss: 1.285
(pid=1411) [3,  4000] loss: 0.699
(pid=1361) [2,  6000] loss: 0.518
(pid=1360) [7,  2000] loss: 1.983
(pid=1352) [7,  2000] loss: 1.173
(pid=1394) [2,  8000] loss: 0.397
Result for DEFAULT_36af6_00001:
  accuracy: 0.2226
  date: 2020-11-17_21-48-13
  done: false
  experiment_id: 8e656487739948289f62de48efea3e7e
  experiment_tag: 1_batch_size=16,l1=32,l2=256,lr=0.03023
  hostname: 8283a11465db
  iterations_since_restore: 7
  loss: 2.024003084373474
  node_ip: 172.17.0.2
  pid: 1360
  should_checkpoint: true
  time_since_restore: 161.32618713378906
  time_this_iter_s: 21.744994401931763
  time_total_s: 161.32618713378906
  timestamp: 1605649693
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: 36af6_00001

== Status ==
Memory usage on this node: 7.8/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.7057348250627518 | Iter 2.000: -1.5738880816936494 | Iter 1.000: -1.7627342871665954
Resources requested: 14/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00001 | RUNNING    | 172.17.0.2:1360 |           16 |   32 |  256 | 0.0302299   | 2.024   |     0.2226 |                    7 |
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.37874 |     0.5038 |                    2 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.64394 |     0.3863 |                    1 |
| DEFAULT_36af6_00004 | RUNNING    | 172.17.0.2:1352 |           16 |    8 |   64 | 0.00135388  | 1.21383 |     0.5688 |                    6 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.34791 |     0.5127 |                    2 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.63232 |     0.3981 |                    1 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.4844  |     0.4367 |                    4 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1411) [3,  6000] loss: 0.461
(pid=1359) [5,  2000] loss: 1.479
Result for DEFAULT_36af6_00004:
  accuracy: 0.5814
  date: 2020-11-17_21-48-14
  done: false
  experiment_id: d4fb53be5e144dd8b008748bb2810fb5
  experiment_tag: 4_batch_size=16,l1=8,l2=64,lr=0.0013539
  hostname: 8283a11465db
  iterations_since_restore: 7
  loss: 1.1786626031398773
  node_ip: 172.17.0.2
  pid: 1352
  should_checkpoint: true
  time_since_restore: 162.54430413246155
  time_this_iter_s: 21.80343246459961
  time_total_s: 162.54430413246155
  timestamp: 1605649694
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: 36af6_00004

(pid=1361) [2,  8000] loss: 0.386
(pid=1366) [3,  4000] loss: 0.635
(pid=1394) [2, 10000] loss: 0.317
(pid=1361) [2, 10000] loss: 0.310
(pid=1411) [3,  8000] loss: 0.337
(pid=1366) [3,  6000] loss: 0.408
(pid=1359) [5,  4000] loss: 0.733
(pid=1360) [8,  2000] loss: 2.073
(pid=1352) [8,  2000] loss: 1.155
(pid=1394) [2, 12000] loss: 0.262
(pid=1361) [2, 12000] loss: 0.250
(pid=1411) [3, 10000] loss: 0.271
Result for DEFAULT_36af6_00001:
  accuracy: 0.1667
  date: 2020-11-17_21-48-35
  done: false
  experiment_id: 8e656487739948289f62de48efea3e7e
  experiment_tag: 1_batch_size=16,l1=32,l2=256,lr=0.03023
  hostname: 8283a11465db
  iterations_since_restore: 8
  loss: 2.074481844329834
  node_ip: 172.17.0.2
  pid: 1360
  should_checkpoint: true
  time_since_restore: 183.24734449386597
  time_this_iter_s: 21.921157360076904
  time_total_s: 183.24734449386597
  timestamp: 1605649715
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: 36af6_00001

== Status ==
Memory usage on this node: 7.7/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -2.074481844329834 | Iter 4.000: -1.7057348250627518 | Iter 2.000: -1.5738880816936494 | Iter 1.000: -1.7627342871665954
Resources requested: 14/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00001 | RUNNING    | 172.17.0.2:1360 |           16 |   32 |  256 | 0.0302299   | 2.07448 |     0.1667 |                    8 |
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.37874 |     0.5038 |                    2 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.64394 |     0.3863 |                    1 |
| DEFAULT_36af6_00004 | RUNNING    | 172.17.0.2:1352 |           16 |    8 |   64 | 0.00135388  | 1.17866 |     0.5814 |                    7 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.34791 |     0.5127 |                    2 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.63232 |     0.3981 |                    1 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.4844  |     0.4367 |                    4 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_36af6_00004:
  accuracy: 0.5859
  date: 2020-11-17_21-48-36
  done: false
  experiment_id: d4fb53be5e144dd8b008748bb2810fb5
  experiment_tag: 4_batch_size=16,l1=8,l2=64,lr=0.0013539
  hostname: 8283a11465db
  iterations_since_restore: 8
  loss: 1.1793484716415406
  node_ip: 172.17.0.2
  pid: 1352
  should_checkpoint: true
  time_since_restore: 184.07562708854675
  time_this_iter_s: 21.531322956085205
  time_total_s: 184.07562708854675
  timestamp: 1605649716
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: 36af6_00004

Result for DEFAULT_36af6_00009:
  accuracy: 0.4308
  date: 2020-11-17_21-48-36
  done: false
  experiment_id: c505a3e22c7745a98d825bf5a7d81b9d
  experiment_tag: 9_batch_size=8,l1=4,l2=256,lr=0.00033368
  hostname: 8283a11465db
  iterations_since_restore: 5
  loss: 1.4775157031536101
  node_ip: 172.17.0.2
  pid: 1359
  should_checkpoint: true
  time_since_restore: 184.74778056144714
  time_this_iter_s: 34.73709988594055
  time_total_s: 184.74778056144714
  timestamp: 1605649716
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: 36af6_00009

(pid=1366) [3,  8000] loss: 0.306
(pid=1394) [2, 14000] loss: 0.218
Result for DEFAULT_36af6_00002:
  accuracy: 0.4985
  date: 2020-11-17_21-48-42
  done: false
  experiment_id: 73eebd293f93477d8354d22cef5833df
  experiment_tag: 2_batch_size=4,l1=8,l2=64,lr=0.0015565
  hostname: 8283a11465db
  iterations_since_restore: 3
  loss: 1.381571624109149
  node_ip: 172.17.0.2
  pid: 1411
  should_checkpoint: true
  time_since_restore: 190.64486002922058
  time_this_iter_s: 60.31021785736084
  time_total_s: 190.64486002922058
  timestamp: 1605649722
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: 36af6_00002

== Status ==
Memory usage on this node: 7.8/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -1.6269151579856873 | Iter 4.000: -1.7057348250627518 | Iter 2.000: -1.5738880816936494 | Iter 1.000: -1.7627342871665954
Resources requested: 14/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00001 | RUNNING    | 172.17.0.2:1360 |           16 |   32 |  256 | 0.0302299   | 2.07448 |     0.1667 |                    8 |
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.38157 |     0.4985 |                    3 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.64394 |     0.3863 |                    1 |
| DEFAULT_36af6_00004 | RUNNING    | 172.17.0.2:1352 |           16 |    8 |   64 | 0.00135388  | 1.17935 |     0.5859 |                    8 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.34791 |     0.5127 |                    2 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.63232 |     0.3981 |                    1 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.47752 |     0.4308 |                    5 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1361) [2, 14000] loss: 0.213
(pid=1366) [3, 10000] loss: 0.244
(pid=1359) [6,  2000] loss: 1.432
(pid=1394) [2, 16000] loss: 0.190
(pid=1360) [9,  2000] loss: 2.126
(pid=1352) [9,  2000] loss: 1.131
(pid=1411) [4,  2000] loss: 1.346
(pid=1361) [2, 16000] loss: 0.184
Result for DEFAULT_36af6_00005:
  accuracy: 0.5654
  date: 2020-11-17_21-48-56
  done: false
  experiment_id: 90fceda1a64e4151a4e2229c7baf1924
  experiment_tag: 5_batch_size=4,l1=256,l2=128,lr=0.00094287
  hostname: 8283a11465db
  iterations_since_restore: 3
  loss: 1.2136261095643044
  node_ip: 172.17.0.2
  pid: 1366
  should_checkpoint: true
  time_since_restore: 204.39768624305725
  time_this_iter_s: 64.6527464389801
  time_total_s: 204.39768624305725
  timestamp: 1605649736
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: 36af6_00005

== Status ==
Memory usage on this node: 7.7/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -1.6269151579856873 | Iter 4.000: -1.7057348250627518 | Iter 2.000: -1.5738880816936494 | Iter 1.000: -1.7627342871665954
Resources requested: 14/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00001 | RUNNING    | 172.17.0.2:1360 |           16 |   32 |  256 | 0.0302299   | 2.07448 |     0.1667 |                    8 |
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.38157 |     0.4985 |                    3 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.64394 |     0.3863 |                    1 |
| DEFAULT_36af6_00004 | RUNNING    | 172.17.0.2:1352 |           16 |    8 |   64 | 0.00135388  | 1.17935 |     0.5859 |                    8 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.21363 |     0.5654 |                    3 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.63232 |     0.3981 |                    1 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.47752 |     0.4308 |                    5 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_36af6_00001:
  accuracy: 0.2023
  date: 2020-11-17_21-48-57
  done: false
  experiment_id: 8e656487739948289f62de48efea3e7e
  experiment_tag: 1_batch_size=16,l1=32,l2=256,lr=0.03023
  hostname: 8283a11465db
  iterations_since_restore: 9
  loss: 2.0678629428863524
  node_ip: 172.17.0.2
  pid: 1360
  should_checkpoint: true
  time_since_restore: 205.65170431137085
  time_this_iter_s: 22.404359817504883
  time_total_s: 205.65170431137085
  timestamp: 1605649737
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: 36af6_00001

Result for DEFAULT_36af6_00004:
  accuracy: 0.5748
  date: 2020-11-17_21-48-57
  done: false
  experiment_id: d4fb53be5e144dd8b008748bb2810fb5
  experiment_tag: 4_batch_size=16,l1=8,l2=64,lr=0.0013539
  hostname: 8283a11465db
  iterations_since_restore: 9
  loss: 1.2079984548568725
  node_ip: 172.17.0.2
  pid: 1352
  should_checkpoint: true
  time_since_restore: 205.78290629386902
  time_this_iter_s: 21.707279205322266
  time_total_s: 205.78290629386902
  timestamp: 1605649737
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: 36af6_00004

(pid=1394) [2, 18000] loss: 0.174
(pid=1359) [6,  4000] loss: 0.713
(pid=1361) [2, 18000] loss: 0.160
(pid=1411) [4,  4000] loss: 0.665
(pid=1366) [4,  2000] loss: 1.128
(pid=1394) [2, 20000] loss: 0.152
Result for DEFAULT_36af6_00009:
  accuracy: 0.4491
  date: 2020-11-17_21-49-11
  done: false
  experiment_id: c505a3e22c7745a98d825bf5a7d81b9d
  experiment_tag: 9_batch_size=8,l1=4,l2=256,lr=0.00033368
  hostname: 8283a11465db
  iterations_since_restore: 6
  loss: 1.4679613151550293
  node_ip: 172.17.0.2
  pid: 1359
  should_checkpoint: true
  time_since_restore: 219.3292829990387
  time_this_iter_s: 34.58150243759155
  time_total_s: 219.3292829990387
  timestamp: 1605649751
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: 36af6_00009

== Status ==
Memory usage on this node: 7.8/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -1.6269151579856873 | Iter 4.000: -1.7057348250627518 | Iter 2.000: -1.5738880816936494 | Iter 1.000: -1.7627342871665954
Resources requested: 14/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00001 | RUNNING    | 172.17.0.2:1360 |           16 |   32 |  256 | 0.0302299   | 2.06786 |     0.2023 |                    9 |
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.38157 |     0.4985 |                    3 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.64394 |     0.3863 |                    1 |
| DEFAULT_36af6_00004 | RUNNING    | 172.17.0.2:1352 |           16 |    8 |   64 | 0.00135388  | 1.208   |     0.5748 |                    9 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.21363 |     0.5654 |                    3 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.63232 |     0.3981 |                    1 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.46796 |     0.4491 |                    6 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1352) [10,  2000] loss: 1.113
(pid=1360) [10,  2000] loss: 2.119
(pid=1361) [2, 20000] loss: 0.143
(pid=1411) [4,  6000] loss: 0.445
(pid=1366) [4,  4000] loss: 0.576
Result for DEFAULT_36af6_00004:
  accuracy: 0.5961
  date: 2020-11-17_21-49-19
  done: true
  experiment_id: d4fb53be5e144dd8b008748bb2810fb5
  experiment_tag: 4_batch_size=16,l1=8,l2=64,lr=0.0013539
  hostname: 8283a11465db
  iterations_since_restore: 10
  loss: 1.1368304990291596
  node_ip: 172.17.0.2
  pid: 1352
  should_checkpoint: true
  time_since_restore: 227.6813929080963
  time_this_iter_s: 21.898486614227295
  time_total_s: 227.6813929080963
  timestamp: 1605649759
  timesteps_since_restore: 0
  training_iteration: 10
  trial_id: 36af6_00004

== Status ==
Memory usage on this node: 7.7/240.1 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: -1.6269151579856873 | Iter 4.000: -1.7057348250627518 | Iter 2.000: -1.5738880816936494 | Iter 1.000: -1.7627342871665954
Resources requested: 14/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00001 | RUNNING    | 172.17.0.2:1360 |           16 |   32 |  256 | 0.0302299   | 2.06786 |     0.2023 |                    9 |
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.38157 |     0.4985 |                    3 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.64394 |     0.3863 |                    1 |
| DEFAULT_36af6_00004 | RUNNING    | 172.17.0.2:1352 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.21363 |     0.5654 |                    3 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.63232 |     0.3981 |                    1 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.46796 |     0.4491 |                    6 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_36af6_00001:
  accuracy: 0.1573
  date: 2020-11-17_21-49-20
  done: true
  experiment_id: 8e656487739948289f62de48efea3e7e
  experiment_tag: 1_batch_size=16,l1=32,l2=256,lr=0.03023
  hostname: 8283a11465db
  iterations_since_restore: 10
  loss: 2.1924891891479494
  node_ip: 172.17.0.2
  pid: 1360
  should_checkpoint: true
  time_since_restore: 228.02747011184692
  time_this_iter_s: 22.375765800476074
  time_total_s: 228.02747011184692
  timestamp: 1605649760
  timesteps_since_restore: 0
  training_iteration: 10
  trial_id: 36af6_00001

Result for DEFAULT_36af6_00003:
  accuracy: 0.4422
  date: 2020-11-17_21-49-22
  done: false
  experiment_id: e59bc8a0908b442eb6d30667cb56539b
  experiment_tag: 3_batch_size=2,l1=4,l2=4,lr=0.00065304
  hostname: 8283a11465db
  iterations_since_restore: 2
  loss: 1.4820770419448615
  node_ip: 172.17.0.2
  pid: 1394
  should_checkpoint: true
  time_since_restore: 230.17002820968628
  time_this_iter_s: 109.07577157020569
  time_total_s: 230.17002820968628
  timestamp: 1605649762
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: 36af6_00003

(pid=1359) [7,  2000] loss: 1.398
(pid=1411) [4,  8000] loss: 0.332
Result for DEFAULT_36af6_00007:
  accuracy: 0.492
  date: 2020-11-17_21-49-26
  done: false
  experiment_id: 982797bde02f40e4a9744d76174e8913
  experiment_tag: 7_batch_size=2,l1=32,l2=32,lr=0.00024587
  hostname: 8283a11465db
  iterations_since_restore: 2
  loss: 1.3931830716356635
  node_ip: 172.17.0.2
  pid: 1361
  should_checkpoint: true
  time_since_restore: 234.17733216285706
  time_this_iter_s: 111.39553666114807
  time_total_s: 234.17733216285706
  timestamp: 1605649766
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: 36af6_00007

== Status ==
Memory usage on this node: 6.6/240.1 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -1.6269151579856873 | Iter 4.000: -1.7057348250627518 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 10/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (5 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.38157 |     0.4985 |                    3 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.48208 |     0.4422 |                    2 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.21363 |     0.5654 |                    3 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.39318 |     0.492  |                    2 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.46796 |     0.4491 |                    6 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1366) [4,  6000] loss: 0.382
(pid=1394) [3,  2000] loss: 1.477
(pid=1411) [4, 10000] loss: 0.266
(pid=1359) [7,  4000] loss: 0.693
(pid=1361) [3,  2000] loss: 1.389
(pid=1394) [3,  4000] loss: 0.745
(pid=1366) [4,  8000] loss: 0.287
Result for DEFAULT_36af6_00002:
  accuracy: 0.5328
  date: 2020-11-17_21-49-41
  done: false
  experiment_id: 73eebd293f93477d8354d22cef5833df
  experiment_tag: 2_batch_size=4,l1=8,l2=64,lr=0.0015565
  hostname: 8283a11465db
  iterations_since_restore: 4
  loss: 1.3144096929624676
  node_ip: 172.17.0.2
  pid: 1411
  should_checkpoint: true
  time_since_restore: 249.349196434021
  time_this_iter_s: 58.704336404800415
  time_total_s: 249.349196434021
  timestamp: 1605649781
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: 36af6_00002

== Status ==
Memory usage on this node: 6.6/240.1 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -1.6269151579856873 | Iter 4.000: -1.4843962660312653 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 10/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (5 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.31441 |     0.5328 |                    4 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.48208 |     0.4422 |                    2 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.21363 |     0.5654 |                    3 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.39318 |     0.492  |                    2 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.46796 |     0.4491 |                    6 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1361) [3,  4000] loss: 0.694
Result for DEFAULT_36af6_00009:
  accuracy: 0.4381
  date: 2020-11-17_21-49-45
  done: false
  experiment_id: c505a3e22c7745a98d825bf5a7d81b9d
  experiment_tag: 9_batch_size=8,l1=4,l2=256,lr=0.00033368
  hostname: 8283a11465db
  iterations_since_restore: 7
  loss: 1.499909722185135
  node_ip: 172.17.0.2
  pid: 1359
  should_checkpoint: true
  time_since_restore: 252.85753798484802
  time_this_iter_s: 33.528254985809326
  time_total_s: 252.85753798484802
  timestamp: 1605649785
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: 36af6_00009

(pid=1394) [3,  6000] loss: 0.489
(pid=1411) [5,  2000] loss: 1.286
(pid=1366) [4, 10000] loss: 0.228
(pid=1361) [3,  6000] loss: 0.462
(pid=1359) [8,  2000] loss: 1.366
(pid=1394) [3,  8000] loss: 0.367
Result for DEFAULT_36af6_00005:
  accuracy: 0.5744
  date: 2020-11-17_21-49-58
  done: false
  experiment_id: 90fceda1a64e4151a4e2229c7baf1924
  experiment_tag: 5_batch_size=4,l1=256,l2=128,lr=0.00094287
  hostname: 8283a11465db
  iterations_since_restore: 4
  loss: 1.21808747260347
  node_ip: 172.17.0.2
  pid: 1366
  should_checkpoint: true
  time_since_restore: 266.2706649303436
  time_this_iter_s: 61.87297868728638
  time_total_s: 266.2706649303436
  timestamp: 1605649798
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: 36af6_00005

== Status ==
Memory usage on this node: 6.6/240.1 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -1.6269151579856873 | Iter 4.000: -1.3994029794968665 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 10/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (5 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.31441 |     0.5328 |                    4 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.48208 |     0.4422 |                    2 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.21809 |     0.5744 |                    4 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.39318 |     0.492  |                    2 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.49991 |     0.4381 |                    7 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1411) [5,  4000] loss: 0.654
(pid=1361) [3,  8000] loss: 0.345
(pid=1394) [3, 10000] loss: 0.296
(pid=1359) [8,  4000] loss: 0.680
(pid=1366) [5,  2000] loss: 1.048
(pid=1411) [5,  6000] loss: 0.437
(pid=1361) [3, 10000] loss: 0.275
(pid=1394) [3, 12000] loss: 0.246
Result for DEFAULT_36af6_00009:
  accuracy: 0.4744
  date: 2020-11-17_21-50-17
  done: false
  experiment_id: c505a3e22c7745a98d825bf5a7d81b9d
  experiment_tag: 9_batch_size=8,l1=4,l2=256,lr=0.00033368
  hostname: 8283a11465db
  iterations_since_restore: 8
  loss: 1.3839171596288682
  node_ip: 172.17.0.2
  pid: 1359
  should_checkpoint: true
  time_since_restore: 285.6455948352814
  time_this_iter_s: 32.78805685043335
  time_total_s: 285.6455948352814
  timestamp: 1605649817
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: 36af6_00009

== Status ==
Memory usage on this node: 6.6/240.1 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -1.3839171596288682 | Iter 4.000: -1.3994029794968665 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 10/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (5 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.31441 |     0.5328 |                    4 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.48208 |     0.4422 |                    2 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.21809 |     0.5744 |                    4 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.39318 |     0.492  |                    2 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.38392 |     0.4744 |                    8 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1366) [5,  4000] loss: 0.528
(pid=1411) [5,  8000] loss: 0.327
(pid=1361) [3, 12000] loss: 0.221
(pid=1394) [3, 14000] loss: 0.210
(pid=1359) [9,  2000] loss: 1.331
(pid=1366) [5,  6000] loss: 0.352
(pid=1361) [3, 14000] loss: 0.189
(pid=1411) [5, 10000] loss: 0.260
(pid=1394) [3, 16000] loss: 0.181
Result for DEFAULT_36af6_00002:
  accuracy: 0.5381
  date: 2020-11-17_21-50-37
  done: false
  experiment_id: 73eebd293f93477d8354d22cef5833df
  experiment_tag: 2_batch_size=4,l1=8,l2=64,lr=0.0015565
  hostname: 8283a11465db
  iterations_since_restore: 5
  loss: 1.2883288912653923
  node_ip: 172.17.0.2
  pid: 1411
  should_checkpoint: true
  time_since_restore: 305.35691928863525
  time_this_iter_s: 56.00772285461426
  time_total_s: 305.35691928863525
  timestamp: 1605649837
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: 36af6_00002

== Status ==
Memory usage on this node: 6.6/240.1 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -1.3839171596288682 | Iter 4.000: -1.3994029794968665 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 10/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (5 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.28833 |     0.5381 |                    5 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.48208 |     0.4422 |                    2 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.21809 |     0.5744 |                    4 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.39318 |     0.492  |                    2 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.38392 |     0.4744 |                    8 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1361) [3, 16000] loss: 0.171
(pid=1366) [5,  8000] loss: 0.267
(pid=1359) [9,  4000] loss: 0.672
(pid=1394) [3, 18000] loss: 0.161
(pid=1411) [6,  2000] loss: 1.269
(pid=1361) [3, 18000] loss: 0.148
Result for DEFAULT_36af6_00009:
  accuracy: 0.4751
  date: 2020-11-17_21-50-50
  done: false
  experiment_id: c505a3e22c7745a98d825bf5a7d81b9d
  experiment_tag: 9_batch_size=8,l1=4,l2=256,lr=0.00033368
  hostname: 8283a11465db
  iterations_since_restore: 9
  loss: 1.393266782283783
  node_ip: 172.17.0.2
  pid: 1359
  should_checkpoint: true
  time_since_restore: 318.3814539909363
  time_this_iter_s: 32.73585915565491
  time_total_s: 318.3814539909363
  timestamp: 1605649850
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: 36af6_00009

== Status ==
Memory usage on this node: 6.6/240.1 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -1.3839171596288682 | Iter 4.000: -1.3994029794968665 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 10/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (5 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.28833 |     0.5381 |                    5 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.48208 |     0.4422 |                    2 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.21809 |     0.5744 |                    4 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.39318 |     0.492  |                    2 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.39327 |     0.4751 |                    9 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1366) [5, 10000] loss: 0.210
(pid=1394) [3, 20000] loss: 0.147
(pid=1411) [6,  4000] loss: 0.652
(pid=1361) [3, 20000] loss: 0.131
Result for DEFAULT_36af6_00005:
  accuracy: 0.597
  date: 2020-11-17_21-50-57
  done: false
  experiment_id: 90fceda1a64e4151a4e2229c7baf1924
  experiment_tag: 5_batch_size=4,l1=256,l2=128,lr=0.00094287
  hostname: 8283a11465db
  iterations_since_restore: 5
  loss: 1.1688944209709764
  node_ip: 172.17.0.2
  pid: 1366
  should_checkpoint: true
  time_since_restore: 325.63561272621155
  time_this_iter_s: 59.36494779586792
  time_total_s: 325.63561272621155
  timestamp: 1605649857
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: 36af6_00005

== Status ==
Memory usage on this node: 6.5/240.1 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -1.3839171596288682 | Iter 4.000: -1.3994029794968665 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 10/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (5 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.28833 |     0.5381 |                    5 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.48208 |     0.4422 |                    2 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.16889 |     0.597  |                    5 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.39318 |     0.492  |                    2 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.39327 |     0.4751 |                    9 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1359) [10,  2000] loss: 1.323
Result for DEFAULT_36af6_00003:
  accuracy: 0.4672
  date: 2020-11-17_21-51-02
  done: false
  experiment_id: e59bc8a0908b442eb6d30667cb56539b
  experiment_tag: 3_batch_size=2,l1=4,l2=4,lr=0.00065304
  hostname: 8283a11465db
  iterations_since_restore: 3
  loss: 1.459887399574369
  node_ip: 172.17.0.2
  pid: 1394
  should_checkpoint: true
  time_since_restore: 330.83889079093933
  time_this_iter_s: 100.66886258125305
  time_total_s: 330.83889079093933
  timestamp: 1605649862
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: 36af6_00003

== Status ==
Memory usage on this node: 6.6/240.1 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -1.3839171596288682 | Iter 4.000: -1.3994029794968665 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 10/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (5 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.28833 |     0.5381 |                    5 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.45989 |     0.4672 |                    3 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.16889 |     0.597  |                    5 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.39318 |     0.492  |                    2 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.39327 |     0.4751 |                    9 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1411) [6,  6000] loss: 0.430
(pid=1366) [6,  2000] loss: 0.955
Result for DEFAULT_36af6_00007:
  accuracy: 0.5104
  date: 2020-11-17_21-51-09
  done: false
  experiment_id: 982797bde02f40e4a9744d76174e8913
  experiment_tag: 7_batch_size=2,l1=32,l2=32,lr=0.00024587
  hostname: 8283a11465db
  iterations_since_restore: 3
  loss: 1.334091033653915
  node_ip: 172.17.0.2
  pid: 1361
  should_checkpoint: true
  time_since_restore: 337.56662940979004
  time_this_iter_s: 103.38929724693298
  time_total_s: 337.56662940979004
  timestamp: 1605649869
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: 36af6_00007

== Status ==
Memory usage on this node: 6.6/240.1 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -1.3839171596288682 | Iter 4.000: -1.3994029794968665 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 10/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (5 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.28833 |     0.5381 |                    5 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.45989 |     0.4672 |                    3 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.16889 |     0.597  |                    5 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.33409 |     0.5104 |                    3 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.39327 |     0.4751 |                    9 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1394) [4,  2000] loss: 1.440
(pid=1359) [10,  4000] loss: 0.662
(pid=1411) [6,  8000] loss: 0.323
(pid=1361) [4,  2000] loss: 1.270
(pid=1366) [6,  4000] loss: 0.492
(pid=1394) [4,  4000] loss: 0.717
Result for DEFAULT_36af6_00009:
  accuracy: 0.4914
  date: 2020-11-17_21-51-23
  done: true
  experiment_id: c505a3e22c7745a98d825bf5a7d81b9d
  experiment_tag: 9_batch_size=8,l1=4,l2=256,lr=0.00033368
  hostname: 8283a11465db
  iterations_since_restore: 10
  loss: 1.3439717058897018
  node_ip: 172.17.0.2
  pid: 1359
  should_checkpoint: true
  time_since_restore: 351.16450452804565
  time_this_iter_s: 32.783050537109375
  time_total_s: 351.16450452804565
  timestamp: 1605649883
  timesteps_since_restore: 0
  training_iteration: 10
  trial_id: 36af6_00009

== Status ==
Memory usage on this node: 6.6/240.1 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.3839171596288682 | Iter 4.000: -1.3994029794968665 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 10/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (5 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.28833 |     0.5381 |                    5 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.45989 |     0.4672 |                    3 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.16889 |     0.597  |                    5 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.33409 |     0.5104 |                    3 |
| DEFAULT_36af6_00009 | RUNNING    | 172.17.0.2:1359 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1411) [6, 10000] loss: 0.256
(pid=1361) [4,  4000] loss: 0.639
(pid=1394) [4,  6000] loss: 0.472
(pid=1366) [6,  6000] loss: 0.328
Result for DEFAULT_36af6_00002:
  accuracy: 0.5406
  date: 2020-11-17_21-51-32
  done: false
  experiment_id: 73eebd293f93477d8354d22cef5833df
  experiment_tag: 2_batch_size=4,l1=8,l2=64,lr=0.0015565
  hostname: 8283a11465db
  iterations_since_restore: 6
  loss: 1.2868849665284157
  node_ip: 172.17.0.2
  pid: 1411
  should_checkpoint: true
  time_since_restore: 360.6178274154663
  time_this_iter_s: 55.260908126831055
  time_total_s: 360.6178274154663
  timestamp: 1605649892
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: 36af6_00002

== Status ==
Memory usage on this node: 5.9/240.1 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.3839171596288682 | Iter 4.000: -1.3994029794968665 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 8/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.28688 |     0.5406 |                    6 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.45989 |     0.4672 |                    3 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.16889 |     0.597  |                    5 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.33409 |     0.5104 |                    3 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |                 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1361) [4,  6000] loss: 0.423
(pid=1394) [4,  8000] loss: 0.357
(pid=1366) [6,  8000] loss: 0.251
(pid=1411) [7,  2000] loss: 1.266
(pid=1361) [4,  8000] loss: 0.321
(pid=1394) [4, 10000] loss: 0.289
(pid=1366) [6, 10000] loss: 0.199
(pid=1411) [7,  4000] loss: 0.627
(pid=1361) [4, 10000] loss: 0.258
(pid=1394) [4, 12000] loss: 0.236
Result for DEFAULT_36af6_00005:
  accuracy: 0.5829
  date: 2020-11-17_21-51-56
  done: false
  experiment_id: 90fceda1a64e4151a4e2229c7baf1924
  experiment_tag: 5_batch_size=4,l1=256,l2=128,lr=0.00094287
  hostname: 8283a11465db
  iterations_since_restore: 6
  loss: 1.1916377913918346
  node_ip: 172.17.0.2
  pid: 1366
  should_checkpoint: true
  time_since_restore: 384.30396127700806
  time_this_iter_s: 58.66834855079651
  time_total_s: 384.30396127700806
  timestamp: 1605649916
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: 36af6_00005

== Status ==
Memory usage on this node: 5.9/240.1 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.3839171596288682 | Iter 4.000: -1.3994029794968665 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 8/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.28688 |     0.5406 |                    6 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.45989 |     0.4672 |                    3 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.19164 |     0.5829 |                    6 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.33409 |     0.5104 |                    3 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |                 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1411) [7,  6000] loss: 0.424
(pid=1361) [4, 12000] loss: 0.218
(pid=1394) [4, 14000] loss: 0.206
(pid=1366) [7,  2000] loss: 0.897
(pid=1411) [7,  8000] loss: 0.325
(pid=1361) [4, 14000] loss: 0.177
(pid=1394) [4, 16000] loss: 0.177
(pid=1366) [7,  4000] loss: 0.461
(pid=1411) [7, 10000] loss: 0.253
(pid=1361) [4, 16000] loss: 0.152
(pid=1394) [4, 18000] loss: 0.155
Result for DEFAULT_36af6_00002:
  accuracy: 0.5209
  date: 2020-11-17_21-52-26
  done: false
  experiment_id: 73eebd293f93477d8354d22cef5833df
  experiment_tag: 2_batch_size=4,l1=8,l2=64,lr=0.0015565
  hostname: 8283a11465db
  iterations_since_restore: 7
  loss: 1.3202102733552457
  node_ip: 172.17.0.2
  pid: 1411
  should_checkpoint: true
  time_since_restore: 414.69909405708313
  time_this_iter_s: 54.08126664161682
  time_total_s: 414.69909405708313
  timestamp: 1605649946
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: 36af6_00002

== Status ==
Memory usage on this node: 5.9/240.1 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.3839171596288682 | Iter 4.000: -1.3994029794968665 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 8/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.32021 |     0.5209 |                    7 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.45989 |     0.4672 |                    3 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.19164 |     0.5829 |                    6 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.33409 |     0.5104 |                    3 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |                 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1366) [7,  6000] loss: 0.297
(pid=1361) [4, 18000] loss: 0.138
(pid=1394) [4, 20000] loss: 0.138
(pid=1411) [8,  2000] loss: 1.243
(pid=1366) [7,  8000] loss: 0.231
(pid=1361) [4, 20000] loss: 0.121
Result for DEFAULT_36af6_00003:
  accuracy: 0.4859
  date: 2020-11-17_21-52-41
  done: false
  experiment_id: e59bc8a0908b442eb6d30667cb56539b
  experiment_tag: 3_batch_size=2,l1=4,l2=4,lr=0.00065304
  hostname: 8283a11465db
  iterations_since_restore: 4
  loss: 1.3983723227754234
  node_ip: 172.17.0.2
  pid: 1394
  should_checkpoint: true
  time_since_restore: 429.7377655506134
  time_this_iter_s: 98.89887475967407
  time_total_s: 429.7377655506134
  timestamp: 1605649961
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: 36af6_00003

== Status ==
Memory usage on this node: 5.9/240.1 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.3839171596288682 | Iter 4.000: -1.3983723227754234 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 8/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.32021 |     0.5209 |                    7 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.39837 |     0.4859 |                    4 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.19164 |     0.5829 |                    6 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.33409 |     0.5104 |                    3 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |                 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1411) [8,  4000] loss: 0.626
(pid=1366) [7, 10000] loss: 0.186
Result for DEFAULT_36af6_00007:
  accuracy: 0.5468
  date: 2020-11-17_21-52-50
  done: false
  experiment_id: 982797bde02f40e4a9744d76174e8913
  experiment_tag: 7_batch_size=2,l1=32,l2=32,lr=0.00024587
  hostname: 8283a11465db
  iterations_since_restore: 4
  loss: 1.2675049975303
  node_ip: 172.17.0.2
  pid: 1361
  should_checkpoint: true
  time_since_restore: 438.3499252796173
  time_this_iter_s: 100.78329586982727
  time_total_s: 438.3499252796173
  timestamp: 1605649970
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: 36af6_00007

== Status ==
Memory usage on this node: 5.9/240.1 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.3839171596288682 | Iter 4.000: -1.3563910078689454 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 8/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.32021 |     0.5209 |                    7 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.39837 |     0.4859 |                    4 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.19164 |     0.5829 |                    6 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.2675  |     0.5468 |                    4 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |                 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1394) [5,  2000] loss: 1.394
Result for DEFAULT_36af6_00005:
  accuracy: 0.6013
  date: 2020-11-17_21-52-55
  done: false
  experiment_id: 90fceda1a64e4151a4e2229c7baf1924
  experiment_tag: 5_batch_size=4,l1=256,l2=128,lr=0.00094287
  hostname: 8283a11465db
  iterations_since_restore: 7
  loss: 1.183037901109457
  node_ip: 172.17.0.2
  pid: 1366
  should_checkpoint: true
  time_since_restore: 443.12420439720154
  time_this_iter_s: 58.82024312019348
  time_total_s: 443.12420439720154
  timestamp: 1605649975
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: 36af6_00005

(pid=1411) [8,  6000] loss: 0.416
(pid=1394) [5,  4000] loss: 0.694
(pid=1361) [5,  2000] loss: 1.219
(pid=1411) [8,  8000] loss: 0.318
(pid=1366) [8,  2000] loss: 0.811
(pid=1394) [5,  6000] loss: 0.467
(pid=1361) [5,  4000] loss: 0.599
(pid=1411) [8, 10000] loss: 0.252
(pid=1366) [8,  4000] loss: 0.426
(pid=1394) [5,  8000] loss: 0.347
(pid=1361) [5,  6000] loss: 0.408
Result for DEFAULT_36af6_00002:
  accuracy: 0.5487
  date: 2020-11-17_21-53-21
  done: false
  experiment_id: 73eebd293f93477d8354d22cef5833df
  experiment_tag: 2_batch_size=4,l1=8,l2=64,lr=0.0015565
  hostname: 8283a11465db
  iterations_since_restore: 8
  loss: 1.2750557748377322
  node_ip: 172.17.0.2
  pid: 1411
  should_checkpoint: true
  time_since_restore: 469.0136640071869
  time_this_iter_s: 54.31456995010376
  time_total_s: 469.0136640071869
  timestamp: 1605650001
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: 36af6_00002

== Status ==
Memory usage on this node: 5.9/240.1 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.3294864672333002 | Iter 4.000: -1.3563910078689454 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 8/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.27506 |     0.5487 |                    8 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.39837 |     0.4859 |                    4 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.18304 |     0.6013 |                    7 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.2675  |     0.5468 |                    4 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |                 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1394) [5, 10000] loss: 0.277
(pid=1366) [8,  6000] loss: 0.282
(pid=1361) [5,  8000] loss: 0.293
(pid=1411) [9,  2000] loss: 1.236
(pid=1394) [5, 12000] loss: 0.238
(pid=1361) [5, 10000] loss: 0.236
(pid=1366) [8,  8000] loss: 0.224
(pid=1411) [9,  4000] loss: 0.619
(pid=1394) [5, 14000] loss: 0.201
(pid=1361) [5, 12000] loss: 0.196
(pid=1366) [8, 10000] loss: 0.176
(pid=1411) [9,  6000] loss: 0.407
(pid=1394) [5, 16000] loss: 0.177
(pid=1361) [5, 14000] loss: 0.168
Result for DEFAULT_36af6_00005:
  accuracy: 0.6009
  date: 2020-11-17_21-53-53
  done: false
  experiment_id: 90fceda1a64e4151a4e2229c7baf1924
  experiment_tag: 5_batch_size=4,l1=256,l2=128,lr=0.00094287
  hostname: 8283a11465db
  iterations_since_restore: 8
  loss: 1.2001436830021441
  node_ip: 172.17.0.2
  pid: 1366
  should_checkpoint: true
  time_since_restore: 500.9162974357605
  time_this_iter_s: 57.79209303855896
  time_total_s: 500.9162974357605
  timestamp: 1605650033
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: 36af6_00005

== Status ==
Memory usage on this node: 5.9/240.1 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.2750557748377322 | Iter 4.000: -1.3563910078689454 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 8/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.27506 |     0.5487 |                    8 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.39837 |     0.4859 |                    4 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.20014 |     0.6009 |                    8 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.2675  |     0.5468 |                    4 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |                 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1411) [9,  8000] loss: 0.312
(pid=1394) [5, 18000] loss: 0.156
(pid=1361) [5, 16000] loss: 0.148
(pid=1366) [9,  2000] loss: 0.730
(pid=1394) [5, 20000] loss: 0.142
(pid=1411) [9, 10000] loss: 0.255
(pid=1361) [5, 18000] loss: 0.130
(pid=1366) [9,  4000] loss: 0.385
Result for DEFAULT_36af6_00002:
  accuracy: 0.5034
  date: 2020-11-17_21-54-14
  done: false
  experiment_id: 73eebd293f93477d8354d22cef5833df
  experiment_tag: 2_batch_size=4,l1=8,l2=64,lr=0.0015565
  hostname: 8283a11465db
  iterations_since_restore: 9
  loss: 1.431942359483242
  node_ip: 172.17.0.2
  pid: 1411
  should_checkpoint: true
  time_since_restore: 522.7910490036011
  time_this_iter_s: 53.777384996414185
  time_total_s: 522.7910490036011
  timestamp: 1605650054
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: 36af6_00002

== Status ==
Memory usage on this node: 5.9/240.1 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.2750557748377322 | Iter 4.000: -1.3563910078689454 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 8/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.43194 |     0.5034 |                    9 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.39837 |     0.4859 |                    4 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.20014 |     0.6009 |                    8 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.2675  |     0.5468 |                    4 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |                 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_36af6_00003:
  accuracy: 0.4669
  date: 2020-11-17_21-54-19
  done: false
  experiment_id: e59bc8a0908b442eb6d30667cb56539b
  experiment_tag: 3_batch_size=2,l1=4,l2=4,lr=0.00065304
  hostname: 8283a11465db
  iterations_since_restore: 5
  loss: 1.4101506413787603
  node_ip: 172.17.0.2
  pid: 1394
  should_checkpoint: true
  time_since_restore: 527.588033914566
  time_this_iter_s: 97.85026836395264
  time_total_s: 527.588033914566
  timestamp: 1605650059
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: 36af6_00003

(pid=1361) [5, 20000] loss: 0.121
(pid=1366) [9,  6000] loss: 0.264
(pid=1411) [10,  2000] loss: 1.242
(pid=1394) [6,  2000] loss: 1.355
Result for DEFAULT_36af6_00007:
  accuracy: 0.5767
  date: 2020-11-17_21-54-31
  done: false
  experiment_id: 982797bde02f40e4a9744d76174e8913
  experiment_tag: 7_batch_size=2,l1=32,l2=32,lr=0.00024587
  hostname: 8283a11465db
  iterations_since_restore: 5
  loss: 1.1906794582221192
  node_ip: 172.17.0.2
  pid: 1361
  should_checkpoint: true
  time_since_restore: 539.5284807682037
  time_this_iter_s: 101.17855548858643
  time_total_s: 539.5284807682037
  timestamp: 1605650071
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: 36af6_00007

== Status ==
Memory usage on this node: 6.0/240.1 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.2750557748377322 | Iter 4.000: -1.3563910078689454 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 8/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.43194 |     0.5034 |                    9 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.41015 |     0.4669 |                    5 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.20014 |     0.6009 |                    8 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.19068 |     0.5767 |                    5 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |                 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1411) [10,  4000] loss: 0.613
(pid=1366) [9,  8000] loss: 0.209
(pid=1394) [6,  4000] loss: 0.685
(pid=1361) [6,  2000] loss: 1.115
(pid=1411) [10,  6000] loss: 0.410
(pid=1366) [9, 10000] loss: 0.165
(pid=1394) [6,  6000] loss: 0.463
(pid=1361) [6,  4000] loss: 0.566
Result for DEFAULT_36af6_00005:
  accuracy: 0.6094
  date: 2020-11-17_21-54-51
  done: false
  experiment_id: 90fceda1a64e4151a4e2229c7baf1924
  experiment_tag: 5_batch_size=4,l1=256,l2=128,lr=0.00094287
  hostname: 8283a11465db
  iterations_since_restore: 9
  loss: 1.2033040826294572
  node_ip: 172.17.0.2
  pid: 1366
  should_checkpoint: true
  time_since_restore: 558.9555423259735
  time_this_iter_s: 58.03924489021301
  time_total_s: 558.9555423259735
  timestamp: 1605650091
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: 36af6_00005

== Status ==
Memory usage on this node: 5.9/240.1 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.2750557748377322 | Iter 4.000: -1.3563910078689454 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 8/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.43194 |     0.5034 |                    9 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.41015 |     0.4669 |                    5 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.2033  |     0.6094 |                    9 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.19068 |     0.5767 |                    5 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |                 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1411) [10,  8000] loss: 0.310
(pid=1394) [6,  8000] loss: 0.337
(pid=1361) [6,  6000] loss: 0.385
(pid=1366) [10,  2000] loss: 0.684
(pid=1411) [10, 10000] loss: 0.248
(pid=1394) [6, 10000] loss: 0.279
(pid=1361) [6,  8000] loss: 0.291
Result for DEFAULT_36af6_00002:
  accuracy: 0.5532
  date: 2020-11-17_21-55-09
  done: true
  experiment_id: 73eebd293f93477d8354d22cef5833df
  experiment_tag: 2_batch_size=4,l1=8,l2=64,lr=0.0015565
  hostname: 8283a11465db
  iterations_since_restore: 10
  loss: 1.2825815960392355
  node_ip: 172.17.0.2
  pid: 1411
  should_checkpoint: true
  time_since_restore: 577.065728187561
  time_this_iter_s: 54.27467918395996
  time_total_s: 577.065728187561
  timestamp: 1605650109
  timesteps_since_restore: 0
  training_iteration: 10
  trial_id: 36af6_00002

== Status ==
Memory usage on this node: 6.0/240.1 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.2750557748377322 | Iter 4.000: -1.3563910078689454 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 8/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00002 | RUNNING    | 172.17.0.2:1411 |            4 |    8 |   64 | 0.00155654  | 1.28258 |     0.5532 |                   10 |
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.41015 |     0.4669 |                    5 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.2033  |     0.6094 |                    9 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.19068 |     0.5767 |                    5 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |                 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1394) [6, 12000] loss: 0.233
(pid=1366) [10,  4000] loss: 0.356
(pid=1361) [6, 10000] loss: 0.234
(pid=1394) [6, 14000] loss: 0.198
(pid=1366) [10,  6000] loss: 0.252
(pid=1361) [6, 12000] loss: 0.187
(pid=1394) [6, 16000] loss: 0.174
(pid=1366) [10,  8000] loss: 0.192
(pid=1361) [6, 14000] loss: 0.162
(pid=1394) [6, 18000] loss: 0.154
(pid=1366) [10, 10000] loss: 0.156
(pid=1361) [6, 16000] loss: 0.143
(pid=1394) [6, 20000] loss: 0.138
Result for DEFAULT_36af6_00005:
  accuracy: 0.6066
  date: 2020-11-17_21-55-48
  done: true
  experiment_id: 90fceda1a64e4151a4e2229c7baf1924
  experiment_tag: 5_batch_size=4,l1=256,l2=128,lr=0.00094287
  hostname: 8283a11465db
  iterations_since_restore: 10
  loss: 1.2412150030929596
  node_ip: 172.17.0.2
  pid: 1366
  should_checkpoint: true
  time_since_restore: 615.896600484848
  time_this_iter_s: 56.94105815887451
  time_total_s: 615.896600484848
  timestamp: 1605650148
  timesteps_since_restore: 0
  training_iteration: 10
  trial_id: 36af6_00005

== Status ==
Memory usage on this node: 5.3/240.1 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.2750557748377322 | Iter 4.000: -1.3563910078689454 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 6/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.41015 |     0.4669 |                    5 |
| DEFAULT_36af6_00005 | RUNNING    | 172.17.0.2:1366 |            4 |  256 |  128 | 0.000942874 | 1.24122 |     0.6066 |                   10 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.19068 |     0.5767 |                    5 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00002 | TERMINATED |                 |            4 |    8 |   64 | 0.00155654  | 1.28258 |     0.5532 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |                 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1361) [6, 18000] loss: 0.127
Result for DEFAULT_36af6_00003:
  accuracy: 0.494
  date: 2020-11-17_21-55-56
  done: false
  experiment_id: e59bc8a0908b442eb6d30667cb56539b
  experiment_tag: 3_batch_size=2,l1=4,l2=4,lr=0.00065304
  hostname: 8283a11465db
  iterations_since_restore: 6
  loss: 1.3862219633713364
  node_ip: 172.17.0.2
  pid: 1394
  should_checkpoint: true
  time_since_restore: 624.1250314712524
  time_this_iter_s: 96.5369975566864
  time_total_s: 624.1250314712524
  timestamp: 1605650156
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: 36af6_00003

== Status ==
Memory usage on this node: 4.7/240.1 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.2750557748377322 | Iter 4.000: -1.3563910078689454 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 4/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.38622 |     0.494  |                    6 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.19068 |     0.5767 |                    5 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00002 | TERMINATED |                 |            4 |    8 |   64 | 0.00155654  | 1.28258 |     0.5532 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00005 | TERMINATED |                 |            4 |  256 |  128 | 0.000942874 | 1.24122 |     0.6066 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |                 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1361) [6, 20000] loss: 0.114
(pid=1394) [7,  2000] loss: 1.351
Result for DEFAULT_36af6_00007:
  accuracy: 0.5815
  date: 2020-11-17_21-56-10
  done: false
  experiment_id: 982797bde02f40e4a9744d76174e8913
  experiment_tag: 7_batch_size=2,l1=32,l2=32,lr=0.00024587
  hostname: 8283a11465db
  iterations_since_restore: 6
  loss: 1.1713567746165208
  node_ip: 172.17.0.2
  pid: 1361
  should_checkpoint: true
  time_since_restore: 638.4690382480621
  time_this_iter_s: 98.9405574798584
  time_total_s: 638.4690382480621
  timestamp: 1605650170
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: 36af6_00007

== Status ==
Memory usage on this node: 4.7/240.1 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.2750557748377322 | Iter 4.000: -1.3563910078689454 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 4/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.38622 |     0.494  |                    6 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.17136 |     0.5815 |                    6 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00002 | TERMINATED |                 |            4 |    8 |   64 | 0.00155654  | 1.28258 |     0.5532 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00005 | TERMINATED |                 |            4 |  256 |  128 | 0.000942874 | 1.24122 |     0.6066 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |                 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1394) [7,  4000] loss: 0.678
(pid=1361) [7,  2000] loss: 1.089
(pid=1394) [7,  6000] loss: 0.459
(pid=1361) [7,  4000] loss: 0.550
(pid=1394) [7,  8000] loss: 0.336
(pid=1361) [7,  6000] loss: 0.363
(pid=1394) [7, 10000] loss: 0.279
(pid=1361) [7,  8000] loss: 0.275
(pid=1394) [7, 12000] loss: 0.220
(pid=1361) [7, 10000] loss: 0.223
(pid=1394) [7, 14000] loss: 0.195
(pid=1361) [7, 12000] loss: 0.188
(pid=1394) [7, 16000] loss: 0.173
(pid=1394) [7, 18000] loss: 0.154
(pid=1361) [7, 14000] loss: 0.155
(pid=1394) [7, 20000] loss: 0.136
(pid=1361) [7, 16000] loss: 0.138
(pid=1361) [7, 18000] loss: 0.126
Result for DEFAULT_36af6_00003:
  accuracy: 0.5077
  date: 2020-11-17_21-57-29
  done: false
  experiment_id: e59bc8a0908b442eb6d30667cb56539b
  experiment_tag: 3_batch_size=2,l1=4,l2=4,lr=0.00065304
  hostname: 8283a11465db
  iterations_since_restore: 7
  loss: 1.3653217664174735
  node_ip: 172.17.0.2
  pid: 1394
  should_checkpoint: true
  time_since_restore: 717.803804397583
  time_this_iter_s: 93.67877292633057
  time_total_s: 717.803804397583
  timestamp: 1605650249
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: 36af6_00003

== Status ==
Memory usage on this node: 4.7/240.1 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.2750557748377322 | Iter 4.000: -1.3563910078689454 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 4/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.36532 |     0.5077 |                    7 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.17136 |     0.5815 |                    6 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00002 | TERMINATED |                 |            4 |    8 |   64 | 0.00155654  | 1.28258 |     0.5532 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00005 | TERMINATED |                 |            4 |  256 |  128 | 0.000942874 | 1.24122 |     0.6066 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |                 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1361) [7, 20000] loss: 0.109
(pid=1394) [8,  2000] loss: 1.337
(pid=1394) [8,  4000] loss: 0.668
Result for DEFAULT_36af6_00007:
  accuracy: 0.5892
  date: 2020-11-17_21-57-47
  done: false
  experiment_id: 982797bde02f40e4a9744d76174e8913
  experiment_tag: 7_batch_size=2,l1=32,l2=32,lr=0.00024587
  hostname: 8283a11465db
  iterations_since_restore: 7
  loss: 1.1568100737834814
  node_ip: 172.17.0.2
  pid: 1361
  should_checkpoint: true
  time_since_restore: 735.3395071029663
  time_this_iter_s: 96.87046885490417
  time_total_s: 735.3395071029663
  timestamp: 1605650267
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: 36af6_00007

== Status ==
Memory usage on this node: 4.7/240.1 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.2750557748377322 | Iter 4.000: -1.3563910078689454 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 4/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.36532 |     0.5077 |                    7 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.15681 |     0.5892 |                    7 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00002 | TERMINATED |                 |            4 |    8 |   64 | 0.00155654  | 1.28258 |     0.5532 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00005 | TERMINATED |                 |            4 |  256 |  128 | 0.000942874 | 1.24122 |     0.6066 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |                 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1394) [8,  6000] loss: 0.451
(pid=1361) [8,  2000] loss: 1.051
(pid=1394) [8,  8000] loss: 0.339
(pid=1361) [8,  4000] loss: 0.515
(pid=1394) [8, 10000] loss: 0.271
(pid=1361) [8,  6000] loss: 0.358
(pid=1394) [8, 12000] loss: 0.228
(pid=1361) [8,  8000] loss: 0.263
(pid=1394) [8, 14000] loss: 0.190
(pid=1361) [8, 10000] loss: 0.215
(pid=1394) [8, 16000] loss: 0.169
(pid=1361) [8, 12000] loss: 0.183
(pid=1394) [8, 18000] loss: 0.151
(pid=1361) [8, 14000] loss: 0.156
(pid=1394) [8, 20000] loss: 0.138
(pid=1361) [8, 16000] loss: 0.135
(pid=1361) [8, 18000] loss: 0.118
Result for DEFAULT_36af6_00003:
  accuracy: 0.5074
  date: 2020-11-17_21-59-03
  done: true
  experiment_id: e59bc8a0908b442eb6d30667cb56539b
  experiment_tag: 3_batch_size=2,l1=4,l2=4,lr=0.00065304
  hostname: 8283a11465db
  iterations_since_restore: 8
  loss: 1.3631895693857223
  node_ip: 172.17.0.2
  pid: 1394
  should_checkpoint: true
  time_since_restore: 811.710126876831
  time_this_iter_s: 93.90632247924805
  time_total_s: 811.710126876831
  timestamp: 1605650343
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: 36af6_00003

== Status ==
Memory usage on this node: 4.7/240.1 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.3191226721117273 | Iter 4.000: -1.3563910078689454 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 4/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00003 | RUNNING    | 172.17.0.2:1394 |            2 |    4 |    4 | 0.000653041 | 1.36319 |     0.5074 |                    8 |
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.15681 |     0.5892 |                    7 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00002 | TERMINATED |                 |            4 |    8 |   64 | 0.00155654  | 1.28258 |     0.5532 |                   10 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00005 | TERMINATED |                 |            4 |  256 |  128 | 0.000942874 | 1.24122 |     0.6066 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |                 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1361) [8, 20000] loss: 0.109
Result for DEFAULT_36af6_00007:
  accuracy: 0.6019
  date: 2020-11-17_21-59-23
  done: false
  experiment_id: 982797bde02f40e4a9744d76174e8913
  experiment_tag: 7_batch_size=2,l1=32,l2=32,lr=0.00024587
  hostname: 8283a11465db
  iterations_since_restore: 8
  loss: 1.1148782885240391
  node_ip: 172.17.0.2
  pid: 1361
  should_checkpoint: true
  time_since_restore: 831.1660561561584
  time_this_iter_s: 95.82654905319214
  time_total_s: 831.1660561561584
  timestamp: 1605650363
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: 36af6_00007

== Status ==
Memory usage on this node: 4.1/240.1 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2750557748377322 | Iter 4.000: -1.3563910078689454 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 2/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.11488 |     0.6019 |                    8 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00002 | TERMINATED |                 |            4 |    8 |   64 | 0.00155654  | 1.28258 |     0.5532 |                   10 |
| DEFAULT_36af6_00003 | TERMINATED |                 |            2 |    4 |    4 | 0.000653041 | 1.36319 |     0.5074 |                    8 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00005 | TERMINATED |                 |            4 |  256 |  128 | 0.000942874 | 1.24122 |     0.6066 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |                 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1361) [9,  2000] loss: 0.997
(pid=1361) [9,  4000] loss: 0.524
(pid=1361) [9,  6000] loss: 0.339
(pid=1361) [9,  8000] loss: 0.263
(pid=1361) [9, 10000] loss: 0.211
(pid=1361) [9, 12000] loss: 0.175
(pid=1361) [9, 14000] loss: 0.148
(pid=1361) [9, 16000] loss: 0.131
(pid=1361) [9, 18000] loss: 0.118
(pid=1361) [9, 20000] loss: 0.106
Result for DEFAULT_36af6_00007:
  accuracy: 0.5913
  date: 2020-11-17_22-00-58
  done: false
  experiment_id: 982797bde02f40e4a9744d76174e8913
  experiment_tag: 7_batch_size=2,l1=32,l2=32,lr=0.00024587
  hostname: 8283a11465db
  iterations_since_restore: 9
  loss: 1.1877186533010244
  node_ip: 172.17.0.2
  pid: 1361
  should_checkpoint: true
  time_since_restore: 925.9873507022858
  time_this_iter_s: 94.82129454612732
  time_total_s: 925.9873507022858
  timestamp: 1605650458
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: 36af6_00007

== Status ==
Memory usage on this node: 4.1/240.1 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2750557748377322 | Iter 4.000: -1.3563910078689454 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 2/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.18772 |     0.5913 |                    9 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00002 | TERMINATED |                 |            4 |    8 |   64 | 0.00155654  | 1.28258 |     0.5532 |                   10 |
| DEFAULT_36af6_00003 | TERMINATED |                 |            2 |    4 |    4 | 0.000653041 | 1.36319 |     0.5074 |                    8 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00005 | TERMINATED |                 |            4 |  256 |  128 | 0.000942874 | 1.24122 |     0.6066 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |                 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=1361) [10,  2000] loss: 0.993
(pid=1361) [10,  4000] loss: 0.501
(pid=1361) [10,  6000] loss: 0.333
(pid=1361) [10,  8000] loss: 0.258
(pid=1361) [10, 10000] loss: 0.205
(pid=1361) [10, 12000] loss: 0.171
(pid=1361) [10, 14000] loss: 0.144
(pid=1361) [10, 16000] loss: 0.130
(pid=1361) [10, 18000] loss: 0.115
(pid=1361) [10, 20000] loss: 0.103
Result for DEFAULT_36af6_00007:
  accuracy: 0.6025
  date: 2020-11-17_22-02-33
  done: true
  experiment_id: 982797bde02f40e4a9744d76174e8913
  experiment_tag: 7_batch_size=2,l1=32,l2=32,lr=0.00024587
  hostname: 8283a11465db
  iterations_since_restore: 10
  loss: 1.121114240665501
  node_ip: 172.17.0.2
  pid: 1361
  should_checkpoint: true
  time_since_restore: 1021.3109564781189
  time_this_iter_s: 95.32360577583313
  time_total_s: 1021.3109564781189
  timestamp: 1605650553
  timesteps_since_restore: 0
  training_iteration: 10
  trial_id: 36af6_00007

== Status ==
Memory usage on this node: 4.1/240.1 GiB
Using AsyncHyperBand: num_stopped=10
Bracket: Iter 8.000: -1.2750557748377322 | Iter 4.000: -1.3563910078689454 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 2/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00007 | RUNNING    | 172.17.0.2:1361 |            2 |   32 |   32 | 0.000245868 | 1.12111 |     0.6025 |                   10 |
| DEFAULT_36af6_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |                 |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00002 | TERMINATED |                 |            4 |    8 |   64 | 0.00155654  | 1.28258 |     0.5532 |                   10 |
| DEFAULT_36af6_00003 | TERMINATED |                 |            2 |    4 |    4 | 0.000653041 | 1.36319 |     0.5074 |                    8 |
| DEFAULT_36af6_00004 | TERMINATED |                 |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00005 | TERMINATED |                 |            4 |  256 |  128 | 0.000942874 | 1.24122 |     0.6066 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |                 |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00008 | TERMINATED |                 |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |                 |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


== Status ==
Memory usage on this node: 4.0/240.1 GiB
Using AsyncHyperBand: num_stopped=10
Bracket: Iter 8.000: -1.2750557748377322 | Iter 4.000: -1.3563910078689454 | Iter 2.000: -1.485812882439792 | Iter 1.000: -1.7627342871665954
Resources requested: 0/32 CPUs, 0/2 GPUs, 0.0/157.71 GiB heap, 0.0/49.37 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT_2020-11-17_21-45-30
Number of trials: 10/10 (10 TERMINATED)
+---------------------+------------+-------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc   |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_36af6_00000 | TERMINATED |       |           16 |  128 |   16 | 0.0845979   | 2.30523 |     0.1022 |                    1 |
| DEFAULT_36af6_00001 | TERMINATED |       |           16 |   32 |  256 | 0.0302299   | 2.19249 |     0.1573 |                   10 |
| DEFAULT_36af6_00002 | TERMINATED |       |            4 |    8 |   64 | 0.00155654  | 1.28258 |     0.5532 |                   10 |
| DEFAULT_36af6_00003 | TERMINATED |       |            2 |    4 |    4 | 0.000653041 | 1.36319 |     0.5074 |                    8 |
| DEFAULT_36af6_00004 | TERMINATED |       |           16 |    8 |   64 | 0.00135388  | 1.13683 |     0.5961 |                   10 |
| DEFAULT_36af6_00005 | TERMINATED |       |            4 |  256 |  128 | 0.000942874 | 1.24122 |     0.6066 |                   10 |
| DEFAULT_36af6_00006 | TERMINATED |       |            4 |    4 |    8 | 0.0779381   | 2.33526 |     0.0989 |                    1 |
| DEFAULT_36af6_00007 | TERMINATED |       |            2 |   32 |   32 | 0.000245868 | 1.12111 |     0.6025 |                   10 |
| DEFAULT_36af6_00008 | TERMINATED |       |           16 |    4 |   64 | 0.0001142   | 2.30275 |     0.0974 |                    4 |
| DEFAULT_36af6_00009 | TERMINATED |       |            8 |    4 |  256 | 0.000333676 | 1.34397 |     0.4914 |                   10 |
+---------------------+------------+-------+--------------+------+------+-------------+---------+------------+----------------------+


Best trial config: {'l1': 32, 'l2': 32, 'lr': 0.00024586762236648017, 'batch_size': 2}
Best trial final validation loss: 1.121114240665501
Best trial final validation accuracy: 0.6025
Files already downloaded and verified
Files already downloaded and verified
Best trial test set accuracy: 0.6106

If you run the code, an example output could look like this:

Most trials have been stopped early in order to avoid wasting resources. The best performing trial achieved a validation accuracy of about 58%, which could be confirmed on the test set.

So that’s it! You can now tune the parameters of your PyTorch models.