Note
Click here to download the full example code
Hyperparameter tuning with Ray Tune¶
Hyperparameter tuning can make the difference between an average model and a highly accurate one. Often simple things like choosing a different learning rate or changing a network layer size can have a dramatic impact on your model performance.
Fortunately, there are tools that help with finding the best combination of parameters. Ray Tune is an industry standard tool for distributed hyperparameter tuning. Ray Tune includes the latest hyperparameter search algorithms, integrates with TensorBoard and other analysis libraries, and natively supports distributed training through Ray’s distributed machine learning engine.
In this tutorial, we will show you how to integrate Ray Tune into your PyTorch training workflow. We will extend this tutorial from the PyTorch documentation for training a CIFAR10 image classifier.
As you will see, we only need to add some slight modifications. In particular, we need to
- wrap data loading and training in functions,
- make some network parameters configurable,
- add checkpointing (optional),
- and define the search space for the model tuning
To run this tutorial, please make sure the following packages are installed:
ray[tune]: Distributed hyperparameter tuning librarytorchvision: For the data transformers
Setup / Imports¶
Let’s start with the imports:
from functools import partial
import numpy as np
import os
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import random_split
import torchvision
import torchvision.transforms as transforms
from ray import tune
from ray.tune import CLIReporter
from ray.tune.schedulers import ASHAScheduler
Most of the imports are needed for building the PyTorch model. Only the last three imports are for Ray Tune.
Data loaders¶
We wrap the data loaders in their own function and pass a global data directory. This way we can share a data directory between different trials.
def load_data(data_dir="./data"):
transform = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])
trainset = torchvision.datasets.CIFAR10(
root=data_dir, train=True, download=True, transform=transform)
testset = torchvision.datasets.CIFAR10(
root=data_dir, train=False, download=True, transform=transform)
return trainset, testset
Configurable neural network¶
We can only tune those parameters that are configurable. In this example, we can specify the layer sizes of the fully connected layers:
class Net(nn.Module):
def __init__(self, l1=120, l2=84):
super(Net, self).__init__()
self.conv1 = nn.Conv2d(3, 6, 5)
self.pool = nn.MaxPool2d(2, 2)
self.conv2 = nn.Conv2d(6, 16, 5)
self.fc1 = nn.Linear(16 * 5 * 5, l1)
self.fc2 = nn.Linear(l1, l2)
self.fc3 = nn.Linear(l2, 10)
def forward(self, x):
x = self.pool(F.relu(self.conv1(x)))
x = self.pool(F.relu(self.conv2(x)))
x = x.view(-1, 16 * 5 * 5)
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
x = self.fc3(x)
return x
The train function¶
Now it gets interesting, because we introduce some changes to the example from the PyTorch documentation.
We wrap the training script in a function train_cifar(config, checkpoint_dir=None, data_dir=None).
As you can guess, the config parameter will receive the hyperparameters we would like to
train with. The checkpoint_dir parameter is used to restore checkpoints. The data_dir specifies
the directory where we load and store the data, so multiple runs can share the same data source.
net = Net(config["l1"], config["l2"])
if checkpoint_dir:
model_state, optimizer_state = torch.load(
os.path.join(checkpoint_dir, "checkpoint"))
net.load_state_dict(model_state)
optimizer.load_state_dict(optimizer_state)
The learning rate of the optimizer is made configurable, too:
optimizer = optim.SGD(net.parameters(), lr=config["lr"], momentum=0.9)
We also split the training data into a training and validation subset. We thus train on 80% of the data and calculate the validation loss on the remaining 20%. The batch sizes with which we iterate through the training and test sets are configurable as well.
Adding (multi) GPU support with DataParallel¶
Image classification benefits largely from GPUs. Luckily, we can continue to use
PyTorch’s abstractions in Ray Tune. Thus, we can wrap our model in nn.DataParallel
to support data parallel training on multiple GPUs:
device = "cpu"
if torch.cuda.is_available():
device = "cuda:0"
if torch.cuda.device_count() > 1:
net = nn.DataParallel(net)
net.to(device)
By using a device variable we make sure that training also works when we have
no GPUs available. PyTorch requires us to send our data to the GPU memory explicitly,
like this:
for i, data in enumerate(trainloader, 0):
inputs, labels = data
inputs, labels = inputs.to(device), labels.to(device)
The code now supports training on CPUs, on a single GPU, and on multiple GPUs. Notably, Ray also supports fractional GPUs so we can share GPUs among trials, as long as the model still fits on the GPU memory. We’ll come back to that later.
Communicating with Ray Tune¶
The most interesting part is the communication with Ray Tune:
with tune.checkpoint_dir(epoch) as checkpoint_dir:
path = os.path.join(checkpoint_dir, "checkpoint")
torch.save((net.state_dict(), optimizer.state_dict()), path)
tune.report(loss=(val_loss / val_steps), accuracy=correct / total)
Here we first save a checkpoint and then report some metrics back to Ray Tune. Specifically, we send the validation loss and accuracy back to Ray Tune. Ray Tune can then use these metrics to decide which hyperparameter configuration lead to the best results. These metrics can also be used to stop bad performing trials early in order to avoid wasting resources on those trials.
The checkpoint saving is optional, however, it is necessary if we wanted to use advanced schedulers like Population Based Training. Also, by saving the checkpoint we can later load the trained models and validate them on a test set.
Full training function¶
The full code example looks like this:
def train_cifar(config, checkpoint_dir=None, data_dir=None):
net = Net(config["l1"], config["l2"])
device = "cpu"
if torch.cuda.is_available():
device = "cuda:0"
if torch.cuda.device_count() > 1:
net = nn.DataParallel(net)
net.to(device)
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=config["lr"], momentum=0.9)
if checkpoint_dir:
model_state, optimizer_state = torch.load(
os.path.join(checkpoint_dir, "checkpoint"))
net.load_state_dict(model_state)
optimizer.load_state_dict(optimizer_state)
trainset, testset = load_data(data_dir)
test_abs = int(len(trainset) * 0.8)
train_subset, val_subset = random_split(
trainset, [test_abs, len(trainset) - test_abs])
trainloader = torch.utils.data.DataLoader(
train_subset,
batch_size=int(config["batch_size"]),
shuffle=True,
num_workers=8)
valloader = torch.utils.data.DataLoader(
val_subset,
batch_size=int(config["batch_size"]),
shuffle=True,
num_workers=8)
for epoch in range(10): # loop over the dataset multiple times
running_loss = 0.0
epoch_steps = 0
for i, data in enumerate(trainloader, 0):
# get the inputs; data is a list of [inputs, labels]
inputs, labels = data
inputs, labels = inputs.to(device), labels.to(device)
# zero the parameter gradients
optimizer.zero_grad()
# forward + backward + optimize
outputs = net(inputs)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
# print statistics
running_loss += loss.item()
epoch_steps += 1
if i % 2000 == 1999: # print every 2000 mini-batches
print("[%d, %5d] loss: %.3f" % (epoch + 1, i + 1,
running_loss / epoch_steps))
running_loss = 0.0
# Validation loss
val_loss = 0.0
val_steps = 0
total = 0
correct = 0
for i, data in enumerate(valloader, 0):
with torch.no_grad():
inputs, labels = data
inputs, labels = inputs.to(device), labels.to(device)
outputs = net(inputs)
_, predicted = torch.max(outputs.data, 1)
total += labels.size(0)
correct += (predicted == labels).sum().item()
loss = criterion(outputs, labels)
val_loss += loss.cpu().numpy()
val_steps += 1
with tune.checkpoint_dir(epoch) as checkpoint_dir:
path = os.path.join(checkpoint_dir, "checkpoint")
torch.save((net.state_dict(), optimizer.state_dict()), path)
tune.report(loss=(val_loss / val_steps), accuracy=correct / total)
print("Finished Training")
As you can see, most of the code is adapted directly from the original example.
Test set accuracy¶
Commonly the performance of a machine learning model is tested on a hold-out test set with data that has not been used for training the model. We also wrap this in a function:
def test_accuracy(net, device="cpu"):
trainset, testset = load_data()
testloader = torch.utils.data.DataLoader(
testset, batch_size=4, shuffle=False, num_workers=2)
correct = 0
total = 0
with torch.no_grad():
for data in testloader:
images, labels = data
images, labels = images.to(device), labels.to(device)
outputs = net(images)
_, predicted = torch.max(outputs.data, 1)
total += labels.size(0)
correct += (predicted == labels).sum().item()
return correct / total
The function also expects a device parameter, so we can do the
test set validation on a GPU.
Configuring the search space¶
Lastly, we need to define Ray Tune’s search space. Here is an example:
config = {
"l1": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
"l2": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
"lr": tune.loguniform(1e-4, 1e-1),
"batch_size": tune.choice([2, 4, 8, 16])
}
The tune.sample_from() function makes it possible to define your own sample
methods to obtain hyperparameters. In this example, the l1 and l2 parameters
should be powers of 2 between 4 and 256, so either 4, 8, 16, 32, 64, 128, or 256.
The lr (learning rate) should be uniformly sampled between 0.0001 and 0.1. Lastly,
the batch size is a choice between 2, 4, 8, and 16.
At each trial, Ray Tune will now randomly sample a combination of parameters from these
search spaces. It will then train a number of models in parallel and find the best
performing one among these. We also use the ASHAScheduler which will terminate bad
performing trials early.
We wrap the train_cifar function with functools.partial to set the constant
data_dir parameter. We can also tell Ray Tune what resources should be
available for each trial:
gpus_per_trial = 2
# ...
result = tune.run(
partial(train_cifar, data_dir=data_dir),
resources_per_trial={"cpu": 8, "gpu": gpus_per_trial},
config=config,
num_samples=num_samples,
scheduler=scheduler,
progress_reporter=reporter,
checkpoint_at_end=True)
You can specify the number of CPUs, which are then available e.g.
to increase the num_workers of the PyTorch DataLoader instances. The selected
number of GPUs are made visible to PyTorch in each trial. Trials do not have access to
GPUs that haven’t been requested for them - so you don’t have to care about two trials
using the same set of resources.
Here we can also specify fractional GPUs, so something like gpus_per_trial=0.5 is
completely valid. The trials will then share GPUs among each other.
You just have to make sure that the models still fit in the GPU memory.
After training the models, we will find the best performing one and load the trained network from the checkpoint file. We then obtain the test set accuracy and report everything by printing.
The full main function looks like this:
def main(num_samples=10, max_num_epochs=10, gpus_per_trial=2):
data_dir = os.path.abspath("./data")
load_data(data_dir)
config = {
"l1": tune.sample_from(lambda _: 2 ** np.random.randint(2, 9)),
"l2": tune.sample_from(lambda _: 2 ** np.random.randint(2, 9)),
"lr": tune.loguniform(1e-4, 1e-1),
"batch_size": tune.choice([2, 4, 8, 16])
}
scheduler = ASHAScheduler(
metric="loss",
mode="min",
max_t=max_num_epochs,
grace_period=1,
reduction_factor=2)
reporter = CLIReporter(
# parameter_columns=["l1", "l2", "lr", "batch_size"],
metric_columns=["loss", "accuracy", "training_iteration"])
result = tune.run(
partial(train_cifar, data_dir=data_dir),
resources_per_trial={"cpu": 2, "gpu": gpus_per_trial},
config=config,
num_samples=num_samples,
scheduler=scheduler,
progress_reporter=reporter)
best_trial = result.get_best_trial("loss", "min", "last")
print("Best trial config: {}".format(best_trial.config))
print("Best trial final validation loss: {}".format(
best_trial.last_result["loss"]))
print("Best trial final validation accuracy: {}".format(
best_trial.last_result["accuracy"]))
best_trained_model = Net(best_trial.config["l1"], best_trial.config["l2"])
device = "cpu"
if torch.cuda.is_available():
device = "cuda:0"
if gpus_per_trial > 1:
best_trained_model = nn.DataParallel(best_trained_model)
best_trained_model.to(device)
best_checkpoint_dir = best_trial.checkpoint.value
model_state, optimizer_state = torch.load(os.path.join(
best_checkpoint_dir, "checkpoint"))
best_trained_model.load_state_dict(model_state)
test_acc = test_accuracy(best_trained_model, device)
print("Best trial test set accuracy: {}".format(test_acc))
if __name__ == "__main__":
# You can change the number of GPUs per trial here:
main(num_samples=10, max_num_epochs=10, gpus_per_trial=0)
Out:
Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to /var/lib/jenkins/workspace/beginner_source/data/cifar-10-python.tar.gz
Extracting /var/lib/jenkins/workspace/beginner_source/data/cifar-10-python.tar.gz to /var/lib/jenkins/workspace/beginner_source/data
Files already downloaded and verified
== Status ==
Memory usage on this node: 4.1/240.1 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 2/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (9 PENDING, 1 RUNNING)
+---------------------+----------+-------+--------------+------+------+-------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr |
|---------------------+----------+-------+--------------+------+------+-------------|
| DEFAULT_1dde1_00000 | RUNNING | | 4 | 128 | 4 | 0.00648265 |
| DEFAULT_1dde1_00001 | PENDING | | 2 | 16 | 4 | 0.000419419 |
| DEFAULT_1dde1_00002 | PENDING | | 16 | 64 | 128 | 0.0486304 |
| DEFAULT_1dde1_00003 | PENDING | | 2 | 32 | 4 | 0.0290972 |
| DEFAULT_1dde1_00004 | PENDING | | 16 | 16 | 128 | 0.0422261 |
| DEFAULT_1dde1_00005 | PENDING | | 8 | 256 | 128 | 0.000362061 |
| DEFAULT_1dde1_00006 | PENDING | | 4 | 256 | 16 | 0.00696066 |
| DEFAULT_1dde1_00007 | PENDING | | 4 | 128 | 32 | 0.00139994 |
| DEFAULT_1dde1_00008 | PENDING | | 8 | 8 | 128 | 0.000178276 |
| DEFAULT_1dde1_00009 | PENDING | | 2 | 32 | 32 | 0.000438826 |
+---------------------+----------+-------+--------------+------+------+-------------+
[2m[36m(pid=1122)[0m Files already downloaded and verified
[2m[36m(pid=1168)[0m Files already downloaded and verified
[2m[36m(pid=1112)[0m Files already downloaded and verified
[2m[36m(pid=1084)[0m Files already downloaded and verified
[2m[36m(pid=1090)[0m Files already downloaded and verified
[2m[36m(pid=1118)[0m Files already downloaded and verified
[2m[36m(pid=1104)[0m Files already downloaded and verified
[2m[36m(pid=1097)[0m Files already downloaded and verified
[2m[36m(pid=1108)[0m Files already downloaded and verified
[2m[36m(pid=1085)[0m Files already downloaded and verified
[2m[36m(pid=1122)[0m Files already downloaded and verified
[2m[36m(pid=1168)[0m Files already downloaded and verified
[2m[36m(pid=1112)[0m Files already downloaded and verified
[2m[36m(pid=1084)[0m Files already downloaded and verified
[2m[36m(pid=1090)[0m Files already downloaded and verified
[2m[36m(pid=1118)[0m Files already downloaded and verified
[2m[36m(pid=1104)[0m Files already downloaded and verified
[2m[36m(pid=1097)[0m Files already downloaded and verified
[2m[36m(pid=1108)[0m Files already downloaded and verified
[2m[36m(pid=1085)[0m Files already downloaded and verified
[2m[36m(pid=1122)[0m [1, 2000] loss: 2.310
[2m[36m(pid=1112)[0m [1, 2000] loss: 2.339
[2m[36m(pid=1097)[0m [1, 2000] loss: 2.300
[2m[36m(pid=1118)[0m [1, 2000] loss: 2.215
[2m[36m(pid=1085)[0m [1, 2000] loss: 2.174
[2m[36m(pid=1104)[0m [1, 2000] loss: 2.049
[2m[36m(pid=1084)[0m [1, 2000] loss: 2.303
[2m[36m(pid=1108)[0m [1, 2000] loss: 2.290
[2m[36m(pid=1090)[0m [1, 2000] loss: 2.080
[2m[36m(pid=1168)[0m [1, 2000] loss: 2.066
[2m[36m(pid=1122)[0m [1, 4000] loss: 1.124
[2m[36m(pid=1097)[0m [1, 4000] loss: 1.107
[2m[36m(pid=1112)[0m [1, 4000] loss: 1.167
[2m[36m(pid=1118)[0m [1, 4000] loss: 1.000
[2m[36m(pid=1085)[0m [1, 4000] loss: 0.936
Result for DEFAULT_1dde1_00004:
accuracy: 0.2354
date: 2020-09-22_01-01-47
done: false
experiment_id: 0e63a81fef9141bdb0635e2bd7085e0e
experiment_tag: 4_batch_size=16,l1=16,l2=128,lr=0.042226
hostname: fed1025d76e5
iterations_since_restore: 1
loss: 1.9921342937469482
node_ip: 172.17.0.2
pid: 1090
should_checkpoint: true
time_since_restore: 30.549424409866333
time_this_iter_s: 30.549424409866333
time_total_s: 30.549424409866333
timestamp: 1600736507
timesteps_since_restore: 0
training_iteration: 1
trial_id: 1dde1_00004
== Status ==
Memory usage on this node: 8.9/240.1 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -1.9921342937469482
Resources requested: 20/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (10 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | RUNNING | | 4 | 128 | 4 | 0.00648265 | | | |
| DEFAULT_1dde1_00001 | RUNNING | | 2 | 16 | 4 | 0.000419419 | | | |
| DEFAULT_1dde1_00002 | RUNNING | | 16 | 64 | 128 | 0.0486304 | | | |
| DEFAULT_1dde1_00003 | RUNNING | | 2 | 32 | 4 | 0.0290972 | | | |
| DEFAULT_1dde1_00004 | RUNNING | 172.17.0.2:1090 | 16 | 16 | 128 | 0.0422261 | 1.99213 | 0.2354 | 1 |
| DEFAULT_1dde1_00005 | RUNNING | | 8 | 256 | 128 | 0.000362061 | | | |
| DEFAULT_1dde1_00006 | RUNNING | | 4 | 256 | 16 | 0.00696066 | | | |
| DEFAULT_1dde1_00007 | RUNNING | | 4 | 128 | 32 | 0.00139994 | | | |
| DEFAULT_1dde1_00008 | RUNNING | | 8 | 8 | 128 | 0.000178276 | | | |
| DEFAULT_1dde1_00009 | RUNNING | | 2 | 32 | 32 | 0.000438826 | | | |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1104)[0m [1, 4000] loss: 0.927
Result for DEFAULT_1dde1_00002:
accuracy: 0.2616
date: 2020-09-22_01-01-48
done: true
experiment_id: 28d45c999f63400bba7dbc305fc5a1be
experiment_tag: 2_batch_size=16,l1=64,l2=128,lr=0.04863
hostname: fed1025d76e5
iterations_since_restore: 1
loss: 2.0254285234451292
node_ip: 172.17.0.2
pid: 1168
should_checkpoint: true
time_since_restore: 31.062172651290894
time_this_iter_s: 31.062172651290894
time_total_s: 31.062172651290894
timestamp: 1600736508
timesteps_since_restore: 0
training_iteration: 1
trial_id: 1dde1_00002
[2m[36m(pid=1084)[0m [1, 4000] loss: 1.150
[2m[36m(pid=1108)[0m [1, 4000] loss: 1.045
[2m[36m(pid=1122)[0m [1, 6000] loss: 0.700
[2m[36m(pid=1097)[0m [1, 6000] loss: 0.668
[2m[36m(pid=1112)[0m [1, 6000] loss: 0.778
[2m[36m(pid=1118)[0m [1, 6000] loss: 0.657
[2m[36m(pid=1085)[0m [1, 6000] loss: 0.557
[2m[36m(pid=1104)[0m [1, 6000] loss: 0.597
Result for DEFAULT_1dde1_00008:
accuracy: 0.1467
date: 2020-09-22_01-02-00
done: true
experiment_id: 07b7e6afe93442f599e1fa458d9d45c7
experiment_tag: 8_batch_size=8,l1=8,l2=128,lr=0.00017828
hostname: fed1025d76e5
iterations_since_restore: 1
loss: 2.2940939085006713
node_ip: 172.17.0.2
pid: 1084
should_checkpoint: true
time_since_restore: 43.29920506477356
time_this_iter_s: 43.29920506477356
time_total_s: 43.29920506477356
timestamp: 1600736520
timesteps_since_restore: 0
training_iteration: 1
trial_id: 1dde1_00008
== Status ==
Memory usage on this node: 8.4/240.1 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -2.0254285234451292
Resources requested: 18/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (9 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | RUNNING | | 4 | 128 | 4 | 0.00648265 | | | |
| DEFAULT_1dde1_00001 | RUNNING | | 2 | 16 | 4 | 0.000419419 | | | |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | RUNNING | | 2 | 32 | 4 | 0.0290972 | | | |
| DEFAULT_1dde1_00004 | RUNNING | 172.17.0.2:1090 | 16 | 16 | 128 | 0.0422261 | 1.99213 | 0.2354 | 1 |
| DEFAULT_1dde1_00005 | RUNNING | | 8 | 256 | 128 | 0.000362061 | | | |
| DEFAULT_1dde1_00006 | RUNNING | | 4 | 256 | 16 | 0.00696066 | | | |
| DEFAULT_1dde1_00007 | RUNNING | | 4 | 128 | 32 | 0.00139994 | | | |
| DEFAULT_1dde1_00008 | RUNNING | 172.17.0.2:1084 | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | | 2 | 32 | 32 | 0.000438826 | | | |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1090)[0m [2, 2000] loss: 2.075
Result for DEFAULT_1dde1_00005:
accuracy: 0.3182
date: 2020-09-22_01-02-03
done: false
experiment_id: 6a1270e5c59840d8acaa81bba30ebb03
experiment_tag: 5_batch_size=8,l1=256,l2=128,lr=0.00036206
hostname: fed1025d76e5
iterations_since_restore: 1
loss: 1.8762480222702027
node_ip: 172.17.0.2
pid: 1108
should_checkpoint: true
time_since_restore: 45.70140099525452
time_this_iter_s: 45.70140099525452
time_total_s: 45.70140099525452
timestamp: 1600736523
timesteps_since_restore: 0
training_iteration: 1
trial_id: 1dde1_00005
[2m[36m(pid=1122)[0m [1, 8000] loss: 0.491
[2m[36m(pid=1097)[0m [1, 8000] loss: 0.463
[2m[36m(pid=1112)[0m [1, 8000] loss: 0.583
[2m[36m(pid=1118)[0m [1, 8000] loss: 0.484
[2m[36m(pid=1085)[0m [1, 8000] loss: 0.396
[2m[36m(pid=1104)[0m [1, 8000] loss: 0.446
Result for DEFAULT_1dde1_00004:
accuracy: 0.1864
date: 2020-09-22_01-02-09
done: false
experiment_id: 0e63a81fef9141bdb0635e2bd7085e0e
experiment_tag: 4_batch_size=16,l1=16,l2=128,lr=0.042226
hostname: fed1025d76e5
iterations_since_restore: 2
loss: 2.11986173248291
node_ip: 172.17.0.2
pid: 1090
should_checkpoint: true
time_since_restore: 52.03898477554321
time_this_iter_s: 21.48956036567688
time_total_s: 52.03898477554321
timestamp: 1600736529
timesteps_since_restore: 0
training_iteration: 2
trial_id: 1dde1_00004
== Status ==
Memory usage on this node: 7.8/240.1 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -2.11986173248291 | Iter 1.000: -2.0087814085960387
Resources requested: 16/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (8 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | RUNNING | | 4 | 128 | 4 | 0.00648265 | | | |
| DEFAULT_1dde1_00001 | RUNNING | | 2 | 16 | 4 | 0.000419419 | | | |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | RUNNING | | 2 | 32 | 4 | 0.0290972 | | | |
| DEFAULT_1dde1_00004 | RUNNING | 172.17.0.2:1090 | 16 | 16 | 128 | 0.0422261 | 2.11986 | 0.1864 | 2 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.87625 | 0.3182 | 1 |
| DEFAULT_1dde1_00006 | RUNNING | | 4 | 256 | 16 | 0.00696066 | | | |
| DEFAULT_1dde1_00007 | RUNNING | | 4 | 128 | 32 | 0.00139994 | | | |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | | 2 | 32 | 32 | 0.000438826 | | | |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1122)[0m [1, 10000] loss: 0.378
[2m[36m(pid=1097)[0m [1, 10000] loss: 0.351
[2m[36m(pid=1112)[0m [1, 10000] loss: 0.468
[2m[36m(pid=1108)[0m [2, 2000] loss: 1.810
[2m[36m(pid=1118)[0m [1, 10000] loss: 0.382
[2m[36m(pid=1085)[0m [1, 10000] loss: 0.305
[2m[36m(pid=1104)[0m [1, 10000] loss: 0.356
[2m[36m(pid=1122)[0m [1, 12000] loss: 0.303
[2m[36m(pid=1097)[0m [1, 12000] loss: 0.282
[2m[36m(pid=1112)[0m [1, 12000] loss: 0.390
[2m[36m(pid=1090)[0m [3, 2000] loss: 2.039
Result for DEFAULT_1dde1_00000:
accuracy: 0.2675
date: 2020-09-22_01-02-24
done: true
experiment_id: 29a6426cadfc4173891a788f70eb2ee4
experiment_tag: 0_batch_size=4,l1=128,l2=4,lr=0.0064826
hostname: fed1025d76e5
iterations_since_restore: 1
loss: 2.025877732205391
node_ip: 172.17.0.2
pid: 1118
should_checkpoint: true
time_since_restore: 67.19454288482666
time_this_iter_s: 67.19454288482666
time_total_s: 67.19454288482666
timestamp: 1600736544
timesteps_since_restore: 0
training_iteration: 1
trial_id: 1dde1_00000
== Status ==
Memory usage on this node: 7.9/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -2.11986173248291 | Iter 1.000: -2.0254285234451292
Resources requested: 16/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (8 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | RUNNING | 172.17.0.2:1118 | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | | 2 | 16 | 4 | 0.000419419 | | | |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | RUNNING | | 2 | 32 | 4 | 0.0290972 | | | |
| DEFAULT_1dde1_00004 | RUNNING | 172.17.0.2:1090 | 16 | 16 | 128 | 0.0422261 | 2.11986 | 0.1864 | 2 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.87625 | 0.3182 | 1 |
| DEFAULT_1dde1_00006 | RUNNING | | 4 | 256 | 16 | 0.00696066 | | | |
| DEFAULT_1dde1_00007 | RUNNING | | 4 | 128 | 32 | 0.00139994 | | | |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | | 2 | 32 | 32 | 0.000438826 | | | |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_1dde1_00007:
accuracy: 0.4737
date: 2020-09-22_01-02-25
done: false
experiment_id: d130aa97ad4d43d1a015b0a98a769c9b
experiment_tag: 7_batch_size=4,l1=128,l2=32,lr=0.0013999
hostname: fed1025d76e5
iterations_since_restore: 1
loss: 1.46903585190773
node_ip: 172.17.0.2
pid: 1085
should_checkpoint: true
time_since_restore: 67.89717030525208
time_this_iter_s: 67.89717030525208
time_total_s: 67.89717030525208
timestamp: 1600736545
timesteps_since_restore: 0
training_iteration: 1
trial_id: 1dde1_00007
Result for DEFAULT_1dde1_00006:
accuracy: 0.3355
date: 2020-09-22_01-02-27
done: false
experiment_id: c90e171237b14250992a8ac860d4bf3d
experiment_tag: 6_batch_size=4,l1=256,l2=16,lr=0.0069607
hostname: fed1025d76e5
iterations_since_restore: 1
loss: 1.746452518939972
node_ip: 172.17.0.2
pid: 1104
should_checkpoint: true
time_since_restore: 69.79255509376526
time_this_iter_s: 69.79255509376526
time_total_s: 69.79255509376526
timestamp: 1600736547
timesteps_since_restore: 0
training_iteration: 1
trial_id: 1dde1_00006
[2m[36m(pid=1108)[0m [2, 4000] loss: 0.840
[2m[36m(pid=1122)[0m [1, 14000] loss: 0.253
Result for DEFAULT_1dde1_00004:
accuracy: 0.2282
date: 2020-09-22_01-02-30
done: false
experiment_id: 0e63a81fef9141bdb0635e2bd7085e0e
experiment_tag: 4_batch_size=16,l1=16,l2=128,lr=0.042226
hostname: fed1025d76e5
iterations_since_restore: 3
loss: 1.9991267137527466
node_ip: 172.17.0.2
pid: 1090
should_checkpoint: true
time_since_restore: 72.84821891784668
time_this_iter_s: 20.809234142303467
time_total_s: 72.84821891784668
timestamp: 1600736550
timesteps_since_restore: 0
training_iteration: 3
trial_id: 1dde1_00004
== Status ==
Memory usage on this node: 7.4/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -2.11986173248291 | Iter 1.000: -1.9921342937469482
Resources requested: 14/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | | 2 | 16 | 4 | 0.000419419 | | | |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | RUNNING | | 2 | 32 | 4 | 0.0290972 | | | |
| DEFAULT_1dde1_00004 | RUNNING | 172.17.0.2:1090 | 16 | 16 | 128 | 0.0422261 | 1.99913 | 0.2282 | 3 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.87625 | 0.3182 | 1 |
| DEFAULT_1dde1_00006 | RUNNING | 172.17.0.2:1104 | 4 | 256 | 16 | 0.00696066 | 1.74645 | 0.3355 | 1 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.46904 | 0.4737 | 1 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | | 2 | 32 | 32 | 0.000438826 | | | |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1097)[0m [1, 14000] loss: 0.234
[2m[36m(pid=1112)[0m [1, 14000] loss: 0.334
[2m[36m(pid=1085)[0m [2, 2000] loss: 1.454
[2m[36m(pid=1104)[0m [2, 2000] loss: 1.762
Result for DEFAULT_1dde1_00005:
accuracy: 0.4157
date: 2020-09-22_01-02-38
done: false
experiment_id: 6a1270e5c59840d8acaa81bba30ebb03
experiment_tag: 5_batch_size=8,l1=256,l2=128,lr=0.00036206
hostname: fed1025d76e5
iterations_since_restore: 2
loss: 1.6115445574760436
node_ip: 172.17.0.2
pid: 1108
should_checkpoint: true
time_since_restore: 80.84446930885315
time_this_iter_s: 35.14306831359863
time_total_s: 80.84446930885315
timestamp: 1600736558
timesteps_since_restore: 0
training_iteration: 2
trial_id: 1dde1_00005
== Status ==
Memory usage on this node: 7.4/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.8657031449794768 | Iter 1.000: -1.9921342937469482
Resources requested: 14/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | | 2 | 16 | 4 | 0.000419419 | | | |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | RUNNING | | 2 | 32 | 4 | 0.0290972 | | | |
| DEFAULT_1dde1_00004 | RUNNING | 172.17.0.2:1090 | 16 | 16 | 128 | 0.0422261 | 1.99913 | 0.2282 | 3 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.61154 | 0.4157 | 2 |
| DEFAULT_1dde1_00006 | RUNNING | 172.17.0.2:1104 | 4 | 256 | 16 | 0.00696066 | 1.74645 | 0.3355 | 1 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.46904 | 0.4737 | 1 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | | 2 | 32 | 32 | 0.000438826 | | | |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1122)[0m [1, 16000] loss: 0.218
[2m[36m(pid=1097)[0m [1, 16000] loss: 0.200
[2m[36m(pid=1112)[0m [1, 16000] loss: 0.292
[2m[36m(pid=1090)[0m [4, 2000] loss: 2.210
[2m[36m(pid=1085)[0m [2, 4000] loss: 0.723
[2m[36m(pid=1122)[0m [1, 18000] loss: 0.192
[2m[36m(pid=1104)[0m [2, 4000] loss: 0.883
[2m[36m(pid=1097)[0m [1, 18000] loss: 0.176
[2m[36m(pid=1112)[0m [1, 18000] loss: 0.260
[2m[36m(pid=1108)[0m [3, 2000] loss: 1.574
Result for DEFAULT_1dde1_00004:
accuracy: 0.2056
date: 2020-09-22_01-02-50
done: false
experiment_id: 0e63a81fef9141bdb0635e2bd7085e0e
experiment_tag: 4_batch_size=16,l1=16,l2=128,lr=0.042226
hostname: fed1025d76e5
iterations_since_restore: 4
loss: 2.0474100847244263
node_ip: 172.17.0.2
pid: 1090
should_checkpoint: true
time_since_restore: 93.15838694572449
time_this_iter_s: 20.310168027877808
time_total_s: 93.15838694572449
timestamp: 1600736570
timesteps_since_restore: 0
training_iteration: 4
trial_id: 1dde1_00004
== Status ==
Memory usage on this node: 7.4/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -2.0474100847244263 | Iter 2.000: -1.8657031449794768 | Iter 1.000: -1.9921342937469482
Resources requested: 14/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | | 2 | 16 | 4 | 0.000419419 | | | |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | RUNNING | | 2 | 32 | 4 | 0.0290972 | | | |
| DEFAULT_1dde1_00004 | RUNNING | 172.17.0.2:1090 | 16 | 16 | 128 | 0.0422261 | 2.04741 | 0.2056 | 4 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.61154 | 0.4157 | 2 |
| DEFAULT_1dde1_00006 | RUNNING | 172.17.0.2:1104 | 4 | 256 | 16 | 0.00696066 | 1.74645 | 0.3355 | 1 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.46904 | 0.4737 | 1 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | | 2 | 32 | 32 | 0.000438826 | | | |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1085)[0m [2, 6000] loss: 0.467
[2m[36m(pid=1122)[0m [1, 20000] loss: 0.171
[2m[36m(pid=1097)[0m [1, 20000] loss: 0.154
[2m[36m(pid=1104)[0m [2, 6000] loss: 0.581
[2m[36m(pid=1112)[0m [1, 20000] loss: 0.233
[2m[36m(pid=1108)[0m [3, 4000] loss: 0.757
[2m[36m(pid=1085)[0m [2, 8000] loss: 0.349
[2m[36m(pid=1090)[0m [5, 2000] loss: 2.049
[2m[36m(pid=1104)[0m [2, 8000] loss: 0.437
Result for DEFAULT_1dde1_00001:
accuracy: 0.3427
date: 2020-09-22_01-03-08
done: false
experiment_id: f8daf9cd06ad4a4b9c3df3513effdf31
experiment_tag: 1_batch_size=2,l1=16,l2=4,lr=0.00041942
hostname: fed1025d76e5
iterations_since_restore: 1
loss: 1.690081286263466
node_ip: 172.17.0.2
pid: 1122
should_checkpoint: true
time_since_restore: 111.34616017341614
time_this_iter_s: 111.34616017341614
time_total_s: 111.34616017341614
timestamp: 1600736588
timesteps_since_restore: 0
training_iteration: 1
trial_id: 1dde1_00001
== Status ==
Memory usage on this node: 7.4/240.1 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -2.0474100847244263 | Iter 2.000: -1.8657031449794768 | Iter 1.000: -1.9341911580085753
Resources requested: 14/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (7 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.69008 | 0.3427 | 1 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | RUNNING | | 2 | 32 | 4 | 0.0290972 | | | |
| DEFAULT_1dde1_00004 | RUNNING | 172.17.0.2:1090 | 16 | 16 | 128 | 0.0422261 | 2.04741 | 0.2056 | 4 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.61154 | 0.4157 | 2 |
| DEFAULT_1dde1_00006 | RUNNING | 172.17.0.2:1104 | 4 | 256 | 16 | 0.00696066 | 1.74645 | 0.3355 | 1 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.46904 | 0.4737 | 1 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | | 2 | 32 | 32 | 0.000438826 | | | |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_1dde1_00009:
accuracy: 0.464
date: 2020-09-22_01-03-09
done: false
experiment_id: cfed644d520f435688772253831a176f
experiment_tag: 9_batch_size=2,l1=32,l2=32,lr=0.00043883
hostname: fed1025d76e5
iterations_since_restore: 1
loss: 1.466922292420268
node_ip: 172.17.0.2
pid: 1097
should_checkpoint: true
time_since_restore: 112.3949773311615
time_this_iter_s: 112.3949773311615
time_total_s: 112.3949773311615
timestamp: 1600736589
timesteps_since_restore: 0
training_iteration: 1
trial_id: 1dde1_00009
Result for DEFAULT_1dde1_00004:
accuracy: 0.208
date: 2020-09-22_01-03-11
done: false
experiment_id: 0e63a81fef9141bdb0635e2bd7085e0e
experiment_tag: 4_batch_size=16,l1=16,l2=128,lr=0.042226
hostname: fed1025d76e5
iterations_since_restore: 5
loss: 2.071832389259338
node_ip: 172.17.0.2
pid: 1090
should_checkpoint: true
time_since_restore: 113.85181093215942
time_this_iter_s: 20.693423986434937
time_total_s: 113.85181093215942
timestamp: 1600736591
timesteps_since_restore: 0
training_iteration: 5
trial_id: 1dde1_00004
Result for DEFAULT_1dde1_00003:
accuracy: 0.0979
date: 2020-09-22_01-03-12
done: true
experiment_id: ab36e11e8e554eb1a9350acb371675f6
experiment_tag: 3_batch_size=2,l1=32,l2=4,lr=0.029097
hostname: fed1025d76e5
iterations_since_restore: 1
loss: 2.3260342679977417
node_ip: 172.17.0.2
pid: 1112
should_checkpoint: true
time_since_restore: 114.85109782218933
time_this_iter_s: 114.85109782218933
time_total_s: 114.85109782218933
timestamp: 1600736592
timesteps_since_restore: 0
training_iteration: 1
trial_id: 1dde1_00003
Result for DEFAULT_1dde1_00005:
accuracy: 0.4589
date: 2020-09-22_01-03-12
done: false
experiment_id: 6a1270e5c59840d8acaa81bba30ebb03
experiment_tag: 5_batch_size=8,l1=256,l2=128,lr=0.00036206
hostname: fed1025d76e5
iterations_since_restore: 3
loss: 1.4968988493680955
node_ip: 172.17.0.2
pid: 1108
should_checkpoint: true
time_since_restore: 115.53939604759216
time_this_iter_s: 34.694926738739014
time_total_s: 115.53939604759216
timestamp: 1600736592
timesteps_since_restore: 0
training_iteration: 3
trial_id: 1dde1_00005
[2m[36m(pid=1085)[0m [2, 10000] loss: 0.274
[2m[36m(pid=1104)[0m [2, 10000] loss: 0.353
[2m[36m(pid=1122)[0m [2, 2000] loss: 1.672
[2m[36m(pid=1097)[0m [2, 2000] loss: 1.524
Result for DEFAULT_1dde1_00007:
accuracy: 0.5112
date: 2020-09-22_01-03-20
done: false
experiment_id: d130aa97ad4d43d1a015b0a98a769c9b
experiment_tag: 7_batch_size=4,l1=128,l2=32,lr=0.0013999
hostname: fed1025d76e5
iterations_since_restore: 2
loss: 1.37119730758667
node_ip: 172.17.0.2
pid: 1085
should_checkpoint: true
time_since_restore: 123.12365698814392
time_this_iter_s: 55.226486682891846
time_total_s: 123.12365698814392
timestamp: 1600736600
timesteps_since_restore: 0
training_iteration: 2
trial_id: 1dde1_00007
== Status ==
Memory usage on this node: 6.9/240.1 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: None | Iter 4.000: -2.0474100847244263 | Iter 2.000: -1.6115445574760436 | Iter 1.000: -1.9341911580085753
Resources requested: 12/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (6 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.69008 | 0.3427 | 1 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | RUNNING | 172.17.0.2:1090 | 16 | 16 | 128 | 0.0422261 | 2.07183 | 0.208 | 5 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.4969 | 0.4589 | 3 |
| DEFAULT_1dde1_00006 | RUNNING | 172.17.0.2:1104 | 4 | 256 | 16 | 0.00696066 | 1.74645 | 0.3355 | 1 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.3712 | 0.5112 | 2 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.46692 | 0.464 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_1dde1_00006:
accuracy: 0.3378
date: 2020-09-22_01-03-23
done: true
experiment_id: c90e171237b14250992a8ac860d4bf3d
experiment_tag: 6_batch_size=4,l1=256,l2=16,lr=0.0069607
hostname: fed1025d76e5
iterations_since_restore: 2
loss: 1.7610837140202522
node_ip: 172.17.0.2
pid: 1104
should_checkpoint: true
time_since_restore: 126.36016178131104
time_this_iter_s: 56.567606687545776
time_total_s: 126.36016178131104
timestamp: 1600736603
timesteps_since_restore: 0
training_iteration: 2
trial_id: 1dde1_00006
[2m[36m(pid=1108)[0m [4, 2000] loss: 1.424
[2m[36m(pid=1090)[0m [6, 2000] loss: 2.023
[2m[36m(pid=1122)[0m [2, 4000] loss: 0.823
[2m[36m(pid=1097)[0m [2, 4000] loss: 0.736
[2m[36m(pid=1085)[0m [3, 2000] loss: 1.296
Result for DEFAULT_1dde1_00004:
accuracy: 0.2284
date: 2020-09-22_01-03-31
done: false
experiment_id: 0e63a81fef9141bdb0635e2bd7085e0e
experiment_tag: 4_batch_size=16,l1=16,l2=128,lr=0.042226
hostname: fed1025d76e5
iterations_since_restore: 6
loss: 2.0067037870407103
node_ip: 172.17.0.2
pid: 1090
should_checkpoint: true
time_since_restore: 133.78824996948242
time_this_iter_s: 19.936439037322998
time_total_s: 133.78824996948242
timestamp: 1600736611
timesteps_since_restore: 0
training_iteration: 6
trial_id: 1dde1_00004
== Status ==
Memory usage on this node: 6.3/240.1 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: None | Iter 4.000: -2.0474100847244263 | Iter 2.000: -1.686314135748148 | Iter 1.000: -1.9341911580085753
Resources requested: 10/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (5 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.69008 | 0.3427 | 1 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | RUNNING | 172.17.0.2:1090 | 16 | 16 | 128 | 0.0422261 | 2.0067 | 0.2284 | 6 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.4969 | 0.4589 | 3 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.3712 | 0.5112 | 2 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.46692 | 0.464 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1122)[0m [2, 6000] loss: 0.540
[2m[36m(pid=1097)[0m [2, 6000] loss: 0.483
[2m[36m(pid=1108)[0m [4, 4000] loss: 0.694
[2m[36m(pid=1085)[0m [3, 4000] loss: 0.636
[2m[36m(pid=1122)[0m [2, 8000] loss: 0.405
[2m[36m(pid=1097)[0m [2, 8000] loss: 0.363
[2m[36m(pid=1090)[0m [7, 2000] loss: 2.042
Result for DEFAULT_1dde1_00005:
accuracy: 0.5079
date: 2020-09-22_01-03-46
done: false
experiment_id: 6a1270e5c59840d8acaa81bba30ebb03
experiment_tag: 5_batch_size=8,l1=256,l2=128,lr=0.00036206
hostname: fed1025d76e5
iterations_since_restore: 4
loss: 1.3617212006807327
node_ip: 172.17.0.2
pid: 1108
should_checkpoint: true
time_since_restore: 148.9325716495514
time_this_iter_s: 33.39317560195923
time_total_s: 148.9325716495514
timestamp: 1600736626
timesteps_since_restore: 0
training_iteration: 4
trial_id: 1dde1_00005
== Status ==
Memory usage on this node: 6.3/240.1 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: None | Iter 4.000: -1.7045656427025795 | Iter 2.000: -1.686314135748148 | Iter 1.000: -1.9341911580085753
Resources requested: 10/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (5 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.69008 | 0.3427 | 1 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | RUNNING | 172.17.0.2:1090 | 16 | 16 | 128 | 0.0422261 | 2.0067 | 0.2284 | 6 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.36172 | 0.5079 | 4 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.3712 | 0.5112 | 2 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.46692 | 0.464 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1085)[0m [3, 6000] loss: 0.425
[2m[36m(pid=1122)[0m [2, 10000] loss: 0.323
[2m[36m(pid=1097)[0m [2, 10000] loss: 0.288
Result for DEFAULT_1dde1_00004:
accuracy: 0.1851
date: 2020-09-22_01-03-50
done: false
experiment_id: 0e63a81fef9141bdb0635e2bd7085e0e
experiment_tag: 4_batch_size=16,l1=16,l2=128,lr=0.042226
hostname: fed1025d76e5
iterations_since_restore: 7
loss: 2.0709309223175048
node_ip: 172.17.0.2
pid: 1090
should_checkpoint: true
time_since_restore: 153.19460248947144
time_this_iter_s: 19.406352519989014
time_total_s: 153.19460248947144
timestamp: 1600736630
timesteps_since_restore: 0
training_iteration: 7
trial_id: 1dde1_00004
[2m[36m(pid=1085)[0m [3, 8000] loss: 0.317
[2m[36m(pid=1122)[0m [2, 12000] loss: 0.265
[2m[36m(pid=1108)[0m [5, 2000] loss: 1.321
[2m[36m(pid=1097)[0m [2, 12000] loss: 0.234
[2m[36m(pid=1090)[0m [8, 2000] loss: 2.059
[2m[36m(pid=1122)[0m [2, 14000] loss: 0.228
[2m[36m(pid=1085)[0m [3, 10000] loss: 0.255
[2m[36m(pid=1097)[0m [2, 14000] loss: 0.198
[2m[36m(pid=1108)[0m [5, 4000] loss: 0.643
Result for DEFAULT_1dde1_00004:
accuracy: 0.2073
date: 2020-09-22_01-04-10
done: false
experiment_id: 0e63a81fef9141bdb0635e2bd7085e0e
experiment_tag: 4_batch_size=16,l1=16,l2=128,lr=0.042226
hostname: fed1025d76e5
iterations_since_restore: 8
loss: 2.0788925355911254
node_ip: 172.17.0.2
pid: 1090
should_checkpoint: true
time_since_restore: 172.85381960868835
time_this_iter_s: 19.65921711921692
time_total_s: 172.85381960868835
timestamp: 1600736650
timesteps_since_restore: 0
training_iteration: 8
trial_id: 1dde1_00004
== Status ==
Memory usage on this node: 6.3/240.1 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -2.0788925355911254 | Iter 4.000: -1.7045656427025795 | Iter 2.000: -1.686314135748148 | Iter 1.000: -1.9341911580085753
Resources requested: 10/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (5 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.69008 | 0.3427 | 1 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | RUNNING | 172.17.0.2:1090 | 16 | 16 | 128 | 0.0422261 | 2.07889 | 0.2073 | 8 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.36172 | 0.5079 | 4 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.3712 | 0.5112 | 2 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.46692 | 0.464 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_1dde1_00007:
accuracy: 0.5556
date: 2020-09-22_01-04-12
done: false
experiment_id: d130aa97ad4d43d1a015b0a98a769c9b
experiment_tag: 7_batch_size=4,l1=128,l2=32,lr=0.0013999
hostname: fed1025d76e5
iterations_since_restore: 3
loss: 1.2556879827991128
node_ip: 172.17.0.2
pid: 1085
should_checkpoint: true
time_since_restore: 174.86849665641785
time_this_iter_s: 51.744839668273926
time_total_s: 174.86849665641785
timestamp: 1600736652
timesteps_since_restore: 0
training_iteration: 3
trial_id: 1dde1_00007
[2m[36m(pid=1122)[0m [2, 16000] loss: 0.195
[2m[36m(pid=1097)[0m [2, 16000] loss: 0.173
Result for DEFAULT_1dde1_00005:
accuracy: 0.5441
date: 2020-09-22_01-04-19
done: false
experiment_id: 6a1270e5c59840d8acaa81bba30ebb03
experiment_tag: 5_batch_size=8,l1=256,l2=128,lr=0.00036206
hostname: fed1025d76e5
iterations_since_restore: 5
loss: 1.2748901859998703
node_ip: 172.17.0.2
pid: 1108
should_checkpoint: true
time_since_restore: 181.70323157310486
time_this_iter_s: 32.77065992355347
time_total_s: 181.70323157310486
timestamp: 1600736659
timesteps_since_restore: 0
training_iteration: 5
trial_id: 1dde1_00005
== Status ==
Memory usage on this node: 6.3/240.1 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -2.0788925355911254 | Iter 4.000: -1.7045656427025795 | Iter 2.000: -1.686314135748148 | Iter 1.000: -1.9341911580085753
Resources requested: 10/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (5 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.69008 | 0.3427 | 1 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | RUNNING | 172.17.0.2:1090 | 16 | 16 | 128 | 0.0422261 | 2.07889 | 0.2073 | 8 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.27489 | 0.5441 | 5 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.25569 | 0.5556 | 3 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.46692 | 0.464 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1122)[0m [2, 18000] loss: 0.175
[2m[36m(pid=1085)[0m [4, 2000] loss: 1.175
[2m[36m(pid=1097)[0m [2, 18000] loss: 0.152
[2m[36m(pid=1090)[0m [9, 2000] loss: 2.048
[2m[36m(pid=1122)[0m [2, 20000] loss: 0.154
Result for DEFAULT_1dde1_00004:
accuracy: 0.2331
date: 2020-09-22_01-04-29
done: false
experiment_id: 0e63a81fef9141bdb0635e2bd7085e0e
experiment_tag: 4_batch_size=16,l1=16,l2=128,lr=0.042226
hostname: fed1025d76e5
iterations_since_restore: 9
loss: 2.026749889564514
node_ip: 172.17.0.2
pid: 1090
should_checkpoint: true
time_since_restore: 192.57573747634888
time_this_iter_s: 19.721917867660522
time_total_s: 192.57573747634888
timestamp: 1600736669
timesteps_since_restore: 0
training_iteration: 9
trial_id: 1dde1_00004
== Status ==
Memory usage on this node: 6.3/240.1 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -2.0788925355911254 | Iter 4.000: -1.7045656427025795 | Iter 2.000: -1.686314135748148 | Iter 1.000: -1.9341911580085753
Resources requested: 10/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (5 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.69008 | 0.3427 | 1 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | RUNNING | 172.17.0.2:1090 | 16 | 16 | 128 | 0.0422261 | 2.02675 | 0.2331 | 9 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.27489 | 0.5441 | 5 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.25569 | 0.5556 | 3 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.46692 | 0.464 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1085)[0m [4, 4000] loss: 0.593
[2m[36m(pid=1097)[0m [2, 20000] loss: 0.136
[2m[36m(pid=1108)[0m [6, 2000] loss: 1.229
[2m[36m(pid=1085)[0m [4, 6000] loss: 0.394
Result for DEFAULT_1dde1_00001:
accuracy: 0.4297
date: 2020-09-22_01-04-41
done: false
experiment_id: f8daf9cd06ad4a4b9c3df3513effdf31
experiment_tag: 1_batch_size=2,l1=16,l2=4,lr=0.00041942
hostname: fed1025d76e5
iterations_since_restore: 2
loss: 1.5426445162922144
node_ip: 172.17.0.2
pid: 1122
should_checkpoint: true
time_since_restore: 204.10034322738647
time_this_iter_s: 92.75418305397034
time_total_s: 204.10034322738647
timestamp: 1600736681
timesteps_since_restore: 0
training_iteration: 2
trial_id: 1dde1_00001
== Status ==
Memory usage on this node: 6.3/240.1 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -2.0788925355911254 | Iter 4.000: -1.7045656427025795 | Iter 2.000: -1.6115445574760436 | Iter 1.000: -1.9341911580085753
Resources requested: 10/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (5 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.54264 | 0.4297 | 2 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | RUNNING | 172.17.0.2:1090 | 16 | 16 | 128 | 0.0422261 | 2.02675 | 0.2331 | 9 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.27489 | 0.5441 | 5 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.25569 | 0.5556 | 3 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.46692 | 0.464 | 1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1108)[0m [6, 4000] loss: 0.612
Result for DEFAULT_1dde1_00009:
accuracy: 0.4776
date: 2020-09-22_01-04-42
done: false
experiment_id: cfed644d520f435688772253831a176f
experiment_tag: 9_batch_size=2,l1=32,l2=32,lr=0.00043883
hostname: fed1025d76e5
iterations_since_restore: 2
loss: 1.4519525066554546
node_ip: 172.17.0.2
pid: 1097
should_checkpoint: true
time_since_restore: 205.36001992225647
time_this_iter_s: 92.96504259109497
time_total_s: 205.36001992225647
timestamp: 1600736682
timesteps_since_restore: 0
training_iteration: 2
trial_id: 1dde1_00009
[2m[36m(pid=1090)[0m [10, 2000] loss: 2.016
[2m[36m(pid=1085)[0m [4, 8000] loss: 0.304
[2m[36m(pid=1122)[0m [3, 2000] loss: 1.510
Result for DEFAULT_1dde1_00004:
accuracy: 0.1939
date: 2020-09-22_01-04-49
done: true
experiment_id: 0e63a81fef9141bdb0635e2bd7085e0e
experiment_tag: 4_batch_size=16,l1=16,l2=128,lr=0.042226
hostname: fed1025d76e5
iterations_since_restore: 10
loss: 2.030654595375061
node_ip: 172.17.0.2
pid: 1090
should_checkpoint: true
time_since_restore: 212.41506910324097
time_this_iter_s: 19.83933162689209
time_total_s: 212.41506910324097
timestamp: 1600736689
timesteps_since_restore: 0
training_iteration: 10
trial_id: 1dde1_00004
== Status ==
Memory usage on this node: 6.3/240.1 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -2.0788925355911254 | Iter 4.000: -1.7045656427025795 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 10/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (5 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.54264 | 0.4297 | 2 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | RUNNING | 172.17.0.2:1090 | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.27489 | 0.5441 | 5 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.25569 | 0.5556 | 3 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.45195 | 0.4776 | 2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1097)[0m [3, 2000] loss: 1.312
Result for DEFAULT_1dde1_00005:
accuracy: 0.5663
date: 2020-09-22_01-04-51
done: false
experiment_id: 6a1270e5c59840d8acaa81bba30ebb03
experiment_tag: 5_batch_size=8,l1=256,l2=128,lr=0.00036206
hostname: fed1025d76e5
iterations_since_restore: 6
loss: 1.223778771853447
node_ip: 172.17.0.2
pid: 1108
should_checkpoint: true
time_since_restore: 214.51789498329163
time_this_iter_s: 32.81466341018677
time_total_s: 214.51789498329163
timestamp: 1600736691
timesteps_since_restore: 0
training_iteration: 6
trial_id: 1dde1_00005
[2m[36m(pid=1122)[0m [3, 4000] loss: 0.750
[2m[36m(pid=1085)[0m [4, 10000] loss: 0.233
[2m[36m(pid=1097)[0m [3, 4000] loss: 0.655
[2m[36m(pid=1108)[0m [7, 2000] loss: 1.171
Result for DEFAULT_1dde1_00007:
accuracy: 0.5539
date: 2020-09-22_01-05-03
done: false
experiment_id: d130aa97ad4d43d1a015b0a98a769c9b
experiment_tag: 7_batch_size=4,l1=128,l2=32,lr=0.0013999
hostname: fed1025d76e5
iterations_since_restore: 4
loss: 1.267463112605363
node_ip: 172.17.0.2
pid: 1085
should_checkpoint: true
time_since_restore: 226.45621824264526
time_this_iter_s: 51.58772158622742
time_total_s: 226.45621824264526
timestamp: 1600736703
timesteps_since_restore: 0
training_iteration: 4
trial_id: 1dde1_00007
== Status ==
Memory usage on this node: 5.8/240.1 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -2.0788925355911254 | Iter 4.000: -1.3617212006807327 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 8/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.54264 | 0.4297 | 2 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | TERMINATED | | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.22378 | 0.5663 | 6 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.26746 | 0.5539 | 4 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.45195 | 0.4776 | 2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1122)[0m [3, 6000] loss: 0.496
[2m[36m(pid=1097)[0m [3, 6000] loss: 0.441
[2m[36m(pid=1122)[0m [3, 8000] loss: 0.375
[2m[36m(pid=1085)[0m [5, 2000] loss: 1.102
[2m[36m(pid=1108)[0m [7, 4000] loss: 0.578
[2m[36m(pid=1097)[0m [3, 8000] loss: 0.331
[2m[36m(pid=1122)[0m [3, 10000] loss: 0.297
[2m[36m(pid=1085)[0m [5, 4000] loss: 0.554
[2m[36m(pid=1097)[0m [3, 10000] loss: 0.263
Result for DEFAULT_1dde1_00005:
accuracy: 0.569
date: 2020-09-22_01-05-23
done: false
experiment_id: 6a1270e5c59840d8acaa81bba30ebb03
experiment_tag: 5_batch_size=8,l1=256,l2=128,lr=0.00036206
hostname: fed1025d76e5
iterations_since_restore: 7
loss: 1.2081069868803025
node_ip: 172.17.0.2
pid: 1108
should_checkpoint: true
time_since_restore: 246.3141324520111
time_this_iter_s: 31.796237468719482
time_total_s: 246.3141324520111
timestamp: 1600736723
timesteps_since_restore: 0
training_iteration: 7
trial_id: 1dde1_00005
== Status ==
Memory usage on this node: 5.7/240.1 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -2.0788925355911254 | Iter 4.000: -1.3617212006807327 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 8/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.54264 | 0.4297 | 2 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | TERMINATED | | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.20811 | 0.569 | 7 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.26746 | 0.5539 | 4 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.45195 | 0.4776 | 2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1122)[0m [3, 12000] loss: 0.249
[2m[36m(pid=1097)[0m [3, 12000] loss: 0.217
[2m[36m(pid=1085)[0m [5, 6000] loss: 0.377
[2m[36m(pid=1108)[0m [8, 2000] loss: 1.100
[2m[36m(pid=1122)[0m [3, 14000] loss: 0.210
[2m[36m(pid=1097)[0m [3, 14000] loss: 0.183
[2m[36m(pid=1085)[0m [5, 8000] loss: 0.280
[2m[36m(pid=1122)[0m [3, 16000] loss: 0.183
[2m[36m(pid=1097)[0m [3, 16000] loss: 0.159
[2m[36m(pid=1108)[0m [8, 4000] loss: 0.556
[2m[36m(pid=1085)[0m [5, 10000] loss: 0.226
[2m[36m(pid=1122)[0m [3, 18000] loss: 0.165
[2m[36m(pid=1097)[0m [3, 18000] loss: 0.140
Result for DEFAULT_1dde1_00007:
accuracy: 0.5792
date: 2020-09-22_01-05-54
done: false
experiment_id: d130aa97ad4d43d1a015b0a98a769c9b
experiment_tag: 7_batch_size=4,l1=128,l2=32,lr=0.0013999
hostname: fed1025d76e5
iterations_since_restore: 5
loss: 1.2025769818842411
node_ip: 172.17.0.2
pid: 1085
should_checkpoint: true
time_since_restore: 276.8607568740845
time_this_iter_s: 50.40453863143921
time_total_s: 276.8607568740845
timestamp: 1600736754
timesteps_since_restore: 0
training_iteration: 5
trial_id: 1dde1_00007
== Status ==
Memory usage on this node: 5.7/240.1 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -2.0788925355911254 | Iter 4.000: -1.3617212006807327 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 8/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.54264 | 0.4297 | 2 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | TERMINATED | | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.20811 | 0.569 | 7 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.20258 | 0.5792 | 5 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.45195 | 0.4776 | 2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_1dde1_00005:
accuracy: 0.5998
date: 2020-09-22_01-05-55
done: false
experiment_id: 6a1270e5c59840d8acaa81bba30ebb03
experiment_tag: 5_batch_size=8,l1=256,l2=128,lr=0.00036206
hostname: fed1025d76e5
iterations_since_restore: 8
loss: 1.133019412434101
node_ip: 172.17.0.2
pid: 1108
should_checkpoint: true
time_since_restore: 277.9527311325073
time_this_iter_s: 31.638598680496216
time_total_s: 277.9527311325073
timestamp: 1600736755
timesteps_since_restore: 0
training_iteration: 8
trial_id: 1dde1_00005
[2m[36m(pid=1122)[0m [3, 20000] loss: 0.146
[2m[36m(pid=1097)[0m [3, 20000] loss: 0.128
[2m[36m(pid=1085)[0m [6, 2000] loss: 1.014
[2m[36m(pid=1108)[0m [9, 2000] loss: 1.057
Result for DEFAULT_1dde1_00001:
accuracy: 0.4713
date: 2020-09-22_01-06-10
done: false
experiment_id: f8daf9cd06ad4a4b9c3df3513effdf31
experiment_tag: 1_batch_size=2,l1=16,l2=4,lr=0.00041942
hostname: fed1025d76e5
iterations_since_restore: 3
loss: 1.4906232760388405
node_ip: 172.17.0.2
pid: 1122
should_checkpoint: true
time_since_restore: 293.61701011657715
time_this_iter_s: 89.51666688919067
time_total_s: 293.61701011657715
timestamp: 1600736770
timesteps_since_restore: 0
training_iteration: 3
trial_id: 1dde1_00001
== Status ==
Memory usage on this node: 5.8/240.1 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.6059559740126133 | Iter 4.000: -1.3617212006807327 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 8/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.49062 | 0.4713 | 3 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | TERMINATED | | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.13302 | 0.5998 | 8 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.20258 | 0.5792 | 5 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.45195 | 0.4776 | 2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1085)[0m [6, 4000] loss: 0.529
Result for DEFAULT_1dde1_00009:
accuracy: 0.5465
date: 2020-09-22_01-06-12
done: false
experiment_id: cfed644d520f435688772253831a176f
experiment_tag: 9_batch_size=2,l1=32,l2=32,lr=0.00043883
hostname: fed1025d76e5
iterations_since_restore: 3
loss: 1.2607018883639947
node_ip: 172.17.0.2
pid: 1097
should_checkpoint: true
time_since_restore: 295.5224437713623
time_this_iter_s: 90.16242384910583
time_total_s: 295.5224437713623
timestamp: 1600736772
timesteps_since_restore: 0
training_iteration: 3
trial_id: 1dde1_00009
[2m[36m(pid=1108)[0m [9, 4000] loss: 0.527
[2m[36m(pid=1122)[0m [4, 2000] loss: 1.435
[2m[36m(pid=1097)[0m [4, 2000] loss: 1.238
[2m[36m(pid=1085)[0m [6, 6000] loss: 0.357
[2m[36m(pid=1122)[0m [4, 4000] loss: 0.704
Result for DEFAULT_1dde1_00005:
accuracy: 0.603
date: 2020-09-22_01-06-27
done: false
experiment_id: 6a1270e5c59840d8acaa81bba30ebb03
experiment_tag: 5_batch_size=8,l1=256,l2=128,lr=0.00036206
hostname: fed1025d76e5
iterations_since_restore: 9
loss: 1.121426535463333
node_ip: 172.17.0.2
pid: 1108
should_checkpoint: true
time_since_restore: 309.83672976493835
time_this_iter_s: 31.88399863243103
time_total_s: 309.83672976493835
timestamp: 1600736787
timesteps_since_restore: 0
training_iteration: 9
trial_id: 1dde1_00005
== Status ==
Memory usage on this node: 5.7/240.1 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.6059559740126133 | Iter 4.000: -1.3617212006807327 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 8/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.49062 | 0.4713 | 3 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | TERMINATED | | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.12143 | 0.603 | 9 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.20258 | 0.5792 | 5 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.2607 | 0.5465 | 3 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1097)[0m [4, 4000] loss: 0.595
[2m[36m(pid=1085)[0m [6, 8000] loss: 0.268
[2m[36m(pid=1122)[0m [4, 6000] loss: 0.466
[2m[36m(pid=1097)[0m [4, 6000] loss: 0.413
[2m[36m(pid=1108)[0m [10, 2000] loss: 0.999
[2m[36m(pid=1085)[0m [6, 10000] loss: 0.218
[2m[36m(pid=1122)[0m [4, 8000] loss: 0.359
[2m[36m(pid=1097)[0m [4, 8000] loss: 0.301
Result for DEFAULT_1dde1_00007:
accuracy: 0.5726
date: 2020-09-22_01-06-45
done: false
experiment_id: d130aa97ad4d43d1a015b0a98a769c9b
experiment_tag: 7_batch_size=4,l1=128,l2=32,lr=0.0013999
hostname: fed1025d76e5
iterations_since_restore: 6
loss: 1.2504841807976366
node_ip: 172.17.0.2
pid: 1085
should_checkpoint: true
time_since_restore: 327.60362005233765
time_this_iter_s: 50.742863178253174
time_total_s: 327.60362005233765
timestamp: 1600736805
timesteps_since_restore: 0
training_iteration: 6
trial_id: 1dde1_00007
== Status ==
Memory usage on this node: 5.8/240.1 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.6059559740126133 | Iter 4.000: -1.3617212006807327 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 8/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.49062 | 0.4713 | 3 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | TERMINATED | | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.12143 | 0.603 | 9 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.25048 | 0.5726 | 6 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.2607 | 0.5465 | 3 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1108)[0m [10, 4000] loss: 0.502
[2m[36m(pid=1122)[0m [4, 10000] loss: 0.280
[2m[36m(pid=1097)[0m [4, 10000] loss: 0.244
[2m[36m(pid=1085)[0m [7, 2000] loss: 0.989
[2m[36m(pid=1122)[0m [4, 12000] loss: 0.234
Result for DEFAULT_1dde1_00005:
accuracy: 0.6115
date: 2020-09-22_01-06-58
done: true
experiment_id: 6a1270e5c59840d8acaa81bba30ebb03
experiment_tag: 5_batch_size=8,l1=256,l2=128,lr=0.00036206
hostname: fed1025d76e5
iterations_since_restore: 10
loss: 1.1189751214504242
node_ip: 172.17.0.2
pid: 1108
should_checkpoint: true
time_since_restore: 341.492472410202
time_this_iter_s: 31.655742645263672
time_total_s: 341.492472410202
timestamp: 1600736818
timesteps_since_restore: 0
training_iteration: 10
trial_id: 1dde1_00005
== Status ==
Memory usage on this node: 5.7/240.1 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.6059559740126133 | Iter 4.000: -1.3617212006807327 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 8/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.49062 | 0.4713 | 3 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | TERMINATED | | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | RUNNING | 172.17.0.2:1108 | 8 | 256 | 128 | 0.000362061 | 1.11898 | 0.6115 | 10 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.25048 | 0.5726 | 6 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.2607 | 0.5465 | 3 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1097)[0m [4, 12000] loss: 0.203
[2m[36m(pid=1085)[0m [7, 4000] loss: 0.508
[2m[36m(pid=1122)[0m [4, 14000] loss: 0.195
[2m[36m(pid=1097)[0m [4, 14000] loss: 0.176
[2m[36m(pid=1085)[0m [7, 6000] loss: 0.338
[2m[36m(pid=1122)[0m [4, 16000] loss: 0.177
[2m[36m(pid=1097)[0m [4, 16000] loss: 0.154
[2m[36m(pid=1085)[0m [7, 8000] loss: 0.260
[2m[36m(pid=1122)[0m [4, 18000] loss: 0.153
[2m[36m(pid=1097)[0m [4, 18000] loss: 0.130
[2m[36m(pid=1122)[0m [4, 20000] loss: 0.136
[2m[36m(pid=1085)[0m [7, 10000] loss: 0.204
[2m[36m(pid=1097)[0m [4, 20000] loss: 0.119
Result for DEFAULT_1dde1_00007:
accuracy: 0.5953
date: 2020-09-22_01-07-34
done: false
experiment_id: d130aa97ad4d43d1a015b0a98a769c9b
experiment_tag: 7_batch_size=4,l1=128,l2=32,lr=0.0013999
hostname: fed1025d76e5
iterations_since_restore: 7
loss: 1.1804789266631006
node_ip: 172.17.0.2
pid: 1085
should_checkpoint: true
time_since_restore: 376.9931342601776
time_this_iter_s: 49.389514207839966
time_total_s: 376.9931342601776
timestamp: 1600736854
timesteps_since_restore: 0
training_iteration: 7
trial_id: 1dde1_00007
== Status ==
Memory usage on this node: 5.1/240.1 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.6059559740126133 | Iter 4.000: -1.3617212006807327 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 6/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.49062 | 0.4713 | 3 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | TERMINATED | | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | TERMINATED | | 8 | 256 | 128 | 0.000362061 | 1.11898 | 0.6115 | 10 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.18048 | 0.5953 | 7 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.2607 | 0.5465 | 3 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_1dde1_00001:
accuracy: 0.5005
date: 2020-09-22_01-07-38
done: false
experiment_id: f8daf9cd06ad4a4b9c3df3513effdf31
experiment_tag: 1_batch_size=2,l1=16,l2=4,lr=0.00041942
hostname: fed1025d76e5
iterations_since_restore: 4
loss: 1.35337228445746
node_ip: 172.17.0.2
pid: 1122
should_checkpoint: true
time_since_restore: 381.63718152046204
time_this_iter_s: 88.02017140388489
time_total_s: 381.63718152046204
timestamp: 1600736858
timesteps_since_restore: 0
training_iteration: 4
trial_id: 1dde1_00001
Result for DEFAULT_1dde1_00009:
accuracy: 0.5718
date: 2020-09-22_01-07-41
done: false
experiment_id: cfed644d520f435688772253831a176f
experiment_tag: 9_batch_size=2,l1=32,l2=32,lr=0.00043883
hostname: fed1025d76e5
iterations_since_restore: 4
loss: 1.1846606592386961
node_ip: 172.17.0.2
pid: 1097
should_checkpoint: true
time_since_restore: 383.6928722858429
time_this_iter_s: 88.17042851448059
time_total_s: 383.6928722858429
timestamp: 1600736861
timesteps_since_restore: 0
training_iteration: 4
trial_id: 1dde1_00009
== Status ==
Memory usage on this node: 5.1/240.1 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.6059559740126133 | Iter 4.000: -1.35337228445746 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 6/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.35337 | 0.5005 | 4 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | TERMINATED | | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | TERMINATED | | 8 | 256 | 128 | 0.000362061 | 1.11898 | 0.6115 | 10 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.18048 | 0.5953 | 7 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.18466 | 0.5718 | 4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1085)[0m [8, 2000] loss: 0.930
[2m[36m(pid=1122)[0m [5, 2000] loss: 1.349
[2m[36m(pid=1097)[0m [5, 2000] loss: 1.148
[2m[36m(pid=1085)[0m [8, 4000] loss: 0.481
[2m[36m(pid=1122)[0m [5, 4000] loss: 0.656
[2m[36m(pid=1097)[0m [5, 4000] loss: 0.573
[2m[36m(pid=1085)[0m [8, 6000] loss: 0.325
[2m[36m(pid=1122)[0m [5, 6000] loss: 0.442
[2m[36m(pid=1097)[0m [5, 6000] loss: 0.393
[2m[36m(pid=1085)[0m [8, 8000] loss: 0.250
[2m[36m(pid=1122)[0m [5, 8000] loss: 0.339
[2m[36m(pid=1097)[0m [5, 8000] loss: 0.287
[2m[36m(pid=1122)[0m [5, 10000] loss: 0.264
[2m[36m(pid=1085)[0m [8, 10000] loss: 0.200
[2m[36m(pid=1097)[0m [5, 10000] loss: 0.237
Result for DEFAULT_1dde1_00007:
accuracy: 0.6036
date: 2020-09-22_01-08-23
done: false
experiment_id: d130aa97ad4d43d1a015b0a98a769c9b
experiment_tag: 7_batch_size=4,l1=128,l2=32,lr=0.0013999
hostname: fed1025d76e5
iterations_since_restore: 8
loss: 1.1812849724005907
node_ip: 172.17.0.2
pid: 1085
should_checkpoint: true
time_since_restore: 426.14055013656616
time_this_iter_s: 49.14741587638855
time_total_s: 426.14055013656616
timestamp: 1600736903
timesteps_since_restore: 0
training_iteration: 8
trial_id: 1dde1_00007
== Status ==
Memory usage on this node: 5.1/240.1 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.1812849724005907 | Iter 4.000: -1.35337228445746 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 6/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.35337 | 0.5005 | 4 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | TERMINATED | | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | TERMINATED | | 8 | 256 | 128 | 0.000362061 | 1.11898 | 0.6115 | 10 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.18128 | 0.6036 | 8 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.18466 | 0.5718 | 4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1122)[0m [5, 12000] loss: 0.223
[2m[36m(pid=1097)[0m [5, 12000] loss: 0.191
[2m[36m(pid=1122)[0m [5, 14000] loss: 0.185
[2m[36m(pid=1085)[0m [9, 2000] loss: 0.902
[2m[36m(pid=1097)[0m [5, 14000] loss: 0.166
[2m[36m(pid=1122)[0m [5, 16000] loss: 0.165
[2m[36m(pid=1085)[0m [9, 4000] loss: 0.470
[2m[36m(pid=1097)[0m [5, 16000] loss: 0.144
[2m[36m(pid=1122)[0m [5, 18000] loss: 0.150
[2m[36m(pid=1097)[0m [5, 18000] loss: 0.129
[2m[36m(pid=1085)[0m [9, 6000] loss: 0.311
[2m[36m(pid=1122)[0m [5, 20000] loss: 0.129
[2m[36m(pid=1097)[0m [5, 20000] loss: 0.117
[2m[36m(pid=1085)[0m [9, 8000] loss: 0.240
Result for DEFAULT_1dde1_00001:
accuracy: 0.525
date: 2020-09-22_01-09-05
done: false
experiment_id: f8daf9cd06ad4a4b9c3df3513effdf31
experiment_tag: 1_batch_size=2,l1=16,l2=4,lr=0.00041942
hostname: fed1025d76e5
iterations_since_restore: 5
loss: 1.3273668685505167
node_ip: 172.17.0.2
pid: 1122
should_checkpoint: true
time_since_restore: 468.13718366622925
time_this_iter_s: 86.50000214576721
time_total_s: 468.13718366622925
timestamp: 1600736945
timesteps_since_restore: 0
training_iteration: 5
trial_id: 1dde1_00001
== Status ==
Memory usage on this node: 5.1/240.1 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.1812849724005907 | Iter 4.000: -1.35337228445746 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 6/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.32737 | 0.525 | 5 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | TERMINATED | | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | TERMINATED | | 8 | 256 | 128 | 0.000362061 | 1.11898 | 0.6115 | 10 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.18128 | 0.6036 | 8 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.18466 | 0.5718 | 4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1085)[0m [9, 10000] loss: 0.191
Result for DEFAULT_1dde1_00009:
accuracy: 0.5693
date: 2020-09-22_01-09-08
done: false
experiment_id: cfed644d520f435688772253831a176f
experiment_tag: 9_batch_size=2,l1=32,l2=32,lr=0.00043883
hostname: fed1025d76e5
iterations_since_restore: 5
loss: 1.2223929223241286
node_ip: 172.17.0.2
pid: 1097
should_checkpoint: true
time_since_restore: 470.7961926460266
time_this_iter_s: 87.10332036018372
time_total_s: 470.7961926460266
timestamp: 1600736948
timesteps_since_restore: 0
training_iteration: 5
trial_id: 1dde1_00009
[2m[36m(pid=1122)[0m [6, 2000] loss: 1.258
Result for DEFAULT_1dde1_00007:
accuracy: 0.6008
date: 2020-09-22_01-09-13
done: false
experiment_id: d130aa97ad4d43d1a015b0a98a769c9b
experiment_tag: 7_batch_size=4,l1=128,l2=32,lr=0.0013999
hostname: fed1025d76e5
iterations_since_restore: 9
loss: 1.2257308665644377
node_ip: 172.17.0.2
pid: 1085
should_checkpoint: true
time_since_restore: 475.85610842704773
time_this_iter_s: 49.71555829048157
time_total_s: 475.85610842704773
timestamp: 1600736953
timesteps_since_restore: 0
training_iteration: 9
trial_id: 1dde1_00007
== Status ==
Memory usage on this node: 5.1/240.1 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.1812849724005907 | Iter 4.000: -1.35337228445746 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 6/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.32737 | 0.525 | 5 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | TERMINATED | | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | TERMINATED | | 8 | 256 | 128 | 0.000362061 | 1.11898 | 0.6115 | 10 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.22573 | 0.6008 | 9 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.22239 | 0.5693 | 5 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1097)[0m [6, 2000] loss: 1.087
[2m[36m(pid=1122)[0m [6, 4000] loss: 0.644
[2m[36m(pid=1085)[0m [10, 2000] loss: 0.860
[2m[36m(pid=1097)[0m [6, 4000] loss: 0.550
[2m[36m(pid=1122)[0m [6, 6000] loss: 0.426
[2m[36m(pid=1085)[0m [10, 4000] loss: 0.443
[2m[36m(pid=1097)[0m [6, 6000] loss: 0.361
[2m[36m(pid=1122)[0m [6, 8000] loss: 0.321
[2m[36m(pid=1097)[0m [6, 8000] loss: 0.281
[2m[36m(pid=1085)[0m [10, 6000] loss: 0.306
[2m[36m(pid=1122)[0m [6, 10000] loss: 0.257
[2m[36m(pid=1097)[0m [6, 10000] loss: 0.228
[2m[36m(pid=1085)[0m [10, 8000] loss: 0.232
[2m[36m(pid=1122)[0m [6, 12000] loss: 0.222
[2m[36m(pid=1097)[0m [6, 12000] loss: 0.190
[2m[36m(pid=1085)[0m [10, 10000] loss: 0.191
[2m[36m(pid=1122)[0m [6, 14000] loss: 0.183
[2m[36m(pid=1097)[0m [6, 14000] loss: 0.161
Result for DEFAULT_1dde1_00007:
accuracy: 0.6077
date: 2020-09-22_01-10-02
done: true
experiment_id: d130aa97ad4d43d1a015b0a98a769c9b
experiment_tag: 7_batch_size=4,l1=128,l2=32,lr=0.0013999
hostname: fed1025d76e5
iterations_since_restore: 10
loss: 1.2082138083937577
node_ip: 172.17.0.2
pid: 1085
should_checkpoint: true
time_since_restore: 524.955450296402
time_this_iter_s: 49.09934186935425
time_total_s: 524.955450296402
timestamp: 1600737002
timesteps_since_restore: 0
training_iteration: 10
trial_id: 1dde1_00007
== Status ==
Memory usage on this node: 5.1/240.1 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.1812849724005907 | Iter 4.000: -1.35337228445746 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 6/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.32737 | 0.525 | 5 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | TERMINATED | | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | TERMINATED | | 8 | 256 | 128 | 0.000362061 | 1.11898 | 0.6115 | 10 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | RUNNING | 172.17.0.2:1085 | 4 | 128 | 32 | 0.00139994 | 1.20821 | 0.6077 | 10 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.22239 | 0.5693 | 5 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1122)[0m [6, 16000] loss: 0.163
[2m[36m(pid=1097)[0m [6, 16000] loss: 0.139
[2m[36m(pid=1122)[0m [6, 18000] loss: 0.141
[2m[36m(pid=1097)[0m [6, 18000] loss: 0.123
[2m[36m(pid=1122)[0m [6, 20000] loss: 0.125
[2m[36m(pid=1097)[0m [6, 20000] loss: 0.114
Result for DEFAULT_1dde1_00001:
accuracy: 0.5372
date: 2020-09-22_01-10-31
done: false
experiment_id: f8daf9cd06ad4a4b9c3df3513effdf31
experiment_tag: 1_batch_size=2,l1=16,l2=4,lr=0.00041942
hostname: fed1025d76e5
iterations_since_restore: 6
loss: 1.306188284834847
node_ip: 172.17.0.2
pid: 1122
should_checkpoint: true
time_since_restore: 553.9988505840302
time_this_iter_s: 85.8616669178009
time_total_s: 553.9988505840302
timestamp: 1600737031
timesteps_since_restore: 0
training_iteration: 6
trial_id: 1dde1_00001
== Status ==
Memory usage on this node: 4.6/240.1 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.1812849724005907 | Iter 4.000: -1.35337228445746 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 4/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.30619 | 0.5372 | 6 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | TERMINATED | | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | TERMINATED | | 8 | 256 | 128 | 0.000362061 | 1.11898 | 0.6115 | 10 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | TERMINATED | | 4 | 128 | 32 | 0.00139994 | 1.20821 | 0.6077 | 10 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.22239 | 0.5693 | 5 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_1dde1_00009:
accuracy: 0.5829
date: 2020-09-22_01-10-34
done: false
experiment_id: cfed644d520f435688772253831a176f
experiment_tag: 9_batch_size=2,l1=32,l2=32,lr=0.00043883
hostname: fed1025d76e5
iterations_since_restore: 6
loss: 1.1963300617878792
node_ip: 172.17.0.2
pid: 1097
should_checkpoint: true
time_since_restore: 557.4718091487885
time_this_iter_s: 86.67561650276184
time_total_s: 557.4718091487885
timestamp: 1600737034
timesteps_since_restore: 0
training_iteration: 6
trial_id: 1dde1_00009
[2m[36m(pid=1122)[0m [7, 2000] loss: 1.218
[2m[36m(pid=1097)[0m [7, 2000] loss: 1.064
[2m[36m(pid=1122)[0m [7, 4000] loss: 0.608
[2m[36m(pid=1097)[0m [7, 4000] loss: 0.540
[2m[36m(pid=1122)[0m [7, 6000] loss: 0.414
[2m[36m(pid=1097)[0m [7, 6000] loss: 0.365
[2m[36m(pid=1122)[0m [7, 8000] loss: 0.313
[2m[36m(pid=1097)[0m [7, 8000] loss: 0.265
[2m[36m(pid=1122)[0m [7, 10000] loss: 0.254
[2m[36m(pid=1097)[0m [7, 10000] loss: 0.221
[2m[36m(pid=1122)[0m [7, 12000] loss: 0.210
[2m[36m(pid=1097)[0m [7, 12000] loss: 0.180
[2m[36m(pid=1122)[0m [7, 14000] loss: 0.183
[2m[36m(pid=1097)[0m [7, 14000] loss: 0.153
[2m[36m(pid=1122)[0m [7, 16000] loss: 0.159
[2m[36m(pid=1097)[0m [7, 16000] loss: 0.136
[2m[36m(pid=1122)[0m [7, 18000] loss: 0.139
[2m[36m(pid=1097)[0m [7, 18000] loss: 0.123
[2m[36m(pid=1122)[0m [7, 20000] loss: 0.125
[2m[36m(pid=1097)[0m [7, 20000] loss: 0.111
Result for DEFAULT_1dde1_00001:
accuracy: 0.5418
date: 2020-09-22_01-11-56
done: false
experiment_id: f8daf9cd06ad4a4b9c3df3513effdf31
experiment_tag: 1_batch_size=2,l1=16,l2=4,lr=0.00041942
hostname: fed1025d76e5
iterations_since_restore: 7
loss: 1.3043943683009596
node_ip: 172.17.0.2
pid: 1122
should_checkpoint: true
time_since_restore: 638.82790184021
time_this_iter_s: 84.82905125617981
time_total_s: 638.82790184021
timestamp: 1600737116
timesteps_since_restore: 0
training_iteration: 7
trial_id: 1dde1_00001
== Status ==
Memory usage on this node: 4.6/240.1 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.1812849724005907 | Iter 4.000: -1.35337228445746 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 4/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.30439 | 0.5418 | 7 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | TERMINATED | | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | TERMINATED | | 8 | 256 | 128 | 0.000362061 | 1.11898 | 0.6115 | 10 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | TERMINATED | | 4 | 128 | 32 | 0.00139994 | 1.20821 | 0.6077 | 10 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.19633 | 0.5829 | 6 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_1dde1_00009:
accuracy: 0.5828
date: 2020-09-22_01-12-00
done: false
experiment_id: cfed644d520f435688772253831a176f
experiment_tag: 9_batch_size=2,l1=32,l2=32,lr=0.00043883
hostname: fed1025d76e5
iterations_since_restore: 7
loss: 1.1821620257275178
node_ip: 172.17.0.2
pid: 1097
should_checkpoint: true
time_since_restore: 643.4907233715057
time_this_iter_s: 86.01891422271729
time_total_s: 643.4907233715057
timestamp: 1600737120
timesteps_since_restore: 0
training_iteration: 7
trial_id: 1dde1_00009
[2m[36m(pid=1122)[0m [8, 2000] loss: 1.203
[2m[36m(pid=1097)[0m [8, 2000] loss: 1.043
[2m[36m(pid=1122)[0m [8, 4000] loss: 0.625
[2m[36m(pid=1097)[0m [8, 4000] loss: 0.529
[2m[36m(pid=1122)[0m [8, 6000] loss: 0.410
[2m[36m(pid=1097)[0m [8, 6000] loss: 0.359
[2m[36m(pid=1122)[0m [8, 8000] loss: 0.305
[2m[36m(pid=1097)[0m [8, 8000] loss: 0.265
[2m[36m(pid=1122)[0m [8, 10000] loss: 0.247
[2m[36m(pid=1097)[0m [8, 10000] loss: 0.208
[2m[36m(pid=1122)[0m [8, 12000] loss: 0.201
[2m[36m(pid=1097)[0m [8, 12000] loss: 0.182
[2m[36m(pid=1122)[0m [8, 14000] loss: 0.181
[2m[36m(pid=1097)[0m [8, 14000] loss: 0.151
[2m[36m(pid=1122)[0m [8, 16000] loss: 0.153
[2m[36m(pid=1097)[0m [8, 16000] loss: 0.136
[2m[36m(pid=1122)[0m [8, 18000] loss: 0.137
[2m[36m(pid=1097)[0m [8, 18000] loss: 0.117
[2m[36m(pid=1122)[0m [8, 20000] loss: 0.121
[2m[36m(pid=1097)[0m [8, 20000] loss: 0.103
Result for DEFAULT_1dde1_00001:
accuracy: 0.5532
date: 2020-09-22_01-13-21
done: true
experiment_id: f8daf9cd06ad4a4b9c3df3513effdf31
experiment_tag: 1_batch_size=2,l1=16,l2=4,lr=0.00041942
hostname: fed1025d76e5
iterations_since_restore: 8
loss: 1.2664230019509326
node_ip: 172.17.0.2
pid: 1122
should_checkpoint: true
time_since_restore: 723.7728803157806
time_this_iter_s: 84.94497847557068
time_total_s: 723.7728803157806
timestamp: 1600737201
timesteps_since_restore: 0
training_iteration: 8
trial_id: 1dde1_00001
== Status ==
Memory usage on this node: 4.6/240.1 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2238539871757617 | Iter 4.000: -1.35337228445746 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 4/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | RUNNING | 172.17.0.2:1122 | 2 | 16 | 4 | 0.000419419 | 1.26642 | 0.5532 | 8 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | TERMINATED | | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | TERMINATED | | 8 | 256 | 128 | 0.000362061 | 1.11898 | 0.6115 | 10 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | TERMINATED | | 4 | 128 | 32 | 0.00139994 | 1.20821 | 0.6077 | 10 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.18216 | 0.5828 | 7 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
Result for DEFAULT_1dde1_00009:
accuracy: 0.5975
date: 2020-09-22_01-13-26
done: false
experiment_id: cfed644d520f435688772253831a176f
experiment_tag: 9_batch_size=2,l1=32,l2=32,lr=0.00043883
hostname: fed1025d76e5
iterations_since_restore: 8
loss: 1.139856807322579
node_ip: 172.17.0.2
pid: 1097
should_checkpoint: true
time_since_restore: 728.941769361496
time_this_iter_s: 85.45104598999023
time_total_s: 728.941769361496
timestamp: 1600737206
timesteps_since_restore: 0
training_iteration: 8
trial_id: 1dde1_00009
== Status ==
Memory usage on this node: 4.1/240.1 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.1812849724005907 | Iter 4.000: -1.35337228445746 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 2/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | TERMINATED | | 2 | 16 | 4 | 0.000419419 | 1.26642 | 0.5532 | 8 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | TERMINATED | | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | TERMINATED | | 8 | 256 | 128 | 0.000362061 | 1.11898 | 0.6115 | 10 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | TERMINATED | | 4 | 128 | 32 | 0.00139994 | 1.20821 | 0.6077 | 10 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.13986 | 0.5975 | 8 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1097)[0m [9, 2000] loss: 1.029
[2m[36m(pid=1097)[0m [9, 4000] loss: 0.509
[2m[36m(pid=1097)[0m [9, 6000] loss: 0.339
[2m[36m(pid=1097)[0m [9, 8000] loss: 0.257
[2m[36m(pid=1097)[0m [9, 10000] loss: 0.207
[2m[36m(pid=1097)[0m [9, 12000] loss: 0.175
[2m[36m(pid=1097)[0m [9, 14000] loss: 0.150
[2m[36m(pid=1097)[0m [9, 16000] loss: 0.129
[2m[36m(pid=1097)[0m [9, 18000] loss: 0.117
[2m[36m(pid=1097)[0m [9, 20000] loss: 0.109
Result for DEFAULT_1dde1_00009:
accuracy: 0.5924
date: 2020-09-22_01-14-50
done: false
experiment_id: cfed644d520f435688772253831a176f
experiment_tag: 9_batch_size=2,l1=32,l2=32,lr=0.00043883
hostname: fed1025d76e5
iterations_since_restore: 9
loss: 1.1540529461038997
node_ip: 172.17.0.2
pid: 1097
should_checkpoint: true
time_since_restore: 813.1155605316162
time_this_iter_s: 84.17379117012024
time_total_s: 813.1155605316162
timestamp: 1600737290
timesteps_since_restore: 0
training_iteration: 9
trial_id: 1dde1_00009
== Status ==
Memory usage on this node: 4.1/240.1 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.1812849724005907 | Iter 4.000: -1.35337228445746 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 2/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | TERMINATED | | 2 | 16 | 4 | 0.000419419 | 1.26642 | 0.5532 | 8 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | TERMINATED | | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | TERMINATED | | 8 | 256 | 128 | 0.000362061 | 1.11898 | 0.6115 | 10 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | TERMINATED | | 4 | 128 | 32 | 0.00139994 | 1.20821 | 0.6077 | 10 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.15405 | 0.5924 | 9 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
[2m[36m(pid=1097)[0m [10, 2000] loss: 0.984
[2m[36m(pid=1097)[0m [10, 4000] loss: 0.490
[2m[36m(pid=1097)[0m [10, 6000] loss: 0.339
[2m[36m(pid=1097)[0m [10, 8000] loss: 0.264
[2m[36m(pid=1097)[0m [10, 10000] loss: 0.206
[2m[36m(pid=1097)[0m [10, 12000] loss: 0.167
[2m[36m(pid=1097)[0m [10, 14000] loss: 0.150
[2m[36m(pid=1097)[0m [10, 16000] loss: 0.127
[2m[36m(pid=1097)[0m [10, 18000] loss: 0.116
[2m[36m(pid=1097)[0m [10, 20000] loss: 0.107
Result for DEFAULT_1dde1_00009:
accuracy: 0.5724
date: 2020-09-22_01-16-15
done: true
experiment_id: cfed644d520f435688772253831a176f
experiment_tag: 9_batch_size=2,l1=32,l2=32,lr=0.00043883
hostname: fed1025d76e5
iterations_since_restore: 10
loss: 1.220944674178597
node_ip: 172.17.0.2
pid: 1097
should_checkpoint: true
time_since_restore: 897.8286879062653
time_this_iter_s: 84.71312737464905
time_total_s: 897.8286879062653
timestamp: 1600737375
timesteps_since_restore: 0
training_iteration: 10
trial_id: 1dde1_00009
== Status ==
Memory usage on this node: 4.1/240.1 GiB
Using AsyncHyperBand: num_stopped=10
Bracket: Iter 8.000: -1.1812849724005907 | Iter 4.000: -1.35337228445746 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 2/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | TERMINATED | | 2 | 16 | 4 | 0.000419419 | 1.26642 | 0.5532 | 8 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | TERMINATED | | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | TERMINATED | | 8 | 256 | 128 | 0.000362061 | 1.11898 | 0.6115 | 10 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | TERMINATED | | 4 | 128 | 32 | 0.00139994 | 1.20821 | 0.6077 | 10 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | RUNNING | 172.17.0.2:1097 | 2 | 32 | 32 | 0.000438826 | 1.22094 | 0.5724 | 10 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
== Status ==
Memory usage on this node: 4.1/240.1 GiB
Using AsyncHyperBand: num_stopped=10
Bracket: Iter 8.000: -1.1812849724005907 | Iter 4.000: -1.35337228445746 | Iter 2.000: -1.577094536884129 | Iter 1.000: -1.9341911580085753
Resources requested: 0/32 CPUs, 0/2 GPUs, 0.0/157.76 GiB heap, 0.0/49.41 GiB objects
Result logdir: /var/lib/jenkins/ray_results/DEFAULT
Number of trials: 10 (10 TERMINATED)
+---------------------+------------+-------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name | status | loc | batch_size | l1 | l2 | lr | loss | accuracy | training_iteration |
|---------------------+------------+-------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_1dde1_00000 | TERMINATED | | 4 | 128 | 4 | 0.00648265 | 2.02588 | 0.2675 | 1 |
| DEFAULT_1dde1_00001 | TERMINATED | | 2 | 16 | 4 | 0.000419419 | 1.26642 | 0.5532 | 8 |
| DEFAULT_1dde1_00002 | TERMINATED | | 16 | 64 | 128 | 0.0486304 | 2.02543 | 0.2616 | 1 |
| DEFAULT_1dde1_00003 | TERMINATED | | 2 | 32 | 4 | 0.0290972 | 2.32603 | 0.0979 | 1 |
| DEFAULT_1dde1_00004 | TERMINATED | | 16 | 16 | 128 | 0.0422261 | 2.03065 | 0.1939 | 10 |
| DEFAULT_1dde1_00005 | TERMINATED | | 8 | 256 | 128 | 0.000362061 | 1.11898 | 0.6115 | 10 |
| DEFAULT_1dde1_00006 | TERMINATED | | 4 | 256 | 16 | 0.00696066 | 1.76108 | 0.3378 | 2 |
| DEFAULT_1dde1_00007 | TERMINATED | | 4 | 128 | 32 | 0.00139994 | 1.20821 | 0.6077 | 10 |
| DEFAULT_1dde1_00008 | TERMINATED | | 8 | 8 | 128 | 0.000178276 | 2.29409 | 0.1467 | 1 |
| DEFAULT_1dde1_00009 | TERMINATED | | 2 | 32 | 32 | 0.000438826 | 1.22094 | 0.5724 | 10 |
+---------------------+------------+-------+--------------+------+------+-------------+---------+------------+----------------------+
Best trial config: {'l1': 256, 'l2': 128, 'lr': 0.00036206118636519547, 'batch_size': 8}
Best trial final validation loss: 1.1189751214504242
Best trial final validation accuracy: 0.6115
Files already downloaded and verified
Files already downloaded and verified
Best trial test set accuracy: 0.6028
If you run the code, an example output could look like this:
Most trials have been stopped early in order to avoid wasting resources. The best performing trial achieved a validation accuracy of about 58%, which could be confirmed on the test set.
So that’s it! You can now tune the parameters of your PyTorch models.
Total running time of the script: ( 15 minutes 20.830 seconds)
