This post compares all the CPU and GPU instances offered by FloydHub, so that you can choose the right instance type for your training job.
For our benchmark we decided to use the same tests as used by the Tensorflow project. The Tensorflow benchmark process is explained here. AlexNet model was tested using the ImageNet data set for this benchmark. We are planning to add results from other models like InceptionV3 and ResNet-50 soon. You can find the FloydHub project with the benchmark runs here and the Github repo here.
Update (12/14/2017): We have now added benchmark results for InceptionV3, Resnet-50, Resnet-152 and VGG-16.
Sneak preview of results
Here is the quick results from benchmarking before we go in to the details. You can compare the performance of each instance by the number of images it can process per second.
Continue reading to learn more about the setup.
We also followed various performance optimizations recommended in the Tensorflow best practices guide
For our benchmarks we will be using the
NHWC format for the CPU instances and
NCHW for the GPU instances to take advantage of cuDNN. This is again as
recommended by Tensorflow.
Using fused batch norm can result in 12-30% speed up. In our code we enabled this by setting:
bn = tf.contrib.layers.batch_norm(input_layer, fused=True, data_format='NCHW')
You can learn about this in Tensorflow documentation.
Mixed precision training
Another optimization we did is to use Mixed precision training: this techinque lowers the required resources by using lower-precision arithmetic. This decreases the required amount of memory and enables training of larger models or training with larger minibatches. It also shortens the training or inference time because half-precision halves the number of bytes accessed, thus reducing the time spent in memory-limited layers. This is an experimental feature but gives significant speed boost.
Building Tensorflow from source
To optimize the performance it is recommneded to build and install Tensorflow from source. All the FloydHub environments are built from source and optimized for the specific instance type.
For this post we will be comparing 2 CPU instances (CPU, CPU2) and 2 GPU instances (GPU, GPU2):
|Instance||CPU Cores||Memory||GPU Type||GPU Memory|
|GPU||8||64 GB||Nvidia Tesla K-80||12 GB dedicated Memory|
|GPU2||8||64 GB||Nvidia Tesla V100||16 GB dedicated Memory|
We used synthetic data for all the tests. Synthetic data removes disk I/O as a variable and historically the numbers for real data is very close to synthetic data. But we are also planning to run another round of benchmark tests with real data soon.
Synthetic data is randomly generated by using tf.truncated_normal then normalized in range [-1,1] and set to the same shape as the data expected ImageNet.
Sample image generated this way:
Below are the results of testing Alexnet with the synthetic data. As you can see the batch sizes are different - we picked the batch size the utilizes the memory available in each instance as much as possible.
The GPU instances are orders of magnitude faster than CPU instances. And the new GPU2 instance performs about 7x the standard GPU instance.
Full training over Imagenet (about 100 epochs on 1.2 images dataset) takes about 7-8h on GPU2 and more than 2 days on GPU.
Here are the benchmark numbers for InceptionV3.
Full training over Imagenet (about 100 epochs on 1.2 images dataset) takes about 5 days on GPU2 and more than 45 days on GPU.
A full training over Imagenet (about 90 epochs on 1.2 images dataset) takes 3 days on GPU2 and about 24 days on GPU.
A full training over Imagenet (about 100 epochs on 1.2 images dataset) takes 9 days on GPU2 and about 70 days on GPU.
A full training over Imagenet (about 74 epochs on 1.2 images dataset) takes 5 days on GPU2 and about 38 days on GPU.
We are planning to keep this post as a live document. When we run new benchmarks on FloydHub we will update the results here. We are hoping you can always refer to this page when making a decision about which FloydHub instance to choose for your next project.