Tooling

In order to loadtest there are a number of useful tools that allow us to monitor and cross check the results.

ApacheBench

https://httpd.apache.org/docs/2.2/programs/ab.html

A simple load test tool for http requests that allows us benchmark throughput. It is reasonably accurate but suffers from a few technical glitches.

Caveats

Each instance of ab will take 100% of a cpu. It is not multithreaded and nor does it share a cpu effectively. It is in essence a big spin loop written in C. If you are running multiple instances on each machine you must have enough cpus to dedicate to the instances.
It can handle up to around 10,000 sockets in a single instance (Although has some issues when you start to get to this num)
It appears to suffer from long stalls for a few packets at high volume and its max packet timings are EXTREMELY SUSPECT under load. (see the difference between read/write roundtrip stats for the server and the ab stats in the load test results).

WRK

A more modern loadtest tool that support multi-core machines. This tool in our testing is far more preferable than ab for testing and does not suffer from many of its downsides. However, there are still some small issues with pre-opening of sockets - although in practice these are only really an issue > 50k connections.

Most of the testing will be performed with this tool.

iftop

iftop is a top equivalent for network traffic and is an good way of monitoring network usage to verify the other tooling numbers and to see what traffic levels as an aggregate you are actually using. It is especially useful for providing baselines as it is able to monitor the loopback address.

Installing

it is only available in epel for redhat so to install on an ec2 instance you must first set up the epel repo.

$ curl -O https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

$ sudo yum install ./epel-release-latest-7.noarch.rpm

Then install iftop

$ sudo yum install iftop

running

You can run to see an interactive view of the output using a simple

$ sudo iftop

or output to a file using something like

$ sudo iftop -b -t > iftop.log

Note: iftop appears to only be accurate to just 1Gb/s. It does have some overhead on the network stack and more importantly it appears to wrap its counters beyond 1Gb making it useless for values beyond this.

Instead we have created a small bash script that parses and prints out the differences using the numbers in /proc/net/dev. This allows us to monitor the network without limitation or much overhead.

See scripts/monitor_interface.sh for details.

TCPDUMP

After installing the epel repo as above install tcpdump (optional)

$ sudo yum install tcpdump

Wireshark

Version 1.10.

Settings

Accurate timings require the following changes to wireshark’s default:

Ensure TCP Contol Protocol -> Protocol Preferences -> allow dissector to reassemble TCP Streams is OFF
Add the target http port of the server if it is non standard to the HTTP Protocol Preferences otherwise http.time will not be available.

Enhanced networking

In order to use the enhanced networking (otherwise you are stuck with generic simple network drivers) you need to install the ixgbevf drivers. This only works on machines that support Enhanced networking.

Redhat 7.1 comes with only version 2.12.1 of the driver which is below the minimum recommended by amazon. Amazon recommend a version which does not work with 7.1 (which is a bit of a pain).

In order to upgrade you must update.

the below instructions are a detail of how to upgrade to a version of the driver which is above the one specified by (Amazon 2.14)

$ sudo yum install wget

$ sudo rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org


$ wget  http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm

$ sudo yum install kmod-ixgbevf

then check by:

[ec2-user@ip-10-2-0-157 ~]$ modinfo ixgbevf
filename:       
      /lib/modules/3.10.0-229.el7.x86_64/weak-updates/ixgbevf/ixgbevf.ko
version:        2.16.1
license:        GPL
description:    Intel(R) 10 Gigabit Virtual Function Network Driver
author:         Intel Corporation, <linux.nics@intel.com>
srcversion:     3B690FE23A02C25EF74012F
alias:          pci:v00008086d00001515sv*sd*bc*sc*i*
alias:          pci:v00008086d000010EDsv*sd*bc*sc*i*
depends:        
vermagic:       3.10.0-123.el7.x86_64 SMP mod_unload modversions 
signer:         The ELRepo Project (http://elrepo.org): ELRepo.org Secure Boot Key
sig_key:        F3:65:AD:34:81:A7:B2:0E:34:27:B6:1B:2A:26:63:5B:83:FE:42:7B
sig_hashalgo:   sha256
parm:           InterruptThrottleRate:Maximum interrupts per second, per vector,
               (956-488281, 0=off, 1=dynamic), default 1 (array of int)

HOWEVER THIS DOES NOT WORK AS INSTANCES DO NOT WORK WITH 2.16 AND ARE UNRESPONSIVE AFTER ENABLING USE AMAZON_LINUX AMI AS THIS COMES WITH 2.14 AND IS ENABED BY DEFAULT

You may have more luck with other/later Linux versions but as a result of this we ended up using AMAZON_LINUX as this came with the correct drivers.

AWS Testing