* Stop all running pods and replication controllers
* Scrape the metrics and check whether they match our expectations
It is worth emphasizing that the main parts of the test are done on full clusters (30 pods per node, 100 nodes) - starting a pod in an empty cluster, even if it has 100 nodes will be much faster.
To measure pod startup latency we are using very simple pods with just a single container running the “gcr.io/google_containers/pause:go” image, which starts and then sleeps forever. The container is guaranteed to be already pre-pulled on nodes (we use it as the so-called pod-infra-container).
##### Performance data
The following table contains percentiles (50th, 90th and 99th) of pod startup time in 100-node clusters which are 10%, 25%, 50% and 100% full.
| | 10%-full |25%-full | 50%-full | 100%-full |
| ------------ | ------------ | ------------ | ------------ | ------------ |
|50th percentile | .90s | 1.08s | 1.33s | 1.94s |
|90th percentile | 1.29s | 1.49s | 1.72s | 2.50s |
| 99th percentile | 1.59s | 1.86s | 2.56s | 4.32s |
As for api-responsiveness, the following graphs present 50th, 90th and 99th percentiles of latencies of API calls grouped by kind of operation and resource type. However, note that this also includes internal system API calls, not just those issued by users (in this case issued by the test itself).
