Future Plans for Tracing Data Collection and Latency Monitoring in Kubernetes

7th chunk of `content/en/blog/_posts/2016-11-00-Visualize-Kubelet-Performance-With-Node-Dashboard.md`

ac18abe804ca0d8250493d3a38fd8d1ac4603f7484e570e50000000100000476

![](https://lh5.googleusercontent.com/51IY9sNPEdtEe-HGz75Q4ggt73ngE0p9gsq6B0m6RDJ13MklYZ3s6xREFhWIxwJt0zFBiY6BvDHwLZ57G9UARfXy1wcAb1DwD48poUrXFHgcRVXUe3tfCoCSpZ477NGTA3A8Njrg) |
| Figure 6. Pod startup latency of build #173. |



In future we plan to use events in Kubernetes which has a fixed log format to collect tracing data more conveniently. Instead of extracting existing log entries, then you can insert your own tracing probes inside Kubelet and obtain the break-down latency of each segment.&nbsp;



You can check the latency between any two probes across different builds in the “TRACING” page, as shown in Figure 7. For example, by selecting "pod\_config\_change" as the start probe, and "pod\_status\_running' as the end probe, it gives the latency variance of Kubelet over continuous builds without status updating overhead. With this feature, developers are able to monitor the performance change of a specific part of code inside Kubelet.   


| ![](https://lh5.googleusercontent.com/nycM01gswI-Z_JxLqHiEjuJZRCg6fwiCiN7HjvKk_iNALN7KihiQB6zdfHDJpf7DLY16qVhIDr6b8qlzOJ9U77fIlBGs-F8eJ3El78pd0wKgNI73PkgEswMFCA5wBLGnYjZqF3PU)

Title: Future Plans for Tracing Data Collection and Latency Monitoring in Kubernetes

Summary

The plan is to utilize Kubernetes events with a fixed log format to collect tracing data more efficiently. Instead of extracting from existing logs, developers can insert custom tracing probes inside Kubelet to obtain a detailed latency breakdown for each segment. The "TRACING" page will allow users to check latency between any two probes across different builds, enabling monitoring of specific code performance changes within Kubelet without status updating overhead.