Envoy, the open source edge and service proxy designed for cloud-native applications, worked with OSTIF and X41 D-Sec to help improve the project’s security posture. The multi-phased engagement, sponsored by Google, focused first on the triaging and closing of bugs, then upon further improving the core fuzzers that continually monitor for security issues using OSS-Fuzz infrastructure at https://github.com/google/oss-fuzz. This engagement is part of a series of security efforts to improve the security and reliability of Envoy. Earlier efforts include the 2018 Audit of Envoy by Cure53 and the Ada Logics fuzzing infrastructure work in 2021.
Phase I of the engagement was designed to help the Envoy Team reduce the number of open bugs identified, reduce noise in the existing fuzzers (so the Envoy team spends less time triaging non-issues), and improve fuzzing coverage where possible. OSTIF sourced four senior security experts from X41 D-sec to complete the engagement.
The result of Phase I was the fixing of 68 security bugs. Furthermore, two critical vulnerabilities were found and proactively fixed. At the beginning of the Phase I engagement, there were 76 Total Fuzz Issues. As fixes and improvements were implemented, new fuzz issues were identified for analysis, allowing for further iteration and fine-tuning.
Figure 1: Total Fuzz Issues vs Fixed at end of Phase I (Sept, 2022)
Phase II was focused on improving the number of executions/second and high signal to noise ratio produced by the fuzzers. Additionally to triaging remaining open high priority fuzzers from Phase I, there was an effort to close another 10 health check related fuzz issues. In order to cover more code faster, while remaining highly efficient and controlling output, X41 was able to analyze bugs from the first phase to determine how the fuzzers were running tests and where they were losing speed and validity. Of Envoy’s original 67 fuzzers, 19 were considered high priority due to testing code paths that are exposed to potential attack vectors; by dataplane traffic in the form of QoDs and bugs that can impact reliability and security of Envoy. Because Envoy is also offered as a service, fuzz testing and hardening the configuration interface is crucial. When fuzzers work on data and configurations simultaneously, there can be a tendency for them to move away from configurations that need deeper testing. This results in less code coverage. Envoy’s code base uses debug assertions, which further slowed down the testing process and often returned noise. To combat and remediate these multiple consequences, X41’s team developed a plan. First, create a way to generate valid configuration files by introspecting the configuration description already augmented by expressions supplying context, and then develop a two-step fuzzing process with the configuration files generated then used for fuzzing.
The team of Markus Vervier, Eric Sesterhenn, Dr. Andre Vehreschild and Dr. Robert Femmer had to develop a way to address the set of points they identified as limiting the fuzzers functionality. They aimed to improve the critical fuzzers that run on HTTP decoders and stream management, specifically HTTP/1.1, HTTP/2, and the HTTP/3 codec interfaces.
Dr. Vehreschild and Dr. Femmer not only developed four new fuzzers, but improved the overall security testing environment of Envoy instrumentally. The created fuzzers targeted HTTP decoders for high throughput focused on primary attack surfaces of Http1 (Balsa and http_parser), Http2 (nghttp2 and oghttp2), and Http3 Quic. To address the issues around configuration files, the team experimented with running configurations repeatedly to explore the data plane deeper before generating another configuration. Configurations found could increase fuzzing speed by a factor of 4 to 20 times. Further work resulted in removing debug assertions and reducing the binary size of fuzzers by a factor of 1.5. By developing ways to create valid and long-running configurations across the configuration and data planes, the performance of fuzzers was increased by a factor of 40.
The graph below strongly suggests that, since March 2022 when Phase I of the engagement began, there has been a downward trend in generation of new fuzz issues due to this collective effort.
Figure 2: New Fuzz Issues since beginning of engagement (March 2022 – June, 2023)
This work took months to complete and was an open discussion-based project that allowed for all members to collaborate, help each other, and discuss the possibilities and testing done.
We sincerely thank Kirtimaan Rajshiva, Adi Peleg, Yan Avlasov, Harvey Tuch, Joshua Marantz, and the entire Envoy Platform Team for funding and guiding this effort along with Markus Vervier, Eric Sesterhenn, Dr. Andre Vehreschild, Dr. Robert Femmer, and Sofie Seuren of X41 D-Sec for their diligent work and insight.
OSTIF is grateful for the opportunity to collaborate and improve security posture for the betterment of FOSS. We would also like to recognize Google, without whose funding this project would not have been possible.
Read the audit report here.
For more information about the work done, see X41’s blog here.
For further information on Envoy, see here.