Open Source Technology Improvement Fund (OSTIF) is beyond proud to announce the completion of our 50th security audit. Since 2015, the nonprofit organization has worked to provide actualized security support to open source projects in a way that is transparent, public, and impactful. We work with open source projects that are underfunded, undersupported, and in need of security knowledge and resources to bring them customized high-quality security audits and engagements. We value transparency and are highly effective and professional in our work; our engagements’ data backs this up. We have found and resolved in our 50 plus engagements over 130 CVEs or high/critical vulnerabilities for just under a cumulative $3 million dollars of funding. Open source security changes happen when we apply security experience to projects effectively- and not only are we executing at the highest level of the industry, we’re doing it for significantly lower costs than other programs.
This milestone took years of hard work, connections, and convincing funders that OSTIF security audits were of economic value and intrinsically beneficial to the health and long-term security goals of projects. It took learning by experience how to best initiate, manage, and complete audits to bring us to the scale of execution we exist in today, which allows us to manage and perform quality engagements consistently and on time. Our colleagues have graced us with incredible work ethic, commitment, and belief in our mission. Our funders have multiplied as we proved ourselves worthy of funding audit after successful audit. Without these collaborations, we could not have reached 10 audits, much less 50.
The following body of this paper will discuss security audits on a higher level by detailing the types of vulnerabilities OSTIF audits have found, lessons learned about the audit process, common auditing mistakes, and the foundation OSTIF has built for our next audit milestone.
The Top 3 Types of Vulnerabilities Found and Fixed
In security, there are hundreds of types of vulnerabilities that can be anything from informational to critical in their potential security impact on a project. These vulnerabilities can be found in source code, dependent projects, installation scripts, etc.- like buried treasure, you have to be in the right place to find what you’re looking for. Locating vulnerabilities is the first step to fixing or resolving them, which hardens the security of projects and preserves their health as well as protecting user security. Audits help projects not only find and resolve vulnerabilities, but also provide long-term benefits to help maintainers maintain security health. Below are a few examples of the most common vulnerabilities OSTIF has found and fixed in our audits.
1. Denial of Service (DoS)
This exploit is a result of an attempt to flood the project with an overwhelming amount of requests, so that valid requests cannot be completed in a timely manner. The threat actor wants to interrupt services for users/clients, often to take advantage of the project when it is exponentially stressed by the amount of code it’s attempting to handle. Examples include the buffer overflow attack, distributed denial of service (DDoS), an application layer attack, and more derivatives of this style such as a SYN flood. Typically this exploit is found in internet facing projects or hardware, which can be targeted due to their defaulting connection to the Internet of Things. This way, the attack can take place from anywhere in the world, to anything attached to the Internet. DoS style attacks can be used against everything from an open source project, to your home’s digital systems, to Fortune 500 companies. Famous examples include the Mirai malware or Google Cloud in 2017, the largest ever recorded DDoS attack.
To avoid being targeted for a DoS attack, use tools to protect exposed traffic, like cloud based or hardware services. Change passwords on hardware or applications from the default setting, and learn the signs that your appliance may be under a DoS attack.
| # of OSTIF CVEs | Identity | Description |
| 14 | CWE-400 | Uncontrolled Resource Consumption |
2. Cross-Site Scripting/OS Command Injection
Cross-site scripting is when a threat actor introduces malicious executable scripts into the code of an application, which then can be initiated by an unknowing user opening a link and starting the script. Prevention of this vulnerability can be done by performing validation of all input originating from untrusted sources. The more checks in place to validate, the better. Cross-site scripting can result in a variety of outcomes based on the threat actor’s end goals, and is similar to the vulnerability OS command injection in that way. OS command injection allows a threat actor to insert themselves in an operating system and thus begin compromising the integrity of the application from its regulating body. Simply put, to avert this vulnerability, either avoid any communication between the operating system and application-layer code or provide thorough validation processes for calling out to OS commands by the user.
Data Validation Should unvalidated data be allowed, there are multiple attacks that can occur (SQL Injection or Cross-site scripting are just two pertinent examples). To prevent this, all data input needs to be validated using rules. That way, all code is safe to process and to communicate with other components. Projects need to have layers of validation and trust checks to guarantee only trusted input. When this exploit is used, it’s due to a lack of processes programmed in the project to validate, so prevention should be based around identifying and fixing the lack of necessary checks.
| # of OSTIF CVEs | Identity | Description |
| 2 | CWE-78 | Improper Neutralization of Special Elements used in an OS Command |
| 2 | CWE-79 | Improper Neutralization of Input During Web Page Generation |
| 5 | CWE-20 | Improper Input Validation |
| 6 | CWE-770 | Allocation of Resources Without Limits or Throttling |
3. Authorization
Not to be confused with authentication vulnerabilities, authorization is the right to certain privileges or the act of granting access to privileges. Privileges can be given to users, or a program, or it can be a process that can only be performed with certain authorizations. Should a threat actor be able to access otherwise off-limits resources, they could quickly escalate to a devastating degree. Thwarting this kind of vulnerability can be overwhelming work, making sure every possible function of a project is only able to be utilized or performed after authorization. Prevention can be done with secure-by-default program design, reviewing authentication and authorization checks for faults, exercising caution when incorporating third-party dependencies, and performing routine pentesting, fuzzing, and security audits across the whole project to ensure consistency and compliance.
Deserialization of Untrusted Data Continually listed over 5 years as one of the top 25 Most Dangerous Software Weaknesses, this vulnerability starts with malformed or untrusted data which can be manipulated by threat actors for an attack when deserialized. Functions that assume information and code given to them is trustworthy could be exploited to undermine application logic, deny service, or affect code execution. The security impact of data considered safe from tampering being exploited is clear, but its prevention is far from clear-cut. A possible solution is to sanitize inputs from vulnerable or outside sources, more specifically, by implementing a unique or uncommon method of deserialization that can control the exposure of sensitive data.
| # of OSTIF CVEs | Identity | Description |
| 2 | CWE-502 | Deserialization of Untrusted Data |
| 2 | CWE-266 | Privilege Escalation |
Conclusion
OSTIF has 26 different kinds of CWEs that have been identified in the process of auditing over 50 open source projects. The select few above are pertinent examples of what kind of vulnerabilities security audits uncover and resolve to provide short and long term benefits to projects. While an audit can put the approximate cost of a CVE in the tens of thousands in some cases, alternatively, companies pay hundreds of millions of dollars when CVEs are exploited by threat actors. With audits, findings aren’t just reported- they’re uncovered, observed, researched, and then can be resolved with recommended fixes and further hardening advice.
The Top 5 Lessons Learned
Over the past eight years of OSTIF’s work and over 50 completed audits, we’ve come to an understanding of how to organize and execute successful security engagements. We’ve been told before that we have a “secret sauce”- well, if that is the case, the recipe is simple. Below we expand upon lessons learned working in cybersecurity audits, and how they have informed our work processes to positively impact security for open source projects.
1. Importance of Clarity
When organizing and running audits, there has to be an emphasis on transparency even when individuals or groups are working on separate or independent parts of an audit. Open and consistent lines of communication ensures that all parties are on the same page, eliminating surprises and potential misunderstandings before they can snowball into bigger issues. More communication between teams leads to better, more successful audit outcomes. Project maintainers can clarify findings that security teams could otherwise not have context for, which in turn allows for more accurate and detailed vulnerability disclosures and fixes. Public documentation can sometimes be wrong or outdated. By clarifying any and all concerns, questions, and comments as they arise in the course of an audit, the final results are more concise, accurate, and whole. Collaboration and transparency from all sides leads to better audit results. OSTIF prides ourselves on prioritizing communication in our work on every level.
2. Success correlates with Scoping
By choosing the correct amount of work or focus for an audit, this allows for audits to be tailored to projects’ needs. With millions of projects, there are differences from minute to gargantuan. Scoping an audit allows for a project’s needs to be tailored, customized, and responded to accurately. On the back end, findings by a correctly scoped audit are more feasibly fixed, resulting in better security and code health. Projects are huge, sometimes millions of lines of code and that does not include dependent libraries or repositories that should be audited as well for vulnerabilities that can affect the connected projects. Taking into account the financial, time, and labor limitations of an audit to inform what work can and should be prioritized is critical for an audit’s overall ease and success. Accurate and skillful scoping is something that OSTIF can offer every project it works with.
3. Interest & Collaboration
It cannot be understated that the audit team has to want to work on the project, and be interested in the work being done. Invested engineers ask more questions, perform more work, chase after leads that might be left alone by less intrigued parties. In short, they want to engage with the project, not just locate flaws. And as a result, the final work and report are often more in depth, with more issues being addressed, and final recommendations are fuller and wide-ranging. This is also a result of bringing in specialists to work on what they do best- you get high quality work from the people who are working in this exact specialty full time on projects. Our modus operandi is to match projects needing audits with teams who not only perform the best work possible, but want to work with these projects to harden their security for the better of all users.
4. Balance of Testing Types
It is important to know how to use security tooling, in what forms and when. Automated testing is important, and allows for more code to be covered faster, but it is not a holistic replacement for human work. Tooling allows for more coverage, directions and points of interest to be identified by the humans working the audit. Long term, the health of code can be improved by running tooling to identify issues before they become larger concerns. Short term, tooling can help projects cover more ground, resolve or find issues, and harden security posture overall. However, the biggest cyberattacks in the world get researched and resolved by human oversight for the same reasons that human labor cost and hours are necessary- they are able to understand context in a way that computers (and AI) cannot. In our experience, utilizing and balancing security audit work between tooling and human resources is pivotal to maximizing coverage and findings.
5. Resolution is Pivotal
Resolving security audit work is delicate and handles confidential, sensitive security information. Knowing how to direct the audit to a finished product that has deliverables and results in all parties being satisfied is not an easy job. For funders and projects, the resolution is the meat of the project, the quantitative findings that are the vulnerabilities and future security concerns of their projects. For the audit team, this is the fruit of their hard work, and they want to know that their findings and recommendations are taken seriously and handled appropriately. If handled incorrectly, this part of auditing can leave the teams at odds, wanting the opposite of the other organization. Being able to guide security audits to a conclusion that is transparent, public, and satisfying for all invested parties is what separates OSTIF from the rest.
Conclusion
These five lessons were learned the hard way, through trial and error as OSTIF built itself up as a director for security engagements. Led by two industry outsiders who saw a need for this work, OSTIF’s methods pull from financial and IT auditing, the hospitality industry, and privacy engineering to create a process that is manicured from start to finish. To achieve more than 50 engagements took long hours, risks, and dedication to our mission. We wouldn’t change a thing- it taught us what we’re capable of.
The Top 4 Common Auditing Mistakes- and how to Avoid Them
This section focuses on what we’ve learned has a negative impact on security audits. Reaching this milestone was an effort of trial and error over eight years, as our team worked with a variety of projects, audit teams, funders, and communities throughout the process. The more audits we directed, the more we learned where the pinch points were, what synchronization was necessary as well as when and how to shepherd audits to a successful end. The following mistakes were learned the hard way- but they can be circumvented with a little know-how.
1. Focus on the Bid, not the Bidder
Picking the lowest cost or highest cost team to do your work is not how to best judge their work ethic or results. With the many teams now in the space, projects or foundations should look at what their security needs are and source a team that can do the best work based on the individual needs of a project. Audits are not one size fits all. Projects have varying sizes, concerns, levels of security and complexity. Taking the time to determine the best team for the job will pay off long term, as they can become your team in perpetuity or direct the future security and health of the project. Furthermore, undetailed or formulaic audit proposals can be reflective of the level of priority and investment a potential audit team has for the engagement. Well researched, comprehensive proposals leave nothing to the imagination for any parties involved and demonstrate a desire to be involved with the project.
2. Lack of Communication
When outsourcing audits, there can be a tendency to slide into no communication over days, weeks, even months as audits or fixes are in process. There’s many opportunities for a lack of communication between players- as funding or a project is secured, when a Statement of Work is created, as the audit team performs initial review, and as maintainers work on fixes for example. This assumption that “everything is fine if we aren’t hearing anything otherwise” results in misunderstandings, missed opportunities, and potentially missed findings. While it can feel like bothering people, especially when contacting unpaid maintainers, communication of the status quo even if it is slow moving, allows for open and honest dialogue. When scoping, reviewing, or testing, audit teams should feel empowered to ask for elaboration or questions that aid in creating efficiency of their work. Ongoing communication decreases the “dump” of findings or questions at the end, and increases alignment of values, needs, and outcomes for an engagement. Communication is what makes an engagement not only feel successful but be impactful, and the lack thereof is what dooms an audit to be less valuable than what it could be.
3. Only Using Automated Tooling
Automated tooling is just that- a tool. It works best when wielded by someone who knows how to use it and understand its feedback. Like a GPS for a human car, assuming that the tool knows how to drive without a human conductor can result in easily avoided mistakes becoming much more serious and harder to resolve. Tooling works best to process large amounts of code at a fast pace to help audit teams cover more code during the audit, and then as perpetual follow-up, providing constant vigilance and results to maintainers to direct them to new bugs or vulnerabilities. The internal balance of an audit’s tooling and manual review is pivotal to efficiently and accurately reviewing as much code as the scope will allow. Assuming that an audit or all security work can be replaced by a failable tool will result in missed findings and more work on the back end to resolve issues.
4. Proper Scoping
Some projects are straightforward to help and easy to work on in their entirety. Others- not so much. With millions of lines of code and dozens of third-party dependencies, it’s not possible to perform thorough work on a limited budget for some engagements. Trying to review too much and perform more than what’s possible due to time and financial constraints will lead to weak results that don’t improve overall performance or individual aspects of the project. Based on a project’s size, functions, and security needs, an ideal audit will be able to perform a thorough review and testing based on the limits of a well thought-out scope. Take the time to discuss with all parties involved- what should be prioritized? Where are the areas of concern, and is any code tested or reviewed consistently? By setting healthy and realistic boundaries for a security engagement, the results are more likely to increase a project’s security health in key areas and overall.
Conclusion
Third party audits can be complex, delicate, sensitive- and that’s just the people involved, not to mention the codebase. Each audit is an opportunity to learn from multiple sources, to grow into a better firm and to take advantage of the access to world-class sources. The process to choose the right audit team, communicating frequently, scope accurately, and investing in human and automated security work will result in audits that pay off in security and results, with time and money saved.
The Next 50 Audits
This milestone of 50 audits is something to celebrate; but is only the beginning of what can be done for open source security with security audits, work engagements, and applied security resources. OSTIF completed 60% of our 50 audits in just 2023 alone through increased funding, bundled projects, and scaling our staff. We have healthy relationships with even more collaborators, including financial backing from some of the biggest names in the tech and open source community. Whatever kind of security an organization is looking to fund, we can provide, and whatever needs a project has, we can fulfill. Our engagements are holistic, impactful, and custom. We plan on keeping them that way. Explained below is how OSTIF has reached the level of success we have, and what we will depend on to take us even further.
Our Resources
Whatever individual needs a project has, we can meet. Our security teams are diverse in knowledge and experience and through quantitative evaluation and long-standing professional relationships, we pair projects with the optimal team for success. We have relationships with teams that specialize in many different languages, functions, fuzzers, testing types and cryptography. Our Advisory Board is composed of professionals with years of open source experience who believe in our mission and the work we can do. When there’s missing funding, resources or fuzzers- we consider it within our mission scope to supply every possible measure we possibly can to our projects.
Our Skills
Our experience informs how we do what we do. We are successful where many have tried in house and failed. You can’t get what we do from anyone else- security project management, from tip to tail, cultivated for projects by those who understand open source intimately. Our staff has a variety of specializations, education, background of experience that directs and influences the success of the work we undertake. One of our best skills is organizing and determining the skills of others- matching needs with capabilities and projects with teams.
Our Size
As the workload has grown, so has our employee and contractor base, meaning we can specialize and work on individual, different internal roles. This gives us more time, flexibility, access, and accountability to our projects and audit teams. We are not affiliated with our funders beyond the projects they sponsor- we are a non-profit organization whose responsibility is to the projects that support our digital infrastructure, not to C-suite interests. We worked hard over eight years to build our community and sponsors from the ground up, and in return we’re rewarded with the opportunities to do what many have found difficult, nigh impossible.
Our Priorities
We work on FOSS minded, individually focused audits to provide results that go back to the community and funders in spades. We work with projects that want our help, paired with teams that are interested in the project to produce fruitful and informational reports. These reports, now numbering over 50, are what drive our organization and mission forward as we gather information and findings to contribute to the health of open source security. Our long term goal has never changed from our mission. We believe that open source security is not a short term initiative or something to finance with leftover funding. It’s an ongoing, perpetual process of engagements performed over and over to facilitate the hardening of security and therefore the user experience.
Conclusion
What would OSTIF do with unlimited resources, what’s our ultimate goal with this foundation? Ideally, we’d set up a program to run audits continuously, on a broad risk-based scale, across the open source project community. While this program is currently a dream, we are solid and steadfast in achieving our goal of scaling- just in the time it took to draft and publish this article about completing 50 audits, we reached 60.
This article is not just about the number of audits (though 50 is a big one!) or findings (of which we have lots!)- it’s about the projects we can fix, improve upon, and what they teach us. Our goalposts keep moving forward, and we keep running even faster towards them. We are excited to see where we are going as an organization, and we hope you’re just as invested in open source security as we are.
While we celebrate this milestone, we refuse to rest on our laurels and look forward to the next projects we can support with security work. We continue to scale up our offerings and audits per month, to prioritize our mission of supporting security efforts in open source projects, increasing our reach and abilities to continue the work of hardening the infrastructure our society relies upon. OSTIF continues to optimize and execute packages for some of the biggest players to the smallest projects in the tech industry, with plans to take on more in 2024 and beyond.
References
The Art of Software Security Assessment- Identifying and Preventing Software Vulnerabilities. Mark Dowd, John McDonald, Justin Schuh. 2006. https://github.com/rng70/Hacking-Resources/blob/master/Exploiting%20Books/The%20Art%20of%20Software%20Security%20Assessment%20-%20Identifying%20and%20Preventing%20Software%20Vulnerabilities.pdf
Common Weakness Enumeration https://cwe.mitre.org/index.html
OSTIF Audit Reports https://github.com/ostif-org/OSTIF/blob/main/Completed-Engagements.md
OWASP Cheat Sheet Series Project. Lead by Jim Manico, Jakub Maćkowski. 2023. https://cheatsheetseries.owasp.org/