CI/CD pipeline security: Know the risks and best practices
Rapid release cycles need not compromise the security of an application and supporting infrastructure. Follow these guidelines to ensure security throughout the CI/CD pipeline.
Is your CI/CD pipeline secure? If it's not, your code could be at risk.
The next generation of continuous integration (CI) tools live online, pull code from cloud-based code repositories, gather dependencies and create artifacts, including container images for easy deployment. At the same time, they create a new attack vector by potentially injecting problems into the build pipeline.
CI/CD security adds risk management into the pipeline, including code and artifact scanning, but before you can improve the security of your pipeline, you must know the security risks of modern CI/CD approaches and how to analyze them.
What is CI/CD security?
Continuous integration automatically detects changes in a version control system, then compiles the code and runs checks to create an artifact. Continuous delivery (CD) does the same, but it can push the software to production with a single click. Both CI and CD involve humans in the process. For example, you can perform final end-to-end testing on a change or review code before merging it onto the mainline branch. Continuous deployment, sometimes referred to as CD+, is a complete pipeline that pushes code to production automatically if the code passes all automated checks.
CI/CD security protects the pipeline from malicious actors who could be interested in escalating their privilege, stealing data or disabling systems. CI/CD pipeline security includes code scanning, security reviews, audits, dependency checking and other forms of risk management.
Continuous integration, delivery, deployment and monitoring apply to different sections of the pipeline.
What are the security risks associated with CI/CD pipelines?
It used to be common for CI processes to run on a server that had a copy of all the libraries used in the compile. Code was original; common security risks were SQL injection, buffer overflow and cross-site scripting (XSS). These were mostly identifiable mistakes that administrators caught in code review.
Today, the Open Worldwide Application Security Project's (OWASP) list of the top 10 security risks to web applications includes broken access control, authentication and access issues, and vulnerable or outdated components. These are issues with integration, dependencies, version control and builds running on SaaS -- in other words, problems that affect the pipeline.
CI software often runs in the cloud, where attackers can guess simple username and password logins. A sophisticated attacker could gain commit rights to an open source project, add an innocuous dependency -- such as an advanced math library for Ruby -- then later add a virus or attack to the dependency. New dependencies also create operational risks: A change in one library can break another. For example, package-lock.jsonwas an implementation on Node.js that kept the dependencies static, which means teams did not get security fixes.
Attackers can also create public dependencies with the same name as an internal library. Or, an artifact system might download a new version of Linux to create a container, then add the latest build to it, creating security vulnerabilities. In all these cases, the CI/CD system could either bring in the changes automatically or not at all, which risks failing to download a patch or fix. With continuous deployment, the code might just go automatically to production without any human involvement at all.
Beyond the pipeline, today's programmers are using tools such as Stack Overflow or ChatGPT to find new code quickly, add it, check to see that it works operationally and then commit to version control. Code that is cut and pasted from the internet is expanding in use, but time to review and understand that code is decreasing. This is not exactly a prescription for success.
In summary, the following are some causes of the security risks introduced by CI/CD pipelines:
Weak authentication.
Compromised open source components.
Operational risks from adding new dependencies.
Dependency confusion attacks.
Static dependencies that miss updates.
Importance of CI/CD security testing
Code reviews and audits rely on human expertise, slow things down and are incapable of finding and mitigating zero-day vulnerabilities and other risks. Keeping up with the security bulletins for one component, such as a professionally managed operating system with commercial support, can be exhausting. Human penetration testing can be valuable, but it is prohibitively expensive to run continuously.
One way to mitigate the risk is to have automated security checks inside the CI/CD pipeline. There are a few ways to do that:
Code scanning. Static code analysis (SCA) tools can download signatures for security vulnerabilities and malware, then check all the code in a repository for problems. This can include all the code, including the custom source code, dependencies and libraries.
Container scanning.Artifact scanning tools can go through an entire Linux or Windows container in much the same way SCA tools do. You can run a scan on source code, new incoming containers and final containers that include the new application.
Artifact repositories. Some vendors do the work of scanning, updating and managing a list of dependencies in the cloud. This can include pre-vetted container images and code libraries, so you can download new libraries from a trusted source. The vendor might even provide insurance or guarantees to mitigate their impact should their processes fail.
You can perform incremental scans by running scans on every build, which could be fast enough to run inside an incredibly brief build window. This form of security testing for CI/CD can be categorized as static testing because it is not testing the code as it runs. But dynamic testing can also play a role. It involves running the application and trying various injection attacks. Some tools, such as Zed Attack Proxy (ZAP), can perform automated penetration on web screens. Subject7 combines automated GUI-based checks of the software's functionality with ZAP, running ZAP on every page the automated tool checks.
Security scanning in the pipeline can mitigate a great deal of risk for a fraction of the time and effort.
Without automated security testing, you must release far less often and check before every release, lock down the version of the components you released and risk missing a patch, ignore the risk of malicious code, or perhaps attempt to keep track of bulletins by hand. Those are all alternatives, but none are ideal ways to deal with risk. Security scanning in the pipeline can mitigate a great deal of risk for a fraction of the time and effort.
10 CI/CD security best practices
These best practices can help DevOps teams mitigate the security risks introduced by CI/CD pipelines.
1. Conduct threat modeling
Threat modeling exercises help everyone understand the threat environment that an application will create.
Say a malicious actor was able to get control of your systems. What would they do? For that matter, who would the actor be? By identifying the roles, both internal and external -- customer, contractor, competitor, foreign actor, attention-getter and copyright lawyers, for example -- you can see what threats might cause the most issues.
2. Document the CI/CD pipeline and components
The question of what is downloaded from where and with what access permissions should be an issue for architecture and security -- not a black box managed by each team. Integrated development environment security plugins, such as Microsoft DevSkim, SonarLint and Puma Scan, can help you document and assess the components of the CI/CD pipeline by warning developers of potential vulnerabilities and providing remediation advice. You should also scan to make sure passwords and accounts are stored externally in a secret, instead of locally in code.
3. Perform professional hardening
By going through the OWASP CI/CD cheat sheet, you can spot a dozen vulnerabilities and then check if they are present in the pipeline. This hardening can also include checking every artifact, running port scans and turning off any services that could be potential attack surfaces. Microsoft also has guidance to secure pipelines on Azure.
4. Set up checks and safeguards to commit code
Some tools can perform static code analysis, offering a pass/fail with feedback before code even gets into version control. This initiates automated inspection before a pull request, before code review, before a merge to trunk and before the build process runs.
5. Analyze committed code quickly
Once the code is in, continue to provide code reviews but create policies and metrics to get them done quickly. This might require security training and pair or mob programming to introduce feedback as early in the process as possible. SCA and container scanning can also help here.
6. Test for security just like other functions
Do you know what percentage of testing efforts are dedicated to security? If you could figure out what that percentage is, is that the right amount of investment? By benchmarking peers, doing analysis and working on the budget, you can make a conscious choice instead of pinballing from one crisis to the next.
7. Check open source vulnerabilities
Unless you use an external artifact repository, someone must implement a strategy for checking components for vulnerabilities or checking automatic updates and then regularly follow up on those best practices. Even with an external repository of trusted assets, tracking vulnerability bulletins can offer additional insights.
8. Accept and validate with more security tests
Use security scanning tools and tools that implement functional tests to automatically check against common OWASP vulnerabilities, such as XSS, SQL injection and script injection.
9. Continuously monitor after deployment
Continuous security monitoring tools can watch production to find attempts to crack passwords and traffic coming in from a single source and perform network scans to determine internal IT security risks.
What is the difference between DevSecOps and CI/CD pipelines?
At first blush, DevSecOps and CI/CD pipeline security seem the same. DevSecOps extends DevOps by automating security. The goal of DevSecOps is to rethink security across the lifecycle, from beginning to end. For instance, a DevSecOps team might create a single source from which to create a user with role-based access control based on a job analysis and threat modeling. This streamlines the process of creating programmer accounts and ensures the user has just the right permissions, instead of creating a dozen tickets that go to different people in operations that could make mistakes or be inconsistent.
CI/CD pipeline security focuses on the new risks created by extended pipelines and the work inside of the pipelines that mitigates risks. Threat modeling in CI/CD pipeline security focuses on the threats introduced by the pipeline system, not the high-level risks introduced by all the suppliers, customers and technical staff in the entire organization. For example, if outsourced customer service is underpaid and has access to names, social security numbers and dates of birth, internal identity theft could be a bigger risk than external. Those are issues DevSecOps might address that are not really part of CI/CD security.
CI/CD security tools
Which tool is right for you depends on the environment. AWS, Azure and Google Cloud all provide their own secrets management tools to store credentials and keep them out of code. The best tool will likely align with your platform. Here are a few CI/CD security tools to get started:
AWS Security Services. If your pipeline runs on AWS, it makes sense to check the vendor's tool list first before searching for third-party tools.
Azure Container Registry. A single source for secure, consistent Docker and Open Container Initiative containers but for Linux and Windows.
Coverity. A competitor to SonarQube, Coverity focuses on security scanning.
GitHub Advanced Security. Enables automated, done-for-you security scanning on projects hosted on GitHub and is free for public projects.
Jenkins Security Scan. As of 2024, Jenkins remains the most popular CI/CD tool. In many cases, the Jenkins server is hosted locally. This tool can scan your code and Jenkins plugins for security issues.
SonarQube. The de facto standard for metrics about code performance. It is also capable of performing code scans for security issues.
This isn't an exhaustive list, but rather an overview of the main platforms with a preference for the security features on those platforms, combined with a focus on community support and tools that support heterogeneous environments and the cloud. If your organization doesn't have a CI/CD security program in place, the list should give you somewhere to look. It if does, the next step would be to carefully review the list of risks, see what applies to you and evaluate your mitigation strategy.
Sadly, the security work is never done. Instead, it is more like a competitive card game where the best you can do is a little temporary advantage. If you're doing pipeline security well, the question is "What's next?" If you're not, the question is more likely "Where do we start?"
Matt Heusser is managing director at Excelon Development, where he recruits, trains and conducts software testing and development.
