One of the hot topics in software development right now is “DevSecOps“, which stands for “development, security, and operations”, and loosely-defined means “producing secure applications through an continuous integrated automated build process”. It is often mentioned in conjunction with an organization’s goal to “move security left” in the development process, with the goal of addressing security issues early on in the project plan.
It sounds a bit too good to be true in that it offers agility, scalability, and quality, not to mention security. What development organization wouldn’t want that ? The reality, of course, is not so straightforward; and requires expertise, tools, and controls that present training and integration challenges. The topic of DevSecOps is well-covered elsewhere and beyond the scope of our discussion here, but its popularity speaks to a pressing need: the ability to build secure applications directly from the development pipeline, NOT as the by-product of a security-testing step that generates volumes of vulnerabilities and significant rework. As an industry we are still struggling to achieve this, much as we wrestled with the software quality crisis decades ago.
The focus of this article is strictly on the security aspects of DevSecOps. Specifically, we will elaborate activities that can and should be integrated into the Software Development Lifecycle (SDLC) to produce secure applications.
A Quick Tour of the SDLC
Every organization has its own take on this, so we will define a simple and abridged model that we can use for our discussion. (Readers may choose to skip this section if they are already familiar with the terms in bold below. )
To produce software, an application must, at some level of detail and documentation, carry out the following tasks:
- Define what is to be produced. This activity is often termed “Requirements Gathering” and produces “Requirements” consisting of “Functional Requirements” describing how the software is to behave, and “Non-functional Requirements” which describe all other aspects of what is to be produced.
- Define a solution whose implementation will fulfill the requirements. That is, to elaborate a system of collaborating components that will integrate to form the solution system. This activity is typically referred to as “Design“, and can be specified at a high-level (i.e. low in detail) as a “System Architecture”, or at a low-level (i.e. highly detailed) “Detailed Design”.
- The implementation of the system, sometimes referred to as “Construction“. This involves programmers writing code and integrating libraries and preexisting components to produce (ultimately) a functioning solution.
- Verifying that the newly produced system properly and completely fulfills the Requirements. This activity is called QA (Quality Assurance) or “Testing“, and is ideally governed by a documented Test Plan.
- The installation or distribution of the new application or software. This is sometimes referred to “Deployment“. This represents the delivery and installation of the solution into whatever context it belongs in.
- The ongoing correction of flaws and vulnerabilities discovered in the system, as well as the incorporation of new enhancements, features, and capabilities. This activity is commonly referred to as “Maintenance“, which may or may not be carried out by the development group.
Waterfall vs. Agile Methodologies
Once we agree that the above steps must occur within any software development process, our discussion can turn to how those activities are carried out. It is useful to consider a development methodology as a point on a spectrum between two extremes. On the left we have what is known as the Waterfall approach, which is characterized as rigidly linear one in which the development phases (Requirements Gathering, Design, Construction, etc.) are performed once, in order, with each step producing the deliverable for the next. On the other end of the spectrum, we have the Pure Agile Development approach, in which requirements are emergent and development is carried out incrementally and iteratively by a highly skilled cross-functional team empowered to discover requirements and implement them.
Although a full elaboration of each method and the options between is beyond our scope here, suffice it to say that neither end of the spectrum represents a realistic or practical method. At the risk of over-simplifying, the Waterfall method unrealistically assumes that Requirements can be completely and accurately gathered prior to development, and the Agile method assumes that management will allow the development team to embark on a quest to build a solution to a problem that has no clear solution at that time. Both ends of the spectrum should be considered theoretical models that facilitate dialog. Development organizations typically choose a hybrid approach, somewhere along the spectrum that balances control, speed, and customer satisfaction.
Security Opportunities in the SDLC
Regardless of the development approach being utilized, there are several key points during the process at which security can and must be addressed to produce secure software. The following sections present and discuss these activities, making a powerful argument that they should be part of every development project:
This refers to the internal documentation of organizational policy in regards to security concerns that apply to all software development projects. In other words, Security Standards are security requirements that ALL applications produced by the organization must meet. This work is outside the scope of any development project’s SDLC, and may be managed as one or more distinct projects in itself.
Documenting management expectations in the form of technical standards that all applications must follow can simplify individual project Requirements and encourage the development of shared software infrastructure. Standards to consider should include requirements regarding:
- Password Strength and Storage
- Secrets/Key Management
- Database/Datastore Access
- Cloud-based Components and Services
- Technical and User Documentation
- Third-party Software Security
- Sensitive Information Handling (within applications)
- Exception Requests, Review, and Resolution
Clearly the individual standards must meet the needs of the entire organization while also reflecting best security practices. Having such standards in place can simplify and accelerate application requirements gathering.
This is work that must occur during Requirements Gathering, and that may manifest as its own standalone document under the umbrella of “Non-functional Requirements”.
Whereas Security Standards apply to all applications, Security Requirements are specific to a development effort and serve to document security expectations including:
- Input Validation
- Roles and Permissions
- Metacharacter Sanitization/Neutralization
- API Interfaces and expectations
- Third-party software/component integration
- Error Handling and Failure Conditions
Among the other benefits of having documented Security Requirements are the fact that they can serve as input to QA test plans and be the basis for Security Testing test plans.
Security Design Review
This is an activity that should occur at the end of high-level design. That is, once an architecture or high-level design has been accepted and documented, it should be examined by security experts. (Cue Monty Python: “NOBODY expects the Spanish Inquisition !“)
Simply put, to achieve secure products, every design should be reviewed by security experts before being released into Construction. In addition to ensuring compliance with Security Standards, the review should discuss how Security Requirements are to be realized.
This review should be viewed as gateway in the overall process, or as a “red button” on the assembly line that can force security concerns to be acknowledged and resolved. The obvious value of having security experts “green-light” designs is that it can prevent the commission of vulnerable designs to code; effectively detecting and preventing security flaws early in the kill-chain.
Security Tools For Software Developers
Getting security tools into the hands of developers is a common manifestation of “moving security left”. This entails integrating commercial security scanning tools into the coding process to scan source code and alert developers to potential security vulnerabilities.
Typically this involves provisioning SAST (Static Analysis Security Testing) tools for the developer community and ensuring they have the skills to use them effectively. Depending on the product, scans may be performed on-demand or may occur continuously (i.e. be built into the code editor). Tools in context here include Fortify, SonarQube and AppScan. Note that there is potentially a considerable expense here in terms of software licensing and user training.
Application Security Awareness of Developers
Developers do not introduce security vulnerabilities into their software deliberately, they do so by mistake. In fact, it often turns out that they were unaware of the specific vulnerability or how it manifests.
Furthermore, using tools to inform developers of potential vulnerabilities that they do not understand is not likely to lead to the proper resolution.
The prerequisite knowledge required to effectively use security tools such as those described above is easily underestimated and can jeopardize the benefits of tool integration. The successful integration of security scanning tools into the Construction phase requires that all developers possess a solid understanding of the common security vulnerabilities relevant for their technology, such as the OWASP Top 10 and the CWE Top 25. A solid understanding of security vulnerabilities is necessary to discern and ignore false-positives reported by the scanning tools.
In addition to an awareness of industry-wide”common vulnerabilities”, feedback on findings from the organization’s Application Security Testing efforts can also help sensitize developers to real-world vulnerabilities detected in their products.
In our experience, increased developer knowledge of vulnerabilities typically leads to a vast decline in the presence of those issues in applications.
Application Security Testing
This is an activity that typically occurs during the QA period, and requires a stable code-base. It is carried out by dedicated security testers specializing in application security in a non-production environment. Two types of testing should be performed, and they are explained below. The scope of testing in this stage should include testing against Security Requirements using a documented Security Test Plan as well as less-structured unscripted (ad-hoc) testing. Note that having a documented Security Test Plan will yield additional benefits during the maintenance phase.
Application Security Testing should also be repeated periodically to detect any security flaws that may have escaped earlier detection, as well as new vulnerabilities that may have been introduced during Maintenance.
Automated Security Testing
This describes the use of dynamic application testing tools to detect potential vulnerabilities in the software being tested. Dynamic tools differ from the “static” tools prescribed earlier in the process in that dynamic tools run against an executing instance of the application, whereas static tools glean their insights strictly from source code. Tools in context here include Qualys, Nessus, and Burp Suite.
Manual Security Testing
If SAST scanning is not being performed, no application security testing exercise is complete without an examination of the source code. The code-base should be stable and correspond to what is being security tested. Code Review objectives should include the detection of vulnerabilities as well as the failure to utilize secure coding practices.
Another valuable tool during Application Security Testing is a dependency checker that can examine the source code repository and determine whether any third-party libraries are outdated. The integration of outdated libraries can introduce serious vulnerabilities into products. Such checks must be performed periodically to avoid “failure to patch” scenarios.
It is possible to also integrate this type of tool into the Construction phase in addition to during Application Security Testing.
Deployment Environment Inspection
A key aspect of Application Security Testing is an inspection of the deployment environment. An otherwise highly secure system may become vulnerable through security vulnerabilities in its deployment. This activity is typically carried out by an application security tester who is well-versed in the hosting operating system(s).
Maintaining Secure Software
The Maintenance phase of the SDLC is about fixing, changing, and enhancing the software system, and thus provides a perfect opportunity to introduce new security vulnerabilities. This is especially true if maintenance is being performed by a different group than the original development team.
Preventing the introduction of security vulnerabilities during the Maintenance phase can be addressed by awareness training and the use of SAST tools by maintenance personnel. Detection of newly added vulnerabilities can be addressed through regression security testing using the Security Test Plan, and periodic Application Security Testing of the entire system.
Adjustments for Agile Development
It is relatively easy to envision how the activities described above can be integrated into a Waterfall-style development approach, but it is more challenging when pondering their integration into Agile projects. We offer the following thoughts on adapting the activities to incremental and iterative development:
- Security Standards can likely be documented in advance of development sprints. In this regard, the Agile approach would be the same.
- Requirements Gathering is vexing with Agile projects, as requirements are discovered over time and evolve through development. It may be possible to document Security Requirements in advance of development, in part or in-whole. If emergent or evolving, the documentation must be kept complete and accurate so that Application Security testers can develop a complete and consistent Test Plan.
- The need and approach to Developer Application Security Awareness remains the same with Agile development
- The integration of Security Tools into the Construction phase also remains the same, although they will be used iteratively within each sprint as the system is incrementally developed.
- Application Security Testing should be deferred until the development is complete
- The Maintenance phase would be handled in similar fashion as the project’s Construction phase, depending on whether it too is Agile in nature.
Security Risks Inherent in Agile Development
Due to the potential evolutionary nature of Agile development, some of the biggest challenges in Agile development are producing sufficient and accurate documentation and ensuring sufficient testing is performed. It is not difficult for Agile initiatives to complete development only to find their documentation is incomplete, inaccurate, or both. This can be a challenge for testing and security testing in particular. It is also very easy to overlook security requirements during Requirements discovery, as the focus is on what the customer needs and what it looks like, NOT on security features.
These risks need be forefront in the Project Manager’s (PM) mind in order to avoid them as development progresses.
The Cost of Building Secure Software
It may have already occurred to you that in the quest to build more secure software solutions, we are advocating additional steps, and therefore additional effort, and therefore additional cost to the development process.
Indeed we are.
Just as quality does not come without effort or cost, neither does security. This insight can be taken as a project planning step in itself: estimate the additional time and cost of security and make sure your project plan accounts for that time in the project schedule and cost in the budget.
Summary and Conclusion
We have identified numerous critical activities that can be integrated into the SDLC that will significantly and positively impact the security of the resulting software. You may be doing some of them already. We urge you to broaden your thinking and consider how all the activities might be integrated into your specific development process.
About Affinity IT Security
Affinity IT Security Services is a NJ based cybersecurity consulting firm that has been helping clients secure their networks and applications since 2009. Contact us to learn more about how we can help you build new secure applications that security test existing ones.