Development Security Bug Bash

Development Security Bug Bash

Reecetech has been developing software for over 25 years, and web applications for almost half that.

As of 2021 we now have over 1000 active microservices running in over 7500 Kubernetes pods.

Measuring and addressing technical debt has become one of the hardest challenges we face.

With competing priorities, ever changing products and a growing number of microservices, Reecetech recently found that the number of potential security vulnerabilities was growing faster than could be remediated by the team responsible for maintenance, including:

Bug Bash

While a long term solution is being developed, developers were given a dedicated week and tasked with swarming to address as many vulnerabilities as possible.

Each software project (repository) had a card/issue created and populated with the list of CVEs which were extracted from Grype.

A points system was created that scaled with the varying CVE counts and ratings, points were tracked on a dashboard.

scoreboard

Developers self organised into teams of six, and each had one dedicated Platform Engineer assigned.

Starting with the cards that had the highest number of Critical and High rated CVEs each engineer would:

  1. Assign a card to themselves.
  2. Check out the code.
  3. Update the Docker base image.
  4. Run a build which triggers a Grype scan and a number of internal audits to test for vulnerabilities and compliance.
  5. Update any dependencies or framework (Springboot/Django) if required.
  6. Gain peer review.
  7. Release the updated application through the environments.
  8. Mark the card as done, points were awarded to their team.
  9. Repeat.

Results

CVEs resolved (approximate):

CVEs resolved

Retro

What worked well?

What could have been better?

Moving forward

Shifting Tooling Left

We are working on new ways to empower and influence developers to shift left with security.

While our current security tooling has mostly sufficient coverage, it has slow feedback loops where developers need to push their code to CI/CD and wait for security tests to run before they know if there are any vulnerabilities.

While we want to keep security testing in CI/CD, we want to empower developers to test their code before it is pushed.

To enable this we are looking to move from a custom Grype wrapper, to Snyk which provides Docker image, open source dependency scanning, and SAST.

Application Ownership

Each application has been updated with a MAINTAINER label within the Dockerfile, CI checks that this label exists and is set to a valid development team.

Defining SLOs and Measuring SLIs

We want to consider defining SLOs and SLIs for the applications we develop or deploy to define what good looks like and ensure alignment between the business and engineering.

Without Service Level Indicators or Service Level Objectives defined - how do we know if it’s acceptable to have 1 or 100 critical vulnerabilities in a given application?

Increasing Build Frequency

A number of applications had not been built for over 3 years making the effort required to resolve security issues significant.

We want to look into automating builds on all developed software that has not been built for a given number of days and alerting the owners if the build fails.

Enabling Renovate Auto-Remediation

Renovate is currently used to create automated PRs for vulnerabilities in our applications.

We have begun taking this to the next level and enabling automatic merging of these PRs for builds that pass and have sufficient coverage.

Renovate PR

Developer Security Training

We are looking to introduce an external secure development training solution such as application.security or PagerDuty security training to increase developer awareness and understanding of security vulnerabilities.

Introducing Security Champions

We are planning on starting a security champions program to enlist security-minded engineers to advocate for security best practices.

Once trained, the security champions will become the voice for security within their teams and work together to identify improvements to culture, processes and tooling.