How Kubernetes components communicate securely in your cluster
This KubeCon talk by Maya Kaczorowski, Product Manager of Container Security at Google, describes:
- The main Kubernetes components that need trusted communication (API server, kubelet, etc) and how this communication is protected.
- How the cluster certificate authority (CA) works and how it grants certs to Kubernetes components.
- Authentication, integrity, and encryption options available in Kubernetes, and how you can protect other communications in your cluster (e.g. node to nonde and pod to pod)
Cloud Native Security Hub - Discover and share Kubernetes security best practices and configurations (thx Marco Lancini)
An AWS IAM Policy Linter: Parliament
Duo Labs blog post and tool release by Scott Piper describing Parliament, a tool that does linting of AWS IAM policies to detect cases like when a role could escalate its privileges.
Continuously monitor unused IAM roles with AWS Config
Walkthrough on the AWS security blog about how to use an AWS Config rule and a Lambda to continuously checks for inactive roles based on when they were last used.
Detecting Manual AWS Console Actions
If your company does infrastructure as code using tools like Terraform, then ideally no one should be making any changes manually. Arkadiy Tetelman describes how to set up AWS Cloudtrail alerting to detect when a manual change is made through the AWS Console, which he says has been one of the highest siginal/lowest noise alerts they’ve created.
Ramp-Up Learning Guide available for AWS Cloud Security, Governance, and Compliance
AWS released this PDF that lists resources including free digital training offerings, classroom courses, videos, whitepapers, certifications, and other materials, sorted by how they think one should best become familiar with the platform and related technologies.
Reverse Engineering for Beginners (Understanding Assembly Language)
Massive 1,079 page free e-book on reverse engineering from Dennis Yurichev.
If you want to learn about reverse engineering, I’d also recommend Malware Unicorn’s free online Reverse Engineering 101 workshop or Maddie Stone’s Android App Reverse Engineering 101.
Building Secure Reliable Systems: SRE and Security Best Practices
Free book by the Google SRE team, recommended to me by Caleb Sima over drinks last week. Snippets from the description:
Successfully building, deploying, monitoring, and maintaining systems is only possible when security and reliability are central elements in their architecture.
The central idea of this book is its focus on treating security and reliability as a common theme, one which is integral to software and system lifecycles.
An Overview of Cryptography
A pretty lengthy, free book on cryptographic basics, the types of crypto algorithms, trust models, and common algorithms in action. Cryptocurrency hype-free and good for the soul.
Version One, a VC firm that has invested in ~100 start-ups over the past 10 years, released a free Startup Handbook discussing building your team (hiring, culture, compensation), building your organization (leveling up, running), and building your investor base (fundraising, investor communication, advisors).
Purism describes the challenges they faced for Librem 5, the open source, privacy-first phone they’ve been developing, with neat features like a hardware kill switches for the camera/microphone and WiFi/Bluetooth/cellular modem. Challenges included designing the hardware from scratch and developing many of the drivers themselves. They’ve managed to keep much of the same stack as their laptops - PureOS (Debian derivative) and GNOME / GTK+, allowing applications written for desktop to run on the phone, with only slight changes.
Surveillance Giants: How the Business Model of Google and Facebook Threatens Human Rights
This 60 page document by Amnesty International discusses the business of surveillance (called “Surveillance Capitalism” by this well-received book by Shoshana Zuboff), the power/danger of data analytics at scale in terms of microtargeting and manipulating at scale, and how the concentration of power obstructs accountability.
SMS Replacement is Exposing Users to Text, Call Interception Thanks to Sloppy Telecos
Vice article on how various telco’s implementation of the RCS standard is done poorly, opening users to attacks like text message and call interception, spoofed phone numbers, and leaking their coarse location. “Because some of the standard is undefined, there’s a good chance companies may deploy it in their own way and make mistakes.”
Automate All the Things!
Almond The Open, Privacy-Preserving Virtual Assistant
Neat project by the Stanford Open Virtual Assistance lab that lets you speak or type commands to the virtual assistant, Almond, which it then parses using a natural language understanding model, and then takes one or more actions. Almond capabilities are defined in Thingpedia, a crowdsourced repo of commands and interfaces to online services and Internet of Things (imagine an open source Zapier or IFTTT). You can use Almond via your browser, a GNOME desktop app, an Android app, or via CLI. Examples commands include:
- “Show me the weather in San Francisco.”
- “Get the current Bitcoin price and send it to my colleague on Slack.”
- “Wake me up with my Spotify playlist ‘rise and shine’ at 8am every day.”
“Open source home automation that puts local control and privacy first.” Can be run on a Raspberry Pi or a local server. Recently added integration with Almond.
Cloudflare’s AppSec team was less than pleased with existing commercial network vulnerability scanners, so they created Flan Scan (source code), which is a thin wrapper around Nmap that uses the vulners script to map detected services to relevant CVEs.
- Runnable via Docker, comes with sample Kubernetes configuration and deployment files so you can get up and scanning quickly.
- Can push results to a Google Cloud Storage Bucket or an S3 bucket, making it easy to run a number of scans and collect the results in one central location for processing.
- Generates actionable reports so you can quickly identify vulnerable services on the network, the applicable CVEs, and the IP addresses and ports where these services were found.
- Outputs LaTeX ❤️
By complementing osquery’s findings with Flan Scan’s network scans we are working towards comprehensive visibility of the services running at our edge and their vulnerabilities. With two vulnerability trackers in place, we decided to build a tool to manage the increasing number of vulnerability sources. Our tool sends alerts on new vulnerabilities, filters out false positives, and tracks remediated vulnerabilities.
Sounds like they’re building an inventory of their assets. Like an asset… inventory 😉.
Flan Scans results are structured around services. The report enumerates all vulnerable services with a list beneath each one of relevant vulnerabilities and all IP addresses running this service. This structure makes the report shorter and actionable since the services that need to be remediated can be clearly identified.
Security in the Real World™
A handful of people on Twitter were grumbling about Flan Scan because it’s a simple wrapper around
nmap. I think that’s a feature. In the real world (read: not the Black Hat stage), there are no style points to be won from the Russian judge for the complexity of a solution. The goal for most effecctive AppSec teams is to do the simplest thing that works well. The security ROI of a tool is a function of: how much time is required to create and maintain it? What are the ongoing operational and triage time requirements? Some of the top AppSec teams I’ve been able to work with build focused tools that handle specific use cases as precisely as they need to, with no unnecessary frills or complexity. Leave the triple salchow to someone else, we have real security to do.
PayPal Releases SCORE Bot
PayPal open sourced SCORE Bot, their lightweight, continuous code scanning tool that can comment on PRs with findings, tracks metrics, etc. They originally discussed SCORE Bot at AppSec USA 2018, which I wrote a summary of here. There have been a number of talks in this space, here’s what sticks out to me about this one:
- They A/B tested security messaging and found it lead to significantly better outcomes.
- They focus on maximizing security iteration speed, which is quite clever and important. Security tools aren’t often built with this in mind, but I think it can be a game changer.
Here’s a blurb I wrote about the value of maximizing security iteration speed, taken from the summary linked above:
- How valuable would it be if you could notice a common code anti-pattern and then in an hour write up a quick check and roll it out to every repo such that you got coverage on every commit from now on?
- How useful would it be to be able to write a new check, get feedback on its effectiveness, tune it to improve signal, and have multiple rounds of that feedback-driven improvement loop take minutes or at most hours, not days or weeks?
- What if you could add in (or remove) an additional security tool into your CI/CD pipeline in an hour?
- Is there any security automation you're not doing because you know that rolling it out or tuning it will be too time intensive or painful?
Early stage project in Firefox to allow content processes to record their behavior, replay it later, and rewind to earlier states. Replaying processes preserve all the same JS behavior, DOM structures, graphical updates, and most other behaviors that occurred while recording. The browser’s JS debugger can be used to inspect and control the replay. Basically time travel debugging, but built-in to the browser, super cool!
The Siege of Gondor, Part I: Professionals Talk Logistics
Growing up, my dad read The Hobbit and Lord of the Rings to me, which inspired a life-long love of fantasy and science fiction. This fun article series examines the Siege of Gondor from a practical, historical military perspective. Does the strategy of the Witch King make sense? Would it be feasible for an army of orcs of the size portrayed to march the described distances in the requisite amount of time? What about supply chains?
📚 New Summary
tl;dr Alexandra Nassar of Medallia describes how to create a positive vulnerability management culture and process that works for engineers and the security team.
You can read the following full summary on my blog here.
First: Understanding the Lay of the Land
When Alexandra started, there were a number of process and communication challenges causing frustration for both the development and security teams around vulnerability management.
Instead of leaping in and making changes, she took the time to interview engineers, their managers, the security team, and even the PMO (the project management department).
She identified a number of consistent issues:
- Developers didn’t know how to prioritize fixing the vulnerabilities assigned to them vs. the other tasks in their backlog.
- Vulnerability fix tasks could easily get lost in the backlog. Sometimes engineers didn’t even know they had fixes assigned to them.
- Vulnerability-related tasks and info were split across a number of tools and systems. There was no standard workflow and process; instead, issues were often handled as one-offs.
- There was an overreliance on individuals - remediations would be assigned to a particular engineer, which could get dropped when they went on vacation or paternity/maternity leave.
- Developers felt like the security team was setting all of the logistics around remediation (SLA and timeline). They wanted to be a part of the process.
How can we make engineers want to be a part of the process and collaborate?
Alexandra also took a step back and asked teams, “What is a vulnerability to your team?”
Don’t take things like this for granted: different people and teams may view things different, so it’s really important to nail down important terms together.
Where Does Vulnerability Data Live?
After speaking with a number of dev teams, Alexandra learned that they’d prefer to keep track of vuln data in the issue tracking software they already used for standard feature work, Jira.
There are two main ways vulnerability tasks could live in Jira:
- With the associated team and project (e.g. In the SRE team’s project)
- In a separate project dedicated to vulnerability data.
Alexandra ended up choosing #2 because different teams can have their own custom workflows with how they handle their tasks, which she didn’t want to interrupt, and by having the vulnerabilities tracked separately they wouldn’t get lost in teams’ existing backlog. This also gave her full control and flexibility over the workflows she creates.
Determining How it Should Work
When defining the vulnerability management workflow, in addition to interviewing all of the relevant stakeholders, Alexandra also manually sampled a number of old vulns to make sure that every state they could be in was represented in the process she defined.
Other important points:
- Create a consistent intake workflow - Regardless of the source of the vulnerability, whether it was from a tool (like Qualys or Nessus), an external pen test, bug bounty, or another source, all vulnerabilities undergo the same initial triaging process and are input, tagged, and labeled in the same system.
- Make tickets detailed and actionable - Alexandra noticed that teams weren’t working on many of the tickets created by tools, and after speaking with them, she found that it was because the tickets didn’t have enough context such that the teams could fully understand the issue and its importance as well as how to fix it. Now, the security team ensures that every security ticket has sufficient context as well as actionable guidance on how to fix the issue.
Metadata To Track
Medallia tracks several features on security-related Jira tickets:
- Priority - In security we tend to use “risk level”, but that terminology is unfamiliar to developers. By using “priority,” it enables developers to rank the relative importance of the security issues against the other issues in their backlog.
- CVSSv3 Score and Vector - Choose whatever rating system you want, but make sure it’s universal and consistent.
- Team - Security issues are assigned to teams rather than individuals to ensure they don’t get dropped.
- Release tag - Useful for tracking a vulnerability and knowing when you need to validate the fix in production.
- Source - What discovered the vulnerability? Third party pen testing, bug bounty, etc. This info is also quite useful for the compliance team.
- Due date - A flexible recommendation from the security team, which can be adjusted based on developer feedback and their workload / priorities.
Building the Process
One common complaint developers had with the prior process is that after they had completed their part, while waiting on someone else to push things forward, the security issue was still cluttering up their Jira board.
To address this, Alexandra created new Security and Development kanban boards so that relevant tickets only showed up on someone’s board when they have work to do.
In the security dashboard, there are two primary states a ticket can be in: incoming vulnerabilities that need to be triaged and issues that need security attention (e.g. a developer has a question, a fix needs to be validated, etc.).
The developer dashboard has the following states:
- To acknowledge: After the security team has triaged an incoming issue and added the necessary context, it goes here for development teams to start reviewing.
- To Scope: The relevant development team estimates how much work the issue will take to fix and places it into the sprint planning process.
- In progress: Developers are working on the ticket.
- Rejected: The development team doesn’t believe this is an issue, has mitigating factors, or otherwise “Won’t Fix.”
- Due date extension: Due to the expected amount of work and their timelines, a different due date is proposed, which must be approved by the security team.
- Needs security attention: The developers have completed their part, now they need the security team to review what was done.
At 23:30, Alexandra gives a demo of this workflow.
The key here is it’s very clear what work needs to be done by whom at any given point in time.
Good metrics are crucial for understanding where your security program is and if the efforts you make are causing an impact.
When leadership comes to Alexandra and says, “Our Quarterly Business Review is coming up, what goals should we set?”, having metrics enables her to make specific suggestions.
For example, if there are a number of open vulns, perhaps developers may not be scoping the required work appropriately, so in the next quarter, let’s aim to try to get to zero overdue vulns. If there are many high or critical vulns that are overdue, she can focus in on that and try to determine the root cause.
It’s useful to show trends, like presenting an overview by team or org to executives who have many teams reporting to them.
Metrics can also help you deal with fixes more effectively; for example, there may be 100 vulns that will be fixed by the same action, such as applying a certain patch. Alexandra tries to help development teams fix issues more efficiently and strategically by applying labels to certain vulnerability classes or efforts to help development teams treat fixing them like a project, rather than a series of one-offs.
Metrics can also be used during management meetings to create some healthy competition between teams.
How do you handle escalations?
If there’s something that needs immediate attention, Alexandra brings it up in the weekly security check-in meeting she holds every Thursday, attended by the CISO, all of the security managers, and all directors and VPs of the organization. It’s always 30 minutes at the same time and she sets the agenda on Monday. If there’s nothing in particular to discuss she doesn’t cancel the meeting, but lets it become an open discussion forum that people can attend if they want.
Metrics and Data-driven Security Programs
I’m a big fan of using data and metrics to drive security efforts and measure their effectiveness. If you are too, check out my summary of the talk by Koen Hendrix of Riot Games where he measured the maturity level of various development teams and correlated that with their relative bug bounty payouts, and this summary of a talk by Arkadiy Tetelman on how the Airbnb team used bug bounty reports to determine where and how they invested their AppSec team and money.
Culture: Driving Change
You can have the best tools and processes in the world, but ultimately you need buy-in and cultural change for the security team to be viewed as an ally and have what you recommend be adopted.
When Alexandra made these Jira workflow changes, she met with all of the different engineering teams to train them on how to use the Kanban boards she created and socialized the value / better efficiency it provided them.
Don’t assume people know how to use a given tool or process, other people may use it very differently than what makes sense to you.
One of the biggest wins has been leveraging Medallia’s security champions program, which includes an engineer from every team in the company. The champions are valuable liasons with development teams, and Alexandra has found that teams with active security champions have fewer overdue vulns.
Other initiatives to get your colleagues excited about security include gamifying security metrics and making a healthy competition of the number of open or overdue vulns, holding CTFs or bug bashes, and definitely swag. Once Alexandra didn’t have enough budget to get everyone t-shirts, as she’d planned, so she asked their facilities team if they’d be willing to make custom badges for the people who won the CTF. They agreed, so the winners got to walk around with unique, security-themed badges and receive props from their peers.
Brand the hell out of your program - it changes people’s perception of security.
✉️ Wrapping Up
Have questions, comments, or feedback? Just reply directly, I'd love to hear from you.
If you find this newsletter useful and know other people who would too, I'd really appreciate if you'd forward it to them🙏
Thanks for reading!
@clintgibler | @programanalysis