tl;dr sec is a newsletter about AppSec and scaling security, automated bug finding, conference talk and paper summaries, and useful links from around the web. You can subscribe here and see past issues here.

DevSecCon Tel Aviv, Slack's Secure Overlay Network, Dangers of Struct Padding

Hi there,

After a lovely trip to Israel and the UK for DevSecCon Tel Aviv and London, I’m finally back in the Bay Area, where people are constantly pushing the forefront of what’s possible with technology.

That is, the people who have electricity, because the local energy company hasn’t cut their power due to concerns about fires. You know, normal stuff 😅

If you'd rather read this newsletter on our blog, you can read it here.

Nominated for CTO Universe MVP Award

An article I wrote for TechBeacon (Scale your security with DevSecOps: 4 valuable mindsets and principles) was one of 9 articles selected to be finalists for the 2019 CTO Universe MVP Awards in the DevOps category.

If you want, you can vote for my article here.

📜 In this newsletter...
  • Cloud Security: free AWS best practice configs, Slack's secure global overlay network, AWS metadata service hardening
  • Misc Tools: Godbolt compiler explorer, Singularity - a DNS rebinding framework
  • OSINT: bug bounty tips on discovering attack surface, new version of Amass
  • Collecting and Searching Information: 2 open source and 1 commercial tool
  • Programming: hands-on deadlock challenges, practical applications of formal methods
  • Politics: Google's acquiring your health records, Taiwan's social network that promotes debate
  • Padding the struct: How a compiler optimization can disclose stack memory
DevSecCon Tel Aviv:

Opening keynote by Elissa Shevinsky, lightweight threat modeling as code, managing secrets, container security stats, and I discuss the current state of DevSecOps and make predictions about where we're headed.
🔗 Links

Cloud Security - A free repository of customizable AWS security configurations and best practices.

Introducing Nebula, the open source global overlay network from Slack
“Securely connect tens of thousands of computers, hosted at multiple cloud service providers in dozens of locations around the globe.” source code

  • Service provider agnostic
  • Allows traffic filtering by host identity (not just IP address)
  • Hosts identify themselves via certificates that encode user-defined attributes (datacenter, role, environment, etc.).

Add defense in depth against open firewalls, reverse proxies, and SSRF vulnerabilities with enhancements to the EC2 Instance Metadata Service
V2 of the EC2 Instance Metadata Service (IMDSv2) released, with several changes to prevent common attacks:

  • Every request now requires a session, whose secret is obtained via a PUT request to IMDSv2
  • Calls with the X-Forwarded-For header will be dropped
  • The PUT response containing the secret token will have its IP packet TTL set to 1, meaning requests from the EC2 instance itself will work, but the response likely can’t make it to an external attacker.

Misc Tools

The Godbolt Compiler Explorer is a nice and easy way to see how C code gets translated to assembly by a number of compilers (linked by Jack Leadford’s post below).

Singularity - the current state of the art in DNS rebinding by Gerald Doussot and Roger Meyer. I’m calling it out again because their DEF CON 2019 talk is now live on Youtube.

Open-Source Intelligence (OSINT)

Hussein Daher:

When discovering subdomains/domains/assets owned by a company, use the Google Analytics ID to expand your attack surface. The ID is in the HTML code. Reverse search then:

Gwendal Le Coguic:

We always talk about methodology to find subdomains, but what about domains first? If you want to enlarge your scope, I use,,, to find more domains owned by a company

OWASP Amass v3.3.0 released, a Go tool that does “in-depth attack surface mapping and asset discovery.”

  • Supports gathering info from DNS, scraping, certificates, third-party APIs, and web archives.
  • Future releases will support saving results in Neo4J and any graph databases that supports Gremlin (@apachetinkerpop).

Tools for Collecting and Searching Information

Vortimo - “organizes information on webpages that you’ve visited. It records pages you go to, extracts data from it and enrich the data that was extracted. It augments the pages in your browser by allowing you to tag objects as well as decorating objects it deems important.” Chrome extension + server with DB and UI, not open source. (thx Daniel Miessler)

Memex - a fully private browser extension to full-text search your browsing history & bookmarks (open source).

organice - an implementation of Org mode built for mobile and desktop browsers. Open source, uses React, can use Dropbox, Google Drive, and WebDAV as backends.


The Deadlock Empire: Slay dragons, master concurrency!
Hands-on multi-threaded programming challenges with a fun theme 🤘

Tweetstorm by Hillel Wayne on the power of formal methods (TLA+). Here are some snippets to give you a flavor, check it out for some neat talk and blog post links:

TLA+ is a language for “debuggable designs”. It lets you design your system and find bugs in the design itself, not the code. In terms of expressive power, TLA+ is as high above Python as Python is above x86 Assembly.

My first time using TLA+ for work was for a tricky business domain problem, involving lots of edge cases and unreliable vendor APIs. The code was 3000+ lines of Ruby. The spec? 60 lines of TLA+. It found two critical bugs.

This was a battle-tested production system. It had been thoroughly unit-tested, integration-tested, code reviewed, and monitored. We’d spent a year fighting fires and fixing bugs. Months of work on stability.


Google’s secret cache of medical data includes names and full details of millions
Google has a secret “Project Nightingale,” in which they are collecting vast amounts of American healthcare data, unbeknownst to patients. Google is receiving the data from Ascension, the second-largest healthcare provider in the U.S., including patients’ full personal details (name + medical history). This data has not been anonymized and can be accessed by Google staff.

How a social network could save democracy from deadlock
Taiwan has built a platform designed for people from across political divides to express their views. It promotes statements that find support across different groups as well as within them, gamifying finding consensus. Rather than simply letting people vote via an app, the platform gives participants agenda-setting power to not just determine the answer, but also define the question. It doesn’t aim to find a majority of one side over another, but rather achieve consensus across them.

“People compete to bring up the most nuanced statements that can win most people across,” Tang told me.

“They spend far more time discovering their commonalities rather than going down a rabbit hole on a particular issue.”

Padding the struct: How a compiler optimization can disclose stack memory

NCC Group consultant Jack Leadford on how GCC may add padding to struct objects, attempting to make your code faster (by causing field accesses to be memory-aligned), but this can lead to disclosing stack memory!

Jack describes two example scenarios where this behavior may be exploited: an RPC server and a kernel driver. Here’s a useful illustration from a blog post by Alexander Popov:

Related: a Linux Security Summit talk Dealing with Uninitialized Memory in the Kernel by Alexander Potapenko of Google.

⭐️ DevSecCon Tel Aviv 2019 Roundup

DSC Tel Aviv was nice: it was small and personable with a great venue, some single track talks, and then an “unconference” section.

If “unconference” doesn’t mean anything to you, don’t worry, it didn’t to me either until I showed up. Attendees collectively put forth a number of topics or challenges we had, we voted on them, and then self-organized into small groups where we discussed them in detail. Pretty neat.

Here are some key takeaways from the talks.

The Hardest Part of DevSecOps: Working Together

Elissa Shevinsky started the conference off strong with an audience interaction-heavy opening keynote to get people engaged and starting to discuss the most pressing challenges they had.

I hadn’t met Elissa before, but throughout the con I was impressed by how friendly and supportive she was of the people around her. If you run into her at a con, you should say hi!

She’s also the author of the book Lean Out: The Struggle for Gender Equality in Tech and Start-Up Culture and CEO / co-founder of Faster Than Light, an early stage start-up aimed and making static analysis faster by distributing work to many nodes via Kubernetes on AWS. It’ll be interesting to see where this goes.
(home page, try it now)

Lost in (DevSecOps) Space – practical approach for “Lightway” Threat Modeling as a Code

(slides) By Vitaly Davidoff, AVP and Application Security Expert at Citi Innovation Lab.

As a number of other talks have pointed out, traditional threat modeling (TM) approaches, such as STRIDE, PASTA, VAST, and Trike have struggled to keep up in fast-paced, Agile development shops.

Other common TM problems include:

  • The results are not propagated to developers.
  • TMs are updated rarely (if at all) as applications change.
  • TMs are concentrated on diagrams, not countermeasures, and are not integrated within the DevOps development model (CI/CD).

The core idea in this talk is to follow the lead of other DevOps domains, such as infastructure as code and CI/CD processes as code, and implement TM expectations in code as well.

The goal is to build up a series of templates for common threats (specified in JSON or YAML) that can be applied to many applications, for example, risks relevant to any web app, or REST API, GraphQL API, etc.

Not all aspects of threat modelling will be automatable, but you can still get some great value and scalable wins from automating the common cases and low hanging fruit.

Here’s the process Vitaly recommends:

Step 1: Pattern Definition. Source: Stephen De Vries - Threat Modeling with Architectural Risk Patterns, AppSec USA 2016
Step 2: Threat Modeling YAML File. Source: ThreatPlaybook

Secrets are hard to manage. How to manage your secrets and keep them safe

Jake Hall, Principal Consultant at Infinity Works, gave a nice overview of the problems caused by a poor (or no) secret management strategy, how to gradually start managing secrets properly, and concluded with a Vault demo.

Without a good secret management solution:

  • You don’t know what secrets exist and where they are, as they could be distributed across source code, your wiki, etc.
  • You have limited visibility into how they’re used- attribution of actions taken using a shared secret is tough, and without central management, it’s impossible to know which users or services are using a given secret (e.g. DB credentials).
  • If a secret is leaked or otherwise needs to be rotated, lack of central management makes rolling the secret hard, time-consuming, and error prone, likely leading to development slowdown or potential service failures.

How to get your secret sprawl under control:

  1. Find all of your secrets for a given service (e.g. a microservice, DB, or third-party API).
  2. Copy the secrets into your centralized solution.
  3. Integrate your centralized secret management tool with your service.
    • Have your applications read the secret at runtime from the new tool instead of how they were obtaining it before (ENV variable, plaintext file, source code, etc.)
  4. Validate, then run in production.
    • Start with 1-2 services, validate with engineering and DevOps teams that it works in prod.
    • Remove secrets from their old locations, then start rolling them.

To minimize your pain during this transition:

  • When you’re doing the PoC, test each service type you’ll need to support. For example, if you have Java and Python microservices, determine how the secrets management tool will handle both cases.
  • Choose the hardest case(s) first.
    • If you have a service that communicates with multiple third parties or otherwise has a complicated use case, make this service work first, which will allow you to demonstrate viability early on and de-risk future work.

Securing Containers by Breaking In

Liran Tal, Developer Advocate at Snyk, described a number of container security best practices and some stats from the Snyk State of Open Source Security - 2019 study. Some parts that stuck out to me:

  • 20% of Docker images had known vulns that would be fixed simply by rebuilding it (e.g. Dockerfile includes apt-get update or the like).
  • There can be a vast difference in the number of known vulns in system libraries in similar base images, for example:
    • node:10 - 582
    • node:10-slim - 71
    • node:10-alpine - 0

DevSecOps: State of the Union

This was my first keynote, so I spent awhile thinking about how to make it appropriately “keynote-y“™. I decided to make the first half of this talk a summary of modern DevSecOps principles and practices I’ve seen across many companies, and the second half my predictions about the future of DevSecOps and security programs in general.

I’m going to flesh out these predictions in more detail in a blog post, but here they are now as a teaser:

  • Many vulnerability classes will be squashed.
    • OWASP-type bugs (e.g. XSS, CSRF, SQLi) will be solved by better libraries, frameworks, secure defaults, and ecosystem changes.
    • Memory safety issues will gradually die to broader adoption of better languages, like Golang and Rust.
    • Business logic bugs will remain.
  • ML / AI - current offerings are mostly hype, but ML will likely be incorporated in powerful ways in the future in both offensive and defensive applications.
  • Platforms (e.g. cloud providers and code hosting solutions) will invest heavily in security, and security features will become a primary selling point of the platform.
  • Security product companies (e.g. SOAR, ASTO) will replace much in-house custom glue code.
  • Every security professional (SOC analyst, pen tester, reverse engineer, …) will be heavily tool-augmented.

What do you think?

Feel free to shoot me an email or reach out on Twitter and let me know if you agree, disagree, or believe something totally else entirely.

✉️ Wrapping Up
Have questions, comments, or feedback? Just reply directly, I'd love to hear from you.

If you find this newsletter useful and know other people who would too, I'd really appreciate if you'd forward it to them🙏

Thanks for reading!


@clintgibler | @programanalysis
Copyright © 2019 Practical Program Analysis, LLC, All rights reserved.

Want to change how you receive these emails?
You can update your preferences or unsubscribe from this list.


This email was sent to <<Email Address>>
why did I get this?    unsubscribe from this list    update subscription preferences
Practical Program Analysis, LLC · 2035 Sunset Lake Rd Ste B2 · Newark, DE 19702-2600 · USA

Email Marketing Powered by Mailchimp