tl;dr sec is a newsletter about AppSec and scaling security, automated bug finding, conference talk and paper summaries, and useful links from around the web. You can subscribe here and see past issues here.

OSINT + Screenshots, Fuzzing, and 2 Fast 2 Discoverious

Hi there,

In honor of this being a pre-Thanksgiving edition of tl;dr sec, I thought it appropriate to commemorate the origin of this holiday. Which is why this edition includes a blanket covered with smallpox a special helping of good security content, so you can enjoy and then nap for a few hours.

But truly, I want to thank you for taking the time to read tl;dr sec. I know you’re busy, and that time is precious, and I’m honored that you spend time reading the content I put together. I don’t take that trust lightly - I work hard to make tl;dr sec have quality content.

I hope you have a great Thanksgiving, and are able to spend some time relaxing with friends and loved ones 😊

I’m speaking at RSA 2020!

I’ll be giving a talk on DevSecOps, where I’ll give a massive survey of the trends I’ve seen and insights I’ve learned based on dozens of talks, blog posts, and in-person conversations with people at companies doing cool stuff.

Let me know if your team or company is doing something neat I should include!

Upcoming Events

Bay Area Fuzzing Meetup #2 on December 12th at Facebook. The first one was great - high quality talks and conversations afterwards.

📜 In this newsletter...
🔗 Links:
  • Tools: an open source postman, JS library for rendering Markdown/LaTeX, a neat source code visualizer/navigator
  • Web security: reCAPTCHA solutions can be reused (quickly), Kibana RCE prototype pollution walkthrough
  • Professional development: Daniel Miessler on blogging, Tanya Janca on talks, presentations, and social media
  • 2 Fast 2 Discoverious: Bugcrowd and Jeremiah Grossman's company Bitdefender partner on asset inventory, Bishop Fox is in the game too
  • Politics: Data leak shows how China 'brainwashes' Uiighurs, Swedish PM dishes a burn, with a side of delicious, delicious meatballs
  • Fuzzing: Coverage-guided fuzzers for Java, JS, and Python, paper on building faster fuzzers + tool release, a web fuzzer
  • Destroying x86_64 instruction decoders: Trail of Bits describes how to find discrepancies in x86_64 decoders using differential fuzzing

📚 New Summary: Pose a Threat: How Perceptual Analysis Helps Bug Hunters:

In this AppSec Cali 2019 talk, Rob Ragan and Oscar Salazar of Bishop Fox describe how to make external network penetration tests more effective by auto-screenshotting exposed websites and then clustering them based on visual similarity.

They scanned every web app on AWS Elastic Beanstalk and found many credentials, API keys and source code.

At a high level, their approach is:

  1. OSINT (OWASP AMASS) is used to find target domains and subdomains.
  2. Threat intel, wordlists, and other sources are used to find interesting paths to check.
  3. AWS Lambdas are spun up that use headless Chrome to screenshot these paths.
  4. Shrunk screenshots are stored in S3, response bodies and headers are stored in Elasticsearch. Screenshots are grouped by similarity using fuzzy hashing.
  5. Humans review sorted screenshots for leaked sensitive data or promising targets to attack.
🔗 Links


postwoman - API request builder - A free, fast, and beautiful alternative to Postman.

TeXMe - a lightweight JavaScript utility to create self-rendering Markdown + LaTeX documents.

Sourcetrail is a cross-platform source code explorer designed to help you grok unfamiliar source code quickly. It does lightweight static analysis on C, C++, Java, and Python source code to extract class and method definitions, member fields, class hierarchies, and more.

Web Security

Cracking reCAPTCHA, Turbo Intruder style
James Kettle of Portswigger found that you can use reCAPTCHA solutions several times if you go fast enough. The issue was reported to Google 8 months ago but Google didn’t think it was exploitable in the wild.

  • Impact: Attackers can register more spam accounts for each solved CAPTCHA, increasing their spam rate.
  • Fix: If you’re using reCAPTCHA, you’ll need to manually secure it by locking/synchronising on the g-recaptcha-response token.

Exploiting prototype pollution – RCE in Kibana (CVE-2019-7609)
Nice illustrative walkthrough. Prototype pollution occurs when in JavaScript an attacker can overwrite properties of Object.prototype, which other objects inherit from, and that capability is used to cause unintended behavior, for example, making every object’s isAdmin property default to true.


Professional Development

One thing I enjoy reading is how people who are experienced at something do that thing.

So You Want to Start a Blog by Daniel Miessler

Tanya Janca’s tips on giving technical talks and presentations and social media.


2 Fast 2 Discoverious (Moar Asset Inventory)

Two weeks ago, in tl;dr sec #13, I mentioned Bugcrowd’s new “Attack Surface Management” product, in which you pay some of their bug bounty researchers to do asset discovery for you. One concern I had for it is that the approach seemed challenging to guarantee consistent, high coverage.

So it was interesting to see a few days ago this teaser tweet by Jeremiah Grossman that Bugcrowd is partnering with his asset discovery company, Bitdefender. Jeremiah also just tweeted some stats from analyzing the Internet-connected assets belonging to the top ten U.S. banks.

I know what you’re thinking, and the answer is yes, I like putting my degree in Graphic Design to good use.

It’ll be interesting to see where this partnership goes.

Also, the talk summary below from Bishop Fox describes some of the infrastructure they’ve built up around assessing a company’s external attack surface, an up-and-coming service offering they’ve been funding with their $25 million Series A from ForgePoint Capital in early 2019.



Data leak reveals how China ‘brainwashes’ Uighurs in prison camps
Surprise! People didn’t “voluntarily” decide to attend a high security re-education camp.

Swedish PM hits back at China over literature prize threat
China threatened to impose “counter-measures” on Sweden if a dissident Chinese writer, who is currently imprisoned in China, receives a literature award. Swedish PM: “We have no intention of yielding to these sorts of threats. Ever. We have freedom of expression in Sweden and that is what applies here. Full stop.”



ffuf - Fast web fuzzer written in Go

javafuzz - coverage guided fuzz testing for Java. The company ( has released similar libraries for JavaScript and Python.

Building Fast Fuzzers - Paper on how to build faster grammar-based fuzzers.

fzero_fuzzer - “A target-architecture-agnostic grammar-based fuzzer (inspired by the above paper). With no input size constraints, multi-thread support, and all Rust code for no corruption bugs. 5x faster than the worlds fastest grammar-based fuzzer” by Brandon Falk 🔥


Destroying x86_64 instruction decoders with differential fuzzing

Instruction decoding is implicitly treated as a solved problem. Analysis platforms give analysts a false sense of confidence by encouraging them to treat disassembled output as ground truth, without regarding potential errors in the decoder or adversarial instruction sequences in the input.

In this post, Trail of Bits describes and releases mishegos, a tool that uses differential fuzzing to find bugs in x86 decoders (Capstone, libbfd/libopcodes, udis86, XED, and Zydis).

Differential fuzzing (or “differential testing”) is basically a fancy way of saying, “Given 2 or more implementations that are supposed to do the same thing, give them all the same input, and see where the outputs differ.”

As a technique, differential fuzzing is really clever. Let’s think about it a bit more.

  • In standard fuzzing, looking for memory safety-type issues in programs written in C or C++, we can find bugs by throwing random garbage at the programs and observe if they crash, access invalid memory, or exhibit other bad behavior.
  • I call these bugs “context agnostic,” because we don’t need to know anything about the program; if we observe certain classes of behavior, we know there’s a bug, regardless of the program’s purpose or business domain.
  • However, how can we find bugs in complex business logic? For example, parsing a complicated file format, implementing a network protocol, etc.
  • In these cases, how would we know when the program we’re testing incorrectly handles the input? We would need an oracle that implements the business logic we’re testing 100% correctly, which we can then compare the tested program’s output against.
  • However, if we had a perfectly implemented oracle, we’d probably just use that, and not need to build the application we’re testing.
  • Differential fuzzing allows us to test complicated business logic without an oracle by comparing the outputs of multiple programs that are supposed to implement the same functionality, and investigating where the results are different.

    We don’t have to know a priori the correct output for every input or even which program is right when the outputs differ, we just know to investigate when the programs don’t behave identically.

📚 New Summary: Pose a Threat: How Perceptual Analysis Helps Bug Hunters
The problem statement:

You’re a large company with thousands to hundreds of thousands of websites online. Some of them are public intentionally, some aren’t - some are intended to be internal only, behind SSO or a VPN, or have been spun up and forgotten.

How can you figure out:

  1. Everything on the Internet belonging to your company?
  2. Which of those assets are high risk and need to be prioritized?

A strength of the approach described in this talk is you don’t need to know beforehand what you’re looking for. Instead, you can look at groups of similar web applications and focus on outliers to find likely issues.

This has a significant advantage over signature-based approaches, where you need to write a fingerprint of every type of thing you’re looking for, which requires a massive amount of upfront and ongoing engineering effort.

Applications of this research include:

  • Pen testers can assess larger clients’ attack surfaces more efficiently and consistently.
  • Companies can use it to find:
    • Potentially high risk assets or leaked sensitive info.
    • Websites that should not be public.
    • Regressions - fixes that have been undone.
    • Other instances of known bad services that need to be patched or taken down.
Robot Powered Bug Bounty
One neat example the speakers described is that a company gave them access to all of their bug bounty reports. They extracted the paths from those reports then used their infrastructure to scan all of the company’s subdomains they had enumerated for the same bug, finding other instances that hadn’t been reported via the bug bounty program. A security analyst can easily vet if the new instances are true positives by simply following the reproduction steps in the bug bounty submission!

Talk structure:

  • Challenges of current content discovery approaches (technical and process)
  • The architecture and approach of the automated tooling built by the speakers
  • An evaluation of the approach across every web app using AWS Elastic Beanstalk
  • Future plans and audience questions


Content Discovery of Yesteryear

In the olden days (~2008), content discovery was done using tools like OWASP DirBuster. These tools took file and directory wordlists and then requested them on one or more sites to try to find sensitive, hidden paths that could then be exploited.

Editor’s note: Some modern path brute-forcing tools you may want to check out are gobuster, dirble, and dirsearch.

Content Discovery Problems

  • Because you generally were running DirBuster from one machine, you could easily be blocked with one IPS or IDS role.
  • Sometimes you’d DoS the server, as you’re making tens of thousands of requests in rapid succession.

How do you know when requesting a directory or path fails?

After all, you’re probably doing this across thousands of different tech stacks, frameworks, and applications.

Most tools do this by:

  • For each domain, make a request to a path that’s known to be invalid (e.g. /thisdefinitelydoesnotexist).
  • Build an on the fly signature, based on a combination of factors from the server’s response, like HTTP headers, HTML body, HTTP status code, content length, etc.
  • Then, in subsequent requests, the server’s response is compared to the known bad request to determine if the path is valid.

Bringing in the Visual Component

The above process can still be a bit noisy and slow, so a common modern approach is to automatically screenshot every path using a tool like EyeWitness or gowitness. These tools leverage headless Chrome to locally store a screenshot of every page to make it easy to review later.

While this does make human review easier, manually going through hundreds of thousands of screenshots is still prohibitively time-intensive.

Process Problems

In addition to the technical challenges, there are also process challenges when a team of security professionals are doing this work:

Inconsistency – Content discovery steps may be done inconsistently when the scope is large, and it’s not feasible to run large-scale comprehensive dictionaries on every target in a small time window using traditional techniques. Further, different people may have different processes.

Divergence – If multiple people are performing the scanning, they may store the results separately and keeping this info merged/up to date is logistically challenging. As new targets are discovered through an assessment, ensuring all targets were reviewed the same way is tough.

Efficiency – Reviewing results is time consuming when the false positive rate is too high, and the dictionaries used may be outdated or inefficiently used.


Goal: Make it easy for humans to get eyes on interesting screenshots from URL path bruteforcing.

OK, let’s look at the system they devised that addresses these problems.

At a high level, the overall flow is:

  1. OSINT is used to find target domains and subdomains.
  2. Threat intel, wordlists, and other sources are used to find interesting paths to check.
  3. AWS Lambdas are spun up that use headless Chrome to screenshot these paths.
  4. Shrunk screenshots are stored in S3, response bodies and headers are stored in Elasticsearch. Screenshots are grouped by similarity using fuzzy hashing.
  5. Humans review sorted screenshots for leaked sensitive data or promising targets to attack.

Let’s look at a few of these steps in more detail.


OWASP Amass is used to build the initial list of domains and subdomains, as it’s great at aggregating various OSINT info (e.g. certificate transparency, DNS enumeration and brute forcing, scraping, etc.).

Request Paths

On those accumulated domains and subdomains, we need to know which paths to request.

So far, their favorite wordlists have been pathbrute, fuzzdb, and Commonspeak2, though they could also use wordlists from tools like dirbuster and gobuster.

They’ve also incorporated info from Grey Noise, which is a threat intelligence dataset from which they can glean the paths that real attackers are scanning the Internet for.

Though it’s early stages, they’ve also tried scraping paths from all of Exploit DB, CVE data, and getting a list of every web path in Metasploit.

Identify Outliers

The key here is they do a fuzzy hash of images. With cryptographically secure hashes, changing one byte makes the resulting hash totally different. In fuzzy hashing, however, small changes to the original file cause only small changes to the resulting hash, thus allowing you to group “similar” images.

The fuzzy hashing algorithm they use to detect image similarity is probably something like ssdeep.


Alright, so you built some fancy tooling, now time to take it for a spin in the real world!

The speakers decided to target every web app running on AWS Elastic Beanstalk.

The stats:

  • They had 88,022 total Elastic Beanstalk subdomain targets, and they made 10 path requests to each (based on Grey Noise data) => 880,220 total requests.
  • Of these 880,220 requests, 52,586 responded with a 200 OK, which they then screenshotted.
  • 4,007 of these screenshots were flagged as “interesting” to manually review.
  • 30 minutes of human review time yielded 321 confirmed exposures.

So 880,220 requests
=> 52,586 screenshots (~6% of total)
=> 4,007 interesting screenshots (~0.05% of total). Nice!

They found over 9GB of source code (that’s a lot of plaintext files) at paths such as /.git/config, which contained sensitive content like encryption keys, database passwords, API tokens, references to internal systems, and of course you can hunt for vulnerabilities in the source code.

Future Work

One thing they’re planning to do is perform OCR on screenshots, to allow performing filtering based on text in images (e.g. using AWS Rekognition). This should hopefully lead to fewer false positives and false negatives, because the tool will be able to understand when contextually important information, such as “404 Error” or “Login” are in images.

Machine learning has some promising applications here, enabling the system to more quickly identify potential issues, without waiting on a human analyst, enabling companies to fix exposures more rapidly. This could be done unsupervised, or potentially supervised, being fed labeled data from human analysts. Another interesting angle is that it may be possible to train an ML model per-organization, so the model understands what’s sensitive or dangerous to be exposed for that company, which may be quite different than other companies.

  • For example, the ML could learn that seeing a specific logo on the page may indicate whether the page should be externally facing or not. Likewise with ads or other potential indicators.

A few months after this talk, at BlackHat USA 2019’s Arsenal, Bishop Fox’s Dan Petro and Gavin Stroy released eyeballer, a convolutional neural network for analyzing pentest screenshots.

So far they’ve been focusing on breadth, but there could also be some value in applying these techniques at depth, for example, applying the same process for many paths on one target, like Burp Intruder, using common crawl data or another crawler. These pages could similarly be screenshotted and compared, arguments could be fuzzed, etc.

Audience Questions

Have you tried screenshotting with multiple rendering engines? Using only headless Chrome may cause you to miss things.

Some sites will show you more or less functionality based on your user agent (e.g mobile browsers), and some sites may not work on headless Chrome at all ( “This site requires Internet Explorer with ActiveX version…”). This isn’t a feature they’ve implemented yet, but they may in the future.

What’s the smallest set of assets that would still be useful to do this process with?

If you only have a handful of assets, it might not provide a lot of value, but they’ve used it successfully on some small businesses with just hundreds of assets.

✉️ Wrapping Up
Have questions, comments, or feedback? Just reply directly, I'd love to hear from you.

If you find this newsletter useful and know other people who would too, I'd really appreciate if you'd forward it to them🙏

Thanks for reading!


@clintgibler | @programanalysis


Copyright © 2019 Practical Program Analysis, LLC, All rights reserved.

Want to change how you receive these emails?
You can update your preferences or unsubscribe from this list.


This email was sent to <<Email Address>>
why did I get this?    unsubscribe from this list    update subscription preferences
Practical Program Analysis, LLC · 2035 Sunset Lake Rd Ste B2 · Newark, DE 19702-2600 · USA

Email Marketing Powered by Mailchimp