Scaling Traditional Bugfinding

While there's been quite a bit of offensive research into applying traditional application security techniques like fuzzing and protocol analysis to newer technologies, the bulk of the security work needed to lock down most of the internet is considerably less esoteric. We need more techniques focusing on how to take traditional application security bugs, and how to find them using less time and resources. The way to secure the entire web looks a lot more like doing doing traditional engineering than some of the more exploitation focused research.

Traditional penetration testing of services is quite effective at finding bugs in websites. We know how to find security vulnerabilities by black boxing applications or doing source code analysis. The difficulty in relying exclusively on these methods is the cost. If you have a very large website to secure, relying exclusively on human analysis means you either gate every project around a limited security resource or you build frameworks that solve traditional bug categories.

A method i've found particularly effective has been to have good frameworks, and monitor code at diff time for errors. It's often possible to catch security bugs by monitoring the use of key parts of the framework needed to invoke security bugs by use of fairly cheap and simple techniques like string comparison for known bad patterns. Hooking into your source control system at diff time to monitor for these bad patterns lets you work with developers before the buggy code ships. Over time it also puts you in a position to train developers to use the frameworks you prefer. In this respect it's doubly powerful. Create the correct frameworks, monitor misuse or non-conformance, and train developers to use the newer better methods. This can build a positive reinforcement cycle where over time developers train each other to solve common application security problems in a safe way.

The big advantage of this method for bug finding is that you can be fairly precise about the economics of finding certain types of bugs. If it takes 10 minutes to assess source code for a cross site scripting bug, and your pattern has a 20% signal to noise ratio, you find a bug every fifty minutes of work. The individual patterns can be isolated, allowing you to improve and tune the amount of time needed to find bugs. By focusing on maximizing bugs per hour, a bug sieve method like this can provide an effective means of more cheaply finding traditional application security bugs than manual penetration testing.