Details, Details, Details...How Much is Enough?

So you think being a penetration tester is the coolest thing around right?  Me too..  but there is one aspect that people usually don’t think about: Report Writing.  It is one of the most important parts of an assessment because it provides the customer with data they can then use to make important decisions regarding their security.

For developers, specifically, the level of detail provided in the report can play a big role in their ability to accurately resolve identified issues.  Some findings are very simple and don’t require a lot of details.  They are easy to reproduce and remediate. Or are they?   To security consultants and penetration testers most vulnerabilities seem simple and make sense.  Talk about cross-site scripting and most testers immediately understand the problem.  Take the same conversation to a developer and it may not be so clear.  Not because they are not as smart, or anything like that, but because they don’t deal with the same issues on a daily basis.

When we write our report, we need to make sure that it is going to be useful to everyone that needs to see it.  We include an executive summary for that high-level overview for the executives, while providing low level findings for those that have to analyze the report.  How do we know when we have put enough detail.. or possibly worse, too much detail into the report.

I know what you are thinking, how can there be too much detail.  I think it is possible to put so much detail that the report becomes excessively long and the reader loses focus and stops paying attention.  This is a bad thing because if the reader doesn’t catch everything, something significant could be overlooked.  This is similar to if there are not enough details for the reader to fully comprehend the issue identified.

We need to find that happy medium that will allow the reader to understand and reproduce the issue identified.   Think about CSRF.  Is it enough just to say that a function is vulnerable?   Should we include the proof of concept code used to validate the flaw.  A proof of concept is great as it allows the developer or reader to see exactly how it was done, and also use that for a test case to see if the flaw is getting fixed.  (Obviously, this doesn’t work if they just modify the code to look for that one PoC file)  We could even go a little farther..  what about creating a screen capture video clip showing the exploit happening?   Although interesting and it does provide that wow factor, the PoC code is usually more beneficial.  But again, it is about the audience.   Managers may like a video rather than just some screen shots.

I always find determining the level of detail somewhat a balancing act for each report that I work on.   My main concern is that I produce a report that reflects the testing that was performed, provides insight into the current security posture of the systems tested, and includes the best recommendations for that specific client’s situation.  As you read through reports, make sure that you think of the information in multiple contexts.  Read it as a security manager.  Read it as a developer.  Determine if the information provided is just to create a report or to actually assist the client in analyzing their security.

James Jardine is a principal security consultant at Secure Ideas. If you are in need of a penetration test or other security consulting services you can contact him at james@secureideas.com or visit the Secure Ideas – Professionally Evil site for services provided.