Wednesday, March 3, 2010
The new school and black swans
I'm currently re-reading "The Black Swan", by Nassim Taleb, in a moment when most information security planning and decision-making techniques look like just plain bullshit to me. So, my mood for accepting absolute truths on this fields is becoming even worse than before.I was reading a post from the "New School of Information Security" blog, which, by the way, is very good. However, there is something from this "new school of thought" that I really have a problem to accept, the idea of measuring the effectiveness of security controls. The post I was referring to includes an example of new techniques to measure and predict the effectiveness of baseball players.Take, for instance, an affirmation like "80 percent of the league couldn’t have made that catch". Thinking on the nice work from Nassim Taleb, people (and so outfielders) physical attributes are usually only slightly different. Checking the past features from league outfielders should not give you enough information to say something like that, specially considering the interval between the games and the constant training for the athletes. It's too much conclusion based on past data that don't have a direct causality relation with the event you are trying to predict.That is also common on security. With the speed of changes and complexity of IT systems, constant changes of user behaviour due to those new systems (social networks?), it is extremely hard to produce a decent forecast of future events based on past data. Why would all the data about the exploitation of OS and web servers vulnerabilities from the past decade be useful to determine exploitation trends of browser vulnerabilities or XSS on social network websites?We should be a little more skeptical on our ability to forecast events, specially security incidents. The great "new school" I'm waiting to see rising is how to protect our data without relying on magic numbers and formulas. That would be innovation.