SIEM dead, time for search?

This is what Raffy is saying:"Some of the problems I see with Security Information Management are (the first four are adapted from the Gartner IDS press release):

• False positives in correlation rules

• Burden on the IS organization by requiring full-time monitoring

• A taxing incident-response process

• An inability to monitor events at rates greater than 10.000 events per second

• High cost of maintaining and build new adapters

• Complexity of modeling environment

However, the biggest problem lies in the fixed event schema. SIMs were built for network-based attacks. They are good at dealing with firewall, IDS, and maybe vulnerability data. Their database schema is built for that. So are the correlation rules. Moving outside of that realm into application layer data and other types of logs can get hard. Fields don’t match up anymore and the pre-built correlation rules don’t fit either.We need a new approach. We need an approach that can deal with all kinds of data. An approach that deals with multi-line messages, with any type of fields, even with entire files as entities. There is a need for a system that can collect data at rates of 100.000 events a second and still perform data analysis. It needs to support large quantities of analytical rules, not just a limited set. The system needs to be easy to use and absorb knowledge from the users.The solution is called IT search."I really agree on the value of IT search, but I believe we have some confusion over the main objectives of each tool. If you are thinking about data mining and a more deep analysis of log data, maybe searching is really a better approach. What I really question is using searching for alerting purposes. I don't think search based architectures for a "log analysis IDS" scale.Raffy hits the point when he mentions that SIEMs target network based devices. I have seen people working to integrate logs from different sources (applications) on those tools having a hard time with the vendors, who simply can't understand the notion of using other log data besides routers, firewalls and IDSes.Of course that logs from applications are not as simple as logs from network devices. Maybe that's why the vendors are avoiding them. They want to sell their products as plug and play boxes, and you can't have a plug and play installation when dealing with custom applications. What I believe is that a effective SIEM (or, if you don't want to define the technology behind it, a consolidated log monitoring) solution installation is more similar to a ERP (or Identity Management) deployment than to an antivirus deployment. If vendors could improve their products not by including more supported log formats but by delivering a fast an easy way to build log parsers, together with a flexible model for the entities that the tool and its rules can work with it would be quite easier to deploy them to provide better value and integrating more log sources.The IAM tools evolved the same way. Since the beginning they could work with LDAP, ActiveDirectory, RACF and other famous identity repositories. The challenge for the adopters, however, was not on integrating these tools but those old legacy applications. The IAMs that have better "universal adapters" are those that can generate the best results. I think it would be the same for SIEMs. All of them can work with CEE or something similar, but those with easy (and intelligent) tools to accept different sources will bring more benefit to the customers. Even search technology can be used in order to do that.So, don't blame SIEM tools, but their architects. When these people can understand where the biggest value for those tools is we will start to see huge benefits from them.