Big Data: Open Source Tools vs Splunk
When it comes to dealing with huge amounts of syslog data, there are few good options available to a system administrator. Large environments composed of hundreds, if not thousands of servers, routers, switches, applications and users generate an immense amount of log data, and sorting through it is not a task for the weak hearted. How are you supposed to manage 10 gigabytes, 20 gigabytes or more of syslog data? And how are you supposed to present this data to users in ways that make sense and provide a tangible benefit?
I have recently compared the main commercially available solution, Splunk, with a couple of the more popular open source products, Google’s Enterprise Log Search and Archive and Graylog. Over the next few minutes, I’ll share my results and a provide a good analysis of the pros and cons of each that just might help you tackle this problem on your own network.
Splunk: Excellent Product, High Price
The first product most IT professionals think of when they hear the buzz words ‘Big Data’ is probably Splunk. Splunk is the largest and most successful of the commercially available data indexing products, and performs its job very well. Messages can be sent directly to Splunk from hosts or a forwarder, or Splunk can be set up to read straight from your syslog files. This information is indexed into ‘Buckets’ and stored for easy recall. Hot buckets store the most current messages, warm buckets the less current, and cold buckets for the older, soon to be erased messages.
Searching and alerting are very well implemented, allowing users to conduct real time searches or schedule searches that are either recurring or happen at a specified time each day. These searches results can be used to create alerts for things such as security issues or messages indicating potential problems. Correct use of saved searches and alerts greatly decreases downtime.
Splunk’s configuration is easy enough to customize and based upon a series of *.conf files. To make a change, you just have to navigate to the appropriate configuration file and edit or insert the correct line. The primary programming language used in Splunk is Python, which counts as another positive due to Python’s widespread use. Their are two major downsides to Splunk that may keep some away: High cost and slow support ticket resolution. Splunk licenses for large environments easily climb above low 5 figures. Support wise, while they have always solved my issues, it has sometimes taken longer than expected to receive updates.
Graylog 2
Graylog is an open source package that claims to perform the same functions as Splunk. I was somewhat skeptical of this, and while my skepticism was well warranted, Graylog did have some pleasant surprises to offer. As far as behind the scenes is concerned, Graylog is written in Java and its web interface is written in Ruby on Rails. I am not a huge fan of Java, but your opinion may differ, but Ruby on Rails works well enough. Graylog does not have the ability to read directly from syslog files, instead you need to send your messages directly to Graylog. Again, this is not a show stopper, but is less convenient.
Also, Graylog relies upon elasticsearch and mongodb, two services that I use sparingly. Dependence on other servers leave Graylog subject not only to its own possible issues, but also to problems originating from elasticsearch and mongodb.
Graylog does have a very friendly and fairly intuitive web interface. You can perform searches of your data just as in Splunk and with similar search functions. Alerting is also a possibility in Graylog, but the alert emails were less than informative on their own, providing only a reference to search results contained on Graylog’s web interface. Overall, Graylog is a fine Splunk alternative if you are willing to give up some minor functionality for a drastic cost savings.
Enterprise Log Search and Archive
ELSA is offered by Google, so already most IT professionals will have an opinion one way or the other. ELSA is heavily based upon Splunk, but is focused on speed more than fancy dashboards. Depending on the average skill of the users that will be interacting with ELSA this is either a huge positive or a large downside. Installation is handled by a shell script, and running this script gets you from zero to accessing ELSA’s web interface in less than 30 minutes. Speed and functionality over design is exactly what you get with Enterprise Log Search and Archive, and there is little to no design flair to be found.
Scheduled reports, searches and alerting are some of the basic features, and ELSA works with Syslog-NG, MySQL, and LDAP/AD for authentication. There is also a plugin architecture available, but overall its best quality is the ability to handle huge numbers of logs, over 30,000 per second to be exact. There were two drawbacks that may keep you from considering this as a solution to your data processing needs. For one, the community surrounding this product is much smaller than that of Graylog and Splunk. Also, I did have to use a cleanly installed CentOS machine in order to get the ELSA installation script to run without extra package installation and configuration. Lastly, users who want or need extensive dashboard functionality will be slightly disappointed, as ELSA’s dashboards are not quite up to par with Splunk.
Conclusion
Overall, the product that offers the most functionality and best support is Splunk. Splunk is easy to use, has lots of available plugins and customizations, and impressive dashboard functionality.
Commercial support is also a huge bonus to many IT professionals who do not wish to add another application to their list of managed services. Graylog and Enterprise Log Search and Archive are both viable alternatives, and each have their own positives. There are three factors that should be weighed heavily before deciding which product will handle your syslog data: If you have the time and are willing to provide support, or have a knowledgeable and skilled user base, Enterprise Log Search and
Archive or Graylog are worth a look. However, if you want something that comes with minimal support responsibilities and can be unleashed on relatively unskilled user bases, Splunk is the product for you.