Translate

Sunday, September 15, 2013

Alternatif lain dari SPLUNK menggunakan FLUENTD



Free Alternative to Splunk Using Fluentd

Splunk is a great tool for searching logs, but its high cost makes it prohibitive for many teams. In this article, we present a free and open source alternative to Splunk by combining three open source projects: ElasticSearch, Kibana, and Fluentd.


ElasticSearch is an open source search engine known for its ease of use. Kibana is an open source Web UI that makes ElasticSearch user friendly for marketers, engineers and data scientists alike. Kibana supports the LogStash format, which in turn is supported by the fluentd-plugin-elasticsearch plugin. And of course, Fluentd is the open source log collector known for its scalability and flexibility.
By combining these three tools we get a scalable, flexible, easy to use log search engine with a great Web UI that provides an end to end solution from log collection to visualization, all for free!
In this guide, we will go over installation, setup, and basic use of this combined log search solution. The contents of this article were tested on Mac OS X Mountain Lion.

Table of Contents

Prerequisites

Java for ElasticSearch

First, we’ll make sure that we have Java version 6 or higher installed as a prerequisite for ElasticSearch.
$ java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06-451-11M4406)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01-451, mixed mode)

Ruby for Kibana

Next, we’ll make sure that we have Ruby version 1.9.3 or higher installed as a prerequisite for Kibana.
$ ruby --version
ruby 1.9.3p385 (2013-02-06 revision 39114) [x86_64-darwin12.2.1]
Now that we’ve checked for prerequisites, we’re now ready to install and set up the three open source tools.

Set Up ElasticSearch

To install ElasticSearch, please download and extract the ElasticSearch package as shown below.
$ curl -O https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.0.RC2.tar.gz
$ tar zxvf elasticsearch-0.90.0.RC2.tar.gz
$ cd elasticsearch-0.90.0.RC2/
Once installation is complete, start ElasticSearch.
$ ./bin/elasticsearch -f

Setup Kibana

To install Kibana, download it via the official webpage or clone its Github repository as shown below. Kibana is a Sinatra application.
$ git clone git://github.com/rashidkpc/Kibana.git
$ cd Kibana
$ bundle install --path vendor/bundle # "gem install bundle" is needed if you don't install bundle
Once installation is complete, start Kibana. You can modify Kibana’s configuration via KibanaConfig.rb.
$ bundle exec ruby kibana.rb

Setup Fluentd (td-agent)

In this guide We’ll install td-agent, the stable release of Fluentd. Please refer to the guides below for detailed installation steps.
Next, we’ll install the ElasticSearch plugin for Fluentd: fluent-plugin-elasticsearch.
$ /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch
We’ll configure td-agent (Fluentd) to interface properly with ElasticSearch. Please modify/etc/td-agent/td-agent.conf as shown below:
<source>
  type syslog
  port 42185
  tag syslog
</source>

<source>
  type forward
</source>

<match syslog.**>
  type elasticsearch
  logstash_format true
  flush_interval 10s # for testing
</match>
fluent-plugin-elasticsearch comes with a logstash_format option that allows Kibana to search stored event logs in ElasticSearch.
Once everything has been set up and configured, we’ll start td-agent.
$ sudo /etc/init.d/td-agent start

Setup rsyslogd

In our final step, we’ll forward the logs from your rsyslogd to Fluentd. Please add the following line to your /etc/rsyslog.conf, and restart rsyslog. This will forward your local syslog to Fluentd, and Fluentd in turn will forward the logs to ElasticSearch.
*.* @127.0.0.1:42185
Please restart the rsyslog service once the modification is complete.
$ sudo /etc/init.d/rsyslog restart

Store and Search Event Logs

Once Fluentd receives some event logs from rsyslog and has flushed them to ElasticSearch, you can search the stored logs using Kibana by accessing http://127.0.0.1:5601/ in your browser.


To manually send logs to ElasticSearch, please use the logger command.
$ logger -t test foobar
When debugging your td-agent configuration, using out_copy + out_stdout will be useful. All the logs including errors can be found at /etc/td-agent/td-agent.log.
<match syslog.**>
  type copy
  <store>
    # for debug (see /var/log/td-agent.log)
    type stdout
  </store>
  <store>
    type elasticsearch
    logstash_format true
    flush_interval 10s # for testing
  </store>
</match>

Dashboard


The upcoming Kibana 3 comes with a beautiful dashboard.

Conclusion

This article introduced using the combination of Fluentd and Kibana (with ElasticSearch) to achieve a free alternative to Splunk: storing and searching machine logs. The examples provided in this article have not been tuned. If you will be using these components in production, you may want to modify some of the configurations (e.g. JVM, ElasticSearch, Fluentd buffer, etc.) according to your needs.

Last updated: 2013-07-02 06:55:20 UTC
at
Labels: