Metrics Data Collection Via Collectd( Part – 2 )
In our earlier post we discussed about the metrics data in context of Splunk, click here to check out that post.
So we decided to demonstrate – How to Splunk metrics data from a linux server using collectd daemon.
About Collectd
Collectd is an open source Unix daemon, capable of gathering different kind of performance metrics from servers and network equipments.
Follow the below steps to start collecting and forwarding metrics data via collectd from your Unix servers to Splunk using HEC method.
Step-1 : Configuring Splunk to store the metrics data;
On your Splunk indexer, create a custom index with the “metrics” data type, here we have created a metrics index named ‘server_metrics’.
Step-2 : Configuring Splunk to receive the metrics data over HEC.
To know more about HEC tokens please click here.
Below is the screenshot showing our “Global Settings”, depending upon your requirement, decide whether you want to use SSL or not,
The screenshot below shows the token that we will use for HEC communication, please check our blog post related to HEC configuration, in case you need help in creating a token.
Installing and setting up collectd on a linux server to forward the server metrics to Splunk
Step-1 : Install collectd on the server from where you want to collect the metrics data.
You can select your preferred installation method depending on the distribution of linux you are using.
Please note that we are using the linux centos 7 distribution to show this demonstration,
When prompted enter “y” as input to proceed with the installation.
Step-2 : Make the required changes to the “collectd.conf”.
Depending on the way you download and install collectd.conf, the location of “collectd.conf” may vary slightly, in our case it is located in the /etc directory.
We decided to only collect the metrics data related to CPU, memory ,a network interface(eth 0) and load on the server for demonstration
You can download the version of conf file that we used for this demonstration from here.
Below are the changes that we made to the “collectd.conf” file,
NOTE: The plugin options are already present, you just need to
uncomment them to enable.
To collect CPU metrics,
LoadPlugin cpu
<Plugin cpu>
ReportByCpu true
</Plugin>
To collect memory metrics,
LoadPlugin memory
<Plugin memory>
ValuesAbsolute true
ValuesPercentage false
</Plugin>
To collect the load metrics,
LoadPlugin load
<Plugin load>
ReportRelative true
</Plugin>
To collect metrics related to interface,
LoadPlugin interface
<Plugin interface>
Interface "eth0"
IgnoreSelected false
ReportInactive true
UniqueName false
</Plugin>
Other configuration changes required for Operation,
Syslog loglevel,
LoadPlugin syslog
<Plugin syslog>
LogLevel info
</Plugin>
Writing the syslogs to a file “/var/log/collectd.log”, (for troubleshooting)
LoadPlugin logfile
<Plugin logfile>
LogLevel info
File "/var/log/collectd.log"
Timestamp true
PrintSeverity false
</Plugin>
Forward the logs over HTTP,
Loadplugin write_http
<Plugin write_http>
<Node "splunk">
URL "https://10.128.0.19:8088/services/collector/raw"
VerifyPeer false
VerifyHost false
Format "JSON"
Header "Authorization: Splunk 4ac9c143-f20e-4592-a51e-1956a3391fea"
Metrics true
Notifications false
StoreRates true
BufferSize 4096
LowSpeedLimit 0
Timeout 0
</Node>
</Plugin>
Important :
<Node "splunk"> - Any relevant naming convention,
here we have mentioned “splunk” for simplicity.
URL "https://10.128.0.19:8088/services/collector/raw" - IP address
of your Splunk Server on which token
is generated.
Header "Authorization: Splunk 4ac9c143-f20e-4592-a51e-1956a3391fea" -
Provide the Splunk token here.
NOTE: You need to replace the highlighted configurations as required for your version of implementation, depending on your process of “Collectd” installation you may need to download some of the plugins separately on your server. You need to install modules according to your OS. For example, on some Linux distributions collectd-write_http.x86_64 must be installed to use the write_http plugin.
Important : In a scenario when you have collectd clients and server, you need to configure the “network plugin” on the client and set up “write_http” plugin on the collectd server.
Step -3 : Start the collectd daemon,
systemctl start collectd
Step-4: Check the status of collectd daemon,
systemctl status collectd
Step-5: Enable the collectd service,
systemctl enable collectd
Now, you might get a permission denied issue as shown in the screenshot of Step-3, this happens due to “SELINUX” default settings, especially on the centos distributions while using non standard ports such as 8088, to resolve this follow the procedure below or skip to the next step,
Run the below command to see the SELinux status,
getenforce
You should get “Enforcing” as the output,
To change it to permissive run the command shown below,
setenforce 0
Value 0 corresponds to Permissive
Your issue should be resolved immediately.
To make the SELINUX settings boot persistence,
Navigate to /etc/selinux,
Edit config file,
# vi config
SELINUX = permissive
Save it and then reboot the system. It will be in Permissive state.
Step-6 : Verify if you are getting the metrics data to your index,
We hope this post was worth a read, keep visiting us for more exciting posts on Splunk.
Happy Splunking!!
