Splunk diag

This post covers splunk diag, a diagnostic tool, which collects the basic information about your splunk instance. To know more about this tool please continue reading this post.

What is Splunk diag?

Sometimes, to help diagnose a problem, Splunk Support might request you to generate a  diagnostic file on your splunk instance. This diag files give Support insight into how an instance is configured and how it has been operating up to the point when the diag command was issued to the instance.

About splunk diag

Generating a diagnostic file, also known as diag, whether in Splunk Web or at the command line, collects the basic information about your Splunk platform instance, including Splunk platform configuration details. It gathers information, such as server specs, OS version, file system, and current open connections, from the machine running the Splunk platform. From the Splunk platform instance, it collects the contents of $SPLUNK_HOME such as app configurations, internal Splunk log files, and index metadata.

NOTE: Diags do not contain any of your indexed data. You can examine the diag file to ensure that no proprietary/confidential data is included. In some environments, custom app objects, like lookup tables, can contain sensitive data. Before you send any files or information to Splunk Support, verify that you are comfortable sending it. The Splunk platform tries to exclude sensitive information from any output from the commands below.

Generating diags using Splunk GUI(Web)

You can select multiple instances in your deployment to generate diags for, and which configurations to use. You can recreate a diag using settings you chose in the past. You can manage previously created diag bundles, including deleting files, viewing the status of diag creation, and downloading diags to your local machine. After you have diags on your local machine, you can upload them to an existing Support case.

To generate and view diags in Splunk Web, you need the get_diag capability.

Follow these steps to access the Splunk Web diag generation page.

1. Log into Splunk Web on a search head or monitoring console in your deployment.
2. Click Settings > Instrumentation
3. Select New Diag

The below images are for your further reference,

Select from the options, the instances for which you want to generate a diag, and click next 

You can decide which diag components to include and exclude(optional), also modify other options as per your requirement, and click create

Image showing the created diag file

Which instance to use, to generate diags?

You can Generate diags in Splunk Web for a remote instance that has at least one of the following server roles:

• A search head that is the only search head in a deployment.
• A clustered search head.
• A clustered indexer.
• An indexer cluster master.

There are chances that you cannot generate a diag for your remote instances from your search head, try again from your monitoring console. Since the monitoring console in distributed mode adds all instances as search peers to the instance hosting the monitoring console.

Choosing the files to be included in your diags

Choose which directories are included with components. By default, all components are included but REST. You can adjust the thoroughness with which some components are collected by using additional options. Also, Components and options you select in Splunk Web override any local settings.

Diags are stored in $SPLUNK_HOME/var/run/diags .If you generate a diag on a remote instance, the diag artifacts are transferred to the search head that invoked the diag.

Run diag at the command line

Be sure to run diag as a user with appropriate access to read Splunk files.

The basic syntax to run diag at the command line is as follows.

On Linux:

$SPLUNK_HOME/bin
#./splunk diag

On Windows:

%SPLUNK_HOME%\bin
splunk diag

Below image depicts the successful creation of diagnosis file,

The generated diag file can be found in the $SPLUNK_HOME

Exclude files from diag

The Splunk platform can be told to leave some files out of the diag. One way to do this is with path exclusions. In Splunk Web, use the Exclude patterns option. At the command line you can use the ‘exclude’ flag. For example:

splunk diag --exclude "*/passwd"

you can use multiple ‘excludes’ :

splunk diag --exclude "*/passwd" --exclude "*/lookups/*"

Files excluded by the ‘exclude’ feature are listed in excluded_filelist.txt in the diag bundle to ensure Splunk Support can interpret the diag.

For example, the most commonly requested files collected are log files and configuration files only for initial analysis. To collect only those two components, use:

$SPLUNK_HOME/bin/splunk diag --collect=log,etc

Run the diag command on a remote instance

In situations when you can’t access a machine from your deployment, you can still generate and gather diag from that instance(except universal forwarders). First, make sure you have the get_diag capability. The admin role has this capability by default. You also need to have the login credentials for the remote server.

The syntax is:

splunk diag -uri https://<host>:<mgmtPort>

Hope!! This post has helped you in understanding and generating the diag file for your splunk instance(s).

Happy Splunking!!