Maximum concurrent searches Splunk can run by default

Maximum Concurrent Searches Splunk Can Run By Default

Hey Splunkers!

I hope you people are all really doing well and enjoying some quality time with Splunk and also not missing any opportunity to learn new things on Splunk. So today we are here to discuss another interesting topic on Concurrent searches.

When you have lots of panels in a single dashboard, sometimes you may see a message like waiting for queued job to startor in the messages section you may find a warning like The maximum number of concurrent historical searches on this instance has been reached.”

1

 If you are also getting this kind of messages too often but you don’t know how to tackle it or why this is coming so frequently, then this blog is for you.

Why Is This happening?

So let’s start with the fundamental cause of this message. Concurrent Searches means that at a same time how many consecutive searches you can run. This message means that you are attempting too many searches at once than your Splunk server can handles / supports.

Now how can you determine how many concurrent searches one can run at a one time?

Equation is pretty simple, it’s described below.

Number of concurrent searches = (max_searches_per_cpu x num_cpus) + base_max_searches 
Number of concurrent searches = (1 * Nos. of CPU core) + 6

where , 1 is the default value for max_searches_per_cpu and 6 is default number for

You can also know about :  DEST_KEY=MetaData:Host

Base_max_searches, which we will discuss in the later half of this blog.

Example – Lets say you have a 4 core CPU system, so you can run {(1*4)+6}=10 concurrent searches.

Example:

I have a 2 core CPU, you can check it from Splunk itself.

Follow below steps to know your CPU core.

Settings > Monitoring Console

2

3

Then we will create a dashboard called testing” with 12 panels. For those panels we have written a simple query to run which is given below and we will run it for all time.

index=_internal 
|stats count

 After creating the dashboard just open that dashboard by clicking, Settings > User Interface > Views > and search for your dashboard in the filter section and open it.

 Then you will notice this.

4

Explanation:

Now as you can see out of those 12 panels 4 panel’s jobs are in  queue i.e. it’s showing waiting for queued job to startand if you click in the messages section you will find thisThe maximum number of concurrent historical searches on this instance has been reached.”

1

We are facing some issues right ??? But why this is happening !!!

According to the equation ( shown above ) our maximum concurrent searches limit are , (2*1)+6 = 8; that’s why out of 12 panels 8 panels are loading and the rest of the panel’s(4)  jobs are in queue.

You can also know about :  Re-index your data into Splunk

How can we tackle this?

Now the first and most easiest step to fix it, wait for the first 8 panels to complete their searching and then the next 4 panels will start loading.

Another step is by making changes in the limits.conf file. If you want to do this.

Go to the following path

$SPLUNK_HOME\etc\system\default and you can see limits.conf

Copy the whole file to $SPLUNK_HOME\etc\system\local  directory and open limits.conf

there you can find attributes like this.

max_searches_per_cpu = 1 (It will multiply by your CPU core, by default it is 1)
base_max_searches = 6 (default 6)

Now we will change the, max_searches_per_cpu = 2
Now save it and restart your Splunk server.

5

Now open the same dashboard you can see the changes.

6

Explanation:

Now my concurrent search is like,  (2*2)+6 = 10;

Because my, max_searches_per_cpu = 2, that’s why out of 12 panels 10 panels queries are loading and the rest of the panel’s(2)  jobs are in queue. After this you will still receive that error message but at least now you can run two panels query which were going in the queue, better than earlier.

This is how you can tackle this problem if you make it  max_searches_per_cpu = 3, then your concurrent searches will be 12, this how you can do it as per your requirement.

You can also know about :  How To Install Splunk On Linux Server?

How to Get Rid From This Problem:

  1. Now the first and most easiest step to fix it, wait for the first 8 panels to complete their searching and then the next 4 panels will start loading.
  2. Or you can make the changes in limits.conf, but which is not recommended that will create extra pressure on your CPU.
  3. Best way to fix this is take higher CPU cores.

I hope you all have understood this topic “Maximum Concurrent Searches Splunk Can Run By Default”

Happy Spluking !!

One comment

  1. Thanks for the article .. may i know what is the maximum number limit we can give for max_searches_per_cpu in limits.conf

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.