SearchHead Clustering

Question 1

Which command does initiate a SHC on a SH?

Answer 1

./splunk edit shcluster-config -mgmt_uri https://sh02:8089 -replication_port 9777 -replication_factor 3 -secret mysecret

Question 2

What do you configure in the shclustering stanza for the mgmt_uri attribute?

Answer 2

The mgmt uri of the instance itself (FQDN)

Question 3

How do you configure a search peer to use a deployer?

Answer 3

./splunk edit shcluster-config -conf_deploy_fetch_url https://dep01:8089

Question 4

Is it requiered to set a deployer in the init configuration?

Answer 4

No, it is not requiered, but to use the deployer feature fully, a deployer needs to be set

Question 5

What is cluster label?

Answer 5

A cluster label is a tag or identifier to recognize the cluster in the MC. It should be set.

./splunk edit shcluster-config -shcluster_label SH_Cluster_Berlin

Question 6

After all the instances are properly initialized, a captain needs to be set. How do you do that?

Answer 6

./splunk bootstrap shcluster-captain -servers_list “https://sh1.example.com:8089,https://sh2.example.com:8089,https://sh3.example.com:8089,https://sh4.example.com:8089” -auth admin:changed

NOTE: The server name in the servers_list must match the mgmt_uri which we defined on each instance during the initial setup

Question 7

What is the role of a captain in a SHC?

Answer 7

A captian is similar to what the CM in an indexer cluster does.

The captain orchestrates the SHC, which means replication of search artifacts, runtime changes, job scheduling, load management, proxy for adhoc and RT searches, alert management, deletation of search artefacts etc.

The captain also makes sure to meet the replication policy.

The replication happens every 5s through the configuration bundle.

Question 8

What is the secret behind the captain election?

Answer 8

The election is a simple process to make sure that there is always a ‘responsible manager’ of the SHC (fault tolerance system).

The process is based on an algorithm called RAFT. RAFT is a common algorithm used in many cluster technologies.

Every node starts up as a ‘follower’. Followers receive an RPC call (aka heartbeat) from the leader (the captain). Captains are elected for a set ‘term’ or period, which is monotonically increasing. The captain communicates with its follower nodes, and will monitor its term as captaincy. When the term expires it will offer itself for re-election. A node remains a follower until it stops receiving a leader heartbeat call. Once a term ends, the leader reverts to follower thus meaning that all the nodes stop receiving the leader heartbeat. *Alternatively the leader may fail to communicate prior to the term expiring, which will also result in the nodes no longer receiving a heartbeat. When a node stops receiving the heartbeat from the leader it will go into a state of candidacy for leadership. It will call out to the other nodes and request to be voted the leader. If you’re thinking ‘but that will cause split votes’ you’re correct! To deal with this raft introduces some arbitrary jitter into the nodes candidacy requests so they don’t all run at the same time. See the whitepaper for exact details. The other nodes respond to the election request and confirm election. Election is assigned once a node receives enough votes to achieve quorum. Quorum is set as >51%. So, in a 5 node SHC, 3 members needs to be fully working and able to vote.

Question 9

What is a short and brief description of what the deployer does?

Answer 9

The deployer is a deployment server dedicated for all searchheads in a SHC. It provides basic features to provide to all searcheads in SHC a basline configuration for apps and user configuration.

Question 10

What is the default push mode which the deployer uses to push out its baseline configuration the searchheads?

Answer 10

merge_to_default

Merges per default the default/local configs into the default directory on the SHC members (can be changed)

Question 11

If app changes happen on the deployer, how many tarballs will be pushed out maximum based on the merge_to_default setting?

Answer 11

1 tarball:

• The deployer merges /local and /default into /default and pushes one tarball to the peers

The amount of tarballs changes based on the push mode.

Question 12

What is the recommended setting for the deployer for built-in apps like the search app?

Answer 12

local_only

Question 13

How can the push mode set?

Answer 13

The push mode can be set globally in [shclustering] stanza on the deployer or it can be set for each app

Question 14

How do you set the push mode on an app level?

Answer 14

By creating a shclustering stanza in app.conf

Question 15

How do you prevent an app from being pushed out to the searchheads in a SHC?

Answer 15

By deleting the app from the configuration bundle directory, which lives under /etc/shcluster/apps.

The deletation does not work if the app state is set to disabled in the app.conf, even if you delete the app on the deployer. State needs to be set to enabled before the app can be deleted.

Question 16

There are two case where the deployer and captain work together, list and explain them

Answer 16

The deployer and captain work normally independetly, but there are two case where they work together.

Case 1)
The deployer never pushed the user baseline configuration directly to the searchheads, it always pushes the user baseline configuration to the captain first and from there (after untar them) the captain pushes it out the the searchheads.

Case 2)
The deployer pushes local app tarballs to the captain too, which untars them and replicates their settings to the members (only if the push mode is set to push local directory)

The captain is in general responsible for runtime change replication related to apps and user context. In addition, the captain is responsible for replicating the user basline configuration and local app (only when the push_mode sends out the local tarball) configurations received from a deployer.

Question 17

How does the deployer push mode work for metadata.default and metadata.local?

Answer 17

The deployer treats an app’s /metadata/local.meta and /metadata/default.meta files the same way that it treats the /local and /default directories.

For example, in the merge_to_default push mode, the deployer merges the app’s /metadata/local.meta and /metadata/default.meta files into a single /metadata/default.meta file and includes it in the default tarball

Question 18

Does the push mode also affect the user configuration?

Answer 18

No, the push mode only works for the app configuration. The user configuration is pushed all the time in the same way indirectly to the captain and from there to the searchheads.

Question 19

What happens in this scenario:

For example, say the cluster members have an existing file $SPLUNK_HOME/etc/users/admin/search/local/savedsearches.conf

And on the deployer, there’s the file $SPLUNK_HOME/etc/shcluster/users/admin/search/local/savedsearches.conf which has the same content as the cluster member with some additional extensions (means, the files are different)

Answer 19

This will result in a final merged configuration on the members, means the extended version on the deployer gets merged with the cluster member version and nothing will be gone.

Question 20

What is considered as a very large app tarball?

Answer 20

If you attempt to push a very large tarball (>200 MB), the operation might fail due to various timeouts. Delete some of the contents from the tarball’s app, if possible, and try again.

Question 21

Where is the staging area for deployers before they push the bundle to the searchheads?

Answer 21

$SPLUNK_HOME/var/run/splunk/deploy

Question 22

What is a possible workaround in case the captain election fails?

Answer 22

To set a static captain

On the member that you want to designate as captain, run this CLI command:
./splunk edit shcluster-config -mode captain -captain_uri : -election false

On each non-captain member, run this CLI command:
./splunk edit shcluster-config -mode member -captain_uri : -election false

Question 23

Is it possible that the captain runs adhoc searches only? If so, how does the configuration looks like?

Answer 23

You can designate the captain member as an ad hoc search head. This prevents members from running scheduled searches while they are serving as captain, so that the captain can dedicate its resources to controlling the activities of the cluster.

Make this change on all cluster members, so that the behavior is the same no matter which member is functioning as captain:

[shclustering]
captain_is_adhoc_searchhead = true

Question 24

Does a searchhead support manual detention?

Answer 24

Yes, a searchhead supports the state of manual detention through ./splunk edit shcluster-config -manual_detention on.

It is useful to do this while you do troubleshooting, diagnostics, upgrading etc.

Manual detention is used by default in searchable rolling restarts.

The searchead stops accepting all new searches from the search scheduler or from users. Existing ad-hoc and scheduled search jobs run to completion. New scheduled searches are distributed by the captain to search head cluster members that are up and not in detention. The searchhead still operates in other areas such as captain election.

Question 25

How do you apply the sh cluster bundle?

Answer 25

./splunk apply shcluster-bundle

Question 26

How do you preserve lookup files across app upgrades in a SHC?

Answer 26

Set the deployer_lookups_push_mode in app.conf to specify how the deployer handles lookup tables when upgrading apps. This setting determines the behavior of the -preserve-lookups option when you push the configuration bundle using the splunk apply shcluster-bundle command.

1) preserve_lookups (default)
This options preserves lookup tables only if you run the splunk apply shcluster-bundle command on the deployer with the -preserve-lookups flag set to true. If you do not set the -preserve-lookups flag to true, the populated lookup tables are overwritten on upgrade.

2) always_preserve
This option always preserves lookup tables, regardless of how you set the -preserve-lookups flag.

3) always_overwrite
This option always overwrites lookup tables with stub files, regardless of how you set the -preserve-lookups flag.

Question 27

What is the maximum network latency in a SHC before captain election will be not reliable anymore?

Answer 27

> 200ms