Monitoring Console

Question 1

Where should the monitoring console be running?

Answer 1

Should be set up on dedicated host

Question 2

Where should the Monitoring Console never be set on?

Answer 2

Production Search Heads
SHC members
Indexers
Deployment server with more than 50 clients
Deployer sharing with CM

Question 3

Monitoring console should be a member of:

Answer 3

a Member of all indexer cluster

Question 4

Monitoring Console should be a search peer of

Answer 4

• All Search Heads (clustered or non-clustered)
• All indexers that are not members of clusters
  All other enterprise instances (deployer, deployment server, license master)

Question 5

Which index contains Operational Data?

Answer 5

_internal

Question 6

Which index contains Resource Usage

Answer 6

_introspection

Question 7

Who is able to access the Monitoring Console?

Answer 7

The MC is only visible to users with an administrative role

Question 8

How does the MC work?

Answer 8

MC utilizes RESTfu(snapshot) and log(historical) searches to check system health.

Question 9

True or False:

MC is considered a single-purpose monitoring box for keeping track of the state of the Splunk deployment

Answer 9

True

Question 10

How would you add a MC?

Answer 10

Add the MC as a search head of the cluster

Question 11

Does the MC directly connect to universal forwarders? What about Heavy Forwarders?

Answer 11

UF: NO
HF: YES

Question 12

Forwarder Monitoring relies upon log-based metrics and saved searches. On which indexes are these logs stored?

Answer 12

_internal provides info about operational things

_introspection provides info about resource usage

Question 13

How are roles for Splunk instances determined?

Answer 13

An instance is queried for a list of its current roles. The MC focuses searches/dashboards based upon ITS OWN VERSION of the instance’s “role”

Question 14

If the MC is peere to a bunch of newly created nodes before their full configuration has been provided, what is it identified as?

Answer 14

indexer

nodes may identify as “indexer” that are not actually doing so, e.g. search heads, before they are given an outputs.conf.

Question 15

Is there a forwarder role in the MC?

Answer 15

There is no forwarder role. Forwarder information is gathered about them by examining their logs.

Question 16

What is the REST endpoint to view server roles?

Answer 16

rest /services/server/info

Question 17

What is the MC role process?

Answer 17

Indexing locally? INDEXER

Other hosts searching it? Search Peer

Splunk started with a serverclass.conf? Deployment Server

Bundle contents created from $SPLUNK_HOME/etc/hcluster/apps? SHC Deployer

Question 18

How do you convince a host that it is an SHC deployer?

Answer 18

Run this command:

$SPLUNK_HOME/bin/splunk apply shcluster-bundle -action stage

Question 19

How do you take away the SHC deployer role?

Answer 19

delete this directory and restart splunk:

$SPLUNK_HOME/var/run/splunk/deploy

Question 20

What are the default search groups provided?

Answer 20

dmc_group_cluster_master: any CMs in the environment
• dmc_group_deployment_server: deployment server
• dmc_group_indexer: any full instance not having an outputs.conf
• dmc_group_kv_store: hosts, typically SH, running KV store
• dmc_group_license_master: any full instance with “self” as the license master
• dmc_group_search_head: any host that is peered to another
• dmc_group_shc_deployer: any SHC deployers in the environment

Question 21

Where are roles configured for the Monitoring Console?

Answer 21

distsearch.conf

Question 22

What are the provided roles for Clustering?

Answer 22

dmc_indexerclustergroup_

• All member of an indexer cluster (CM and indexers)
• If a label is provided, it will be show instead of the GUID of the CM

dmc_searchheadclustergroup_

• all members of a search head cluster
• If a label is provided, it will be show instead of the GUID of the SHC

Question 23

What are the three pieces of information needed for a custom server group?

Answer 23

• Name of the server group
• List of servers
• Default state (true or false)

Example:
[distributedSearch: NYC]
default = false
servers = 192.168.1.1:8089, 192.168.1.2:8089

Question 24

What field can you use to search a mc group on the monitoring console?

Answer 24

splunk_server_group=

Question 25

How do you identify that a node was misconfigured in the monitoring console?

Answer 25

check the search.log for the string “optimized out”

Question 26

Are Health Checks extensible?

Answer 26

Yes

Question 27

Can checks provided in checklist.conf be RESTful or log-driven

Answer 27

Yes