Monitoring Console

Question 1

Where can the MC be installed on?

Answer 1

• Dedicaded SH which is not a member of the cluster
• Deployer
• License master
• Cluster master (only if there are less than 30 indexes, 10 indexer or less than 100k buckets)
• Deployment server (only if less than 50 deployment clients)

Question 2

In which files does health checks live?

Answer 2

checklist.conf

Question 3

List two fields that a health check require to work properly

Answer 3

severity_level

title

Question 4

Which basic field needs to be renamed to make the health check work?

Answer 4

All fields which contain a host (e.g. host or splunk_server) needs to be renamed to instance (e.g. | rename splunk_server as instance) to make them work in the Monitoring Console.

Question 5

Can a Universal Forwarder be directly monitored with the MC?

Answer 5

No, the Universal Forwarder API only has limited options, therefore the MC can not pull data from it.

Instead, the MC uses the internal log files which the Universal Forwarder forwards to the Indexer tier

Heavy Forwarder can be fully monitored since the API has no limitations.

Question 6

In order to make the MC fully working in a distributed deployment, which important step needs to be done on all non-indexer systems?

Answer 6

Forward internal logs to the indexer tier

Question 7

Which step is required to make the MC see all the attached Splunk components? (this question does not focus on forwarding the internal log files).

Answer 7

You must add each instance that you want to monitor to the monitoring console as a search peer, regardless of the server role.

Exception:
Indexers that are part of an indexer cluster (only the CM is added as search peer)

Question 8

What is a cluster label?

Answer 8

It is an optional flag which can be set for each cluster (SHC or IDXC). The idea behind it is to group cluster and give them a name, e.g SHC_Berlin.

It is highly recommended to do that, the MC also recognizes then which instances are associated and groups them together.

Question 9

How does the MC recognizes if the role is ‘Deployment Server’?

Answer 9

It recognizes it if the MC finds a serverclass.conf

Question 10

Forwarder monitoring does not work through API, which indexes are searched by Splunk to analyze forwarder?

Answer 10

_introspection and _internal

Question 11

From which data sources is the MC pulling to populate its dashboards?

Answer 11

MC utilizes REST for snapshots and event data for historical searches.

Question 12

How does the MC recognizes if a system is an indexer?

Answer 12

If the system is indexing locally

Question 13

How does the MC recognizes if a system is a Search Head?

Answer 13

By checking if distsearch.conf does contain search peers

Question 14

How does the MC recognizes if a system is a deployer?

Answer 14

If apps are located in /etc/shcluster

Question 15

List the pre-defined MC roles

Answer 15

• dmc_group_cluster_master: any CMs in the environment
• dmc_group_deployment_server: deployment server
• dmc_group_indexer: any full instance not having an outputs.conf
• dmc_group_kv_store: hosts, typically SH, running KV store
• dmc_group_license_master: any full instance with “self” as the license master
• dmc_group_search_head: any host that is peered to another
• dmc_group_shc_deployer: any SHC deployers in the environment

Question 16

How do you identify that a node was misconfigured in the monitoring console?

Answer 16

Check the search.log for the string “optimized out” or review the MC Settings -> General Setup and check if all components are visible and if they have the correct server role assigned.

Question 17

What are the built-in MC cluster roles?

Answer 17

1) dmc_indexerclustergroup_

• All member of an indexer cluster (CM and indexers)
• If a label is provided, it will be displayed instead of the GUID of the CM

2) dmc_searchheadclustergroup_
- All members of a search head cluster
- If a label is provided, it will be displayed instead of the GUID of the SHC

Question 18

What are the three pieces of information needed for a custom MC server group?

Answer 18

• Name of the server group
• List of servers
• Default state (true or false)

Example:
[distributedSearch: NYC]
default = false
servers = 192.168.1.1:8089, 192.168.1.2:8089

Question 19

Who is able to access the Monitoring Console?

Answer 19

The MC is only visible to users with an administrative role

Question 20

What is the REST endpoint to view server roles?

Answer 20

rest /services/server/info

Question 21

How are roles for Splunk instances determined?

Answer 21

An instance is queried for a list of its current roles. The MC focuses searches/dashboards based upon ITS OWN VERSION of the instance’s “role”

Question 22

Where should the Monitoring Console never be set on?

Answer 22

1) Production Search Heads
2) SHC members
3) Indexers
4) Deployment server with more than 50 clients
5) Deployer sharing with CM