1.0 Deploying Splunk Flashcards

(112 cards)

1
Q

What is an SVA

A

Proven reference architectures for stable, efficient, and repeatable deployments. Guidelines and certified architectures to ensure that their initial deployment is built on a solid foundation.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

Why and how does Splunk grow from standalone to distributed?

A
  • Ingests more data
  • Distributed search across indexers
  • Adding high availability
  • Dedicating LM and CM
  • Adding ES
  • SH Cluster for searching
  • Disaster Recovery
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

What does and doesn’t SVA provide?

A
  • Implementation choices (OS, baremetal vs. virtual vs. Cloud etc.).
  • Deployment sizing.
  • A prescriptive approval of your architecture.
  • A topology suggestion for every possible deployment scenario.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

What does and doesn’t SVA provide?

A

Does:
- Implementation choices (OS, baremetal vs. virtual vs. Cloud etc.).
- Deployment sizing.
- A prescriptive approval of your architecture.
Doesn’t
- A topology suggestion for every possible deployment scenario.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

What is HA? How can Splunk accomplish?

A

continuously operational system bounded by a set of tolerances

ex. IDX cluster. 1 node goes down - still send data to others
SHC - multiple SHs can look at the data.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

What is DR? How can Splunk accomplish?

A

Process of backing-up and restoring service in case of disaster.

  • Standby nodes - backed up copies of node managers
  • Multisite
  • SF and RF
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

What instances are suitable to become MC?

A
  • Dedicated SH that is has connectivity to entire environment.

NEVER INSTALL ON:

  • Prod (distributed) SH
  • Member of SHC
  • An IDX
  • A DS OR LM with > 50 clients
  • Deployer sharing with CM

https://docs.splunk.com/Documentation/Splunk/8.0.3/DMC/WheretohostDMC

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

How to configure MC for single or distributed environment?

A

Single:

1) In Splunk Web, navigate to Monitoring Console > Settings > General Setup.
2) Check that search head, license master, and indexer are listed under Server Roles, and nothing else. If not, click Edit to correct.
3) Click Apply Changes.

Distributed:
1) Log into the instance on which you want to configure the monitoring console. The instance by default is in standalone mode, unconfigured.
2) In Splunk Web, select Monitoring Console > Settings > General Setup.
3) Click Distributed mode.
4) Confirm the following:
The columns labeled instance and machine are populated correctly and show unique values within each column.
- The server roles are correct. For example, a search head that is also a license master must have both server roles listed. If not, click Edit > Edit Server Roles and select the correct server roles for the instance.
- If you are using indexer clustering, make sure the cluster master instance is set to the cluster master server role. If not, click Edit > Edit Server Roles and select the correct server role.
- If you are hosting the monitoring console on an instance other than the cluster master, you must add the cluster master instance as a search peer and configure the monitoring console instance as a search head in that cluster.
- Make sure anything marked as an indexer is actually an indexer.
5) (Optional) Set custom groups. Custom groups are tags that map directly to distributed search groups. You might find groups useful, for example, if you have multisite indexer clustering in which each group can consist of the indexers in one location, or if you have an indexer cluster plus standalone peers. Custom groups are allowed to overlap. For example, one indexer can belong to multiple groups. See Create distributed search groups in the Distributed Search manual.
6) Click Apply Changes.

If you add another node to your deployment later, click Settings > General Setup and check that these items are accurate.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

Why do server roles matter MC?

A

Server roles are used to create searches, reports, and alerts based off what server roles are specified.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Why do groups matter MC?

A

Groups are used to in order to correlate among similar instances. Single clusters etc.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

How are health checks performed on the MC?

A

Each health check item runs a separate search. The searches run sequentially. When one search finishes, the next one starts. After all searches have completed, the results are sorted by severity: Error, Warning, Info, Success, or N/A.

You are able to disable and enable certain health check items as needed as well as change their threshold.

The Health Check page lets you download new health check items provided by the Splunk Health Assistant Add-on on splunkbase.

Or you can create a new health check option.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

What authentication methods are supported by Splunk?

A

LDAP - can’t use if SAML is enabled
SAML and SSO
Native Splunk accounts (created locally/internally)
Scripted authentication

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

Describe LDAP concepts.

A

Standard for accessing AD creds and services.

LDAP directories are arranged in a tree-like structure. The information model is based on entries:
- The distinguished name (DN) is based off attributes
cn=admin1,ou=people,dc=splunk,dc=com

Tree structure with cn at bottom and dc at top.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

Describe LDAP configs.

A

https://docs.splunk.com/Documentation/Splunk/8.0.3/Security/ConfigureLDAPwithSplunkWeb

Authentication.conf

  • host =
  • port =
  • groupBaseDN =
  • groupMemberAttribute =
  • groupNameAttribute =
  • realNameAttribute =
  • userBaseDN =
  • userNameAttribute =
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

List SAML and SSO options

A

Review Slide Deck

Configure:

1) download the Splunk Service Provider Metadata file
2) Import the IdP metadata into Splunk

  • SSO
  • SLO (optional)
  • IdP cert path
  • IdP cert chains
  • Replicate certs
  • Issuer ID
  • Entity ID
  • Sign AuthnRequest
  • Verify SAML Response
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

Roles in Splunk?

A

admin – this role has the most capabilities assigned to it.
power – this role can edit all shared objects (saved searches, etc) and alerts, tag events, and other similar tasks.
user – this role can create and edit its own saved searches, run searches, edit its own preferences, create and edit event types, and other similar tasks.
can_delete – This role allows the user to delete by keyword. This capability is necessary when using the delete search operator.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

How can roles secure data?

A

Restrict index access capability by roles.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

How can data be ingested by indexer?

A
Monitored
Batch
Script - opt/spl/etc/apps/bin/
Modular inputs
Syslog
Network inputs - http
Splunk tcp
REST
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

How does Splunk communicate with Splunk?

A
Ports:
8000 - web
8089 - mgmt
8088 - HEC
9997 - tcp listening
9887 - replication indexers
 - shc replication
8191 - kv store
514 - network input
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q

Troubleshoot data inputs - monitor:

A

TailingProcessor for monitor inputs:

splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus

Will show:

  • what files it found
  • whether they matched the wild card
  • how far into the file it read

./splunk list monitor
list of currently monitored inputs.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
21
Q

Troubleshoot data inputs - conf files:

A

splunk btool conf-name list –debug

gives on-disk configs

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
22
Q

What are examples of indexing artifacts

A

rawdata - compressed form (journal.gz)
Time Series Index (tsidx) - indexes that point to raw data

Buckets - directories of index files organized by age

  • splunk_home/var/lib/splunk/myindex/db
  • bucket locations defined in indexes.conf
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
23
Q

Describe event processing

A

Splunk processes incoming data, stores result events in index

When Splunk indexes events:

  • configures character set encoding
  • configures line breaking for multi line events
  • identifies timestamps
  • extracts fields
  • segments events
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
24
Q

Name and describe the data pipelines

A

Parsing:
UTF-8 - Splunk will attempt to apply UTF-8 encoding to data
Line Breaker - Splunk will split data stream into events using default line breaker
Header - Splunk can multiplex different data streams into one “channel”

Merging/Aggregator:
- Splunk will merge lines separated by line breaker into events
Line breaking v line merging: LINE_BREAKER & SHOULD_LINEMERGE
Determining Time: TIME_PREFIX, TIME_FORMAT, MAX_TSL, DATETIME_CONFIG

Typing:
Regex replacement – performs any regular expression replacements called on in props.conf/tranforms.conf
Annotator – extracts the punct field

Indexing:
TCP/Syslog out – sends data to a remote server
Indexer – writes the data to disk

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
25
Describe the underlying text parsing process:
Splunk breaks events into segments at index and search time. Index time - uses segments to create lexicons - which point to where on disk Search time - uses segments to split terms
26
Describe Times Series Index
Optimized to execute arbitrary boolean keyword searches and return millions of events in revers time order Inverted index - allows fast full text searches - maps keywords to locations in raw data 2 components: lexicon value arrays containing info about events
27
What is a lexicon TSIDX?
Each unique term from the raw event has its own row. Each row has a list of events containing the given term. ``` Looks like a table Term. Postings List (event #) bacon. 0 beets. 2, 5, 7 crab 0, 1, 9 ```
28
What is a lexicon TSIDX?
Each unique term from the raw event has its own row. Each row has a list of events containing the given term. ``` Looks like a table Term. Postings List (event #) bacon. 0 beets. 2, 5, 7 crab 0, 1, 9 ```
29
What is a TSIDX Value Array?
Each raw event has its own row Each row contains metadata including the seek address ``` Looks like a table #. Seek address. _time. host. source. sourcetype. ```
30
What is data retention?
Management of the storage of indexed data. Allows Splunk to expire old data and make room for new data. *Most restrictive rule wins - prompts a change in state
31
What are retention bucket controls?
maxDataSize - max size for hot bucket maxWarmDBCount - max num of warm buckets maxTotalDataSizeMB - max size of an index -> cold to frozen frozenTimePeriodInSecs - max age of bucket -> cold to frozen homePath.maxDataSizeMB - max size for hot/warm storage coldPath.maxDataSizeMB - max size for cold storage maxVolumeDataSizeMB - max size for volume maxHotBuckets - max number of hot buckets timePeriodInSecBeforeTsidxReduction - how long indexers retain tsidx files
32
Bucket Controls - Volumes
- Allows you to manage dis usage across multiple indexes - Allows you to create max data size for them - Typically separates hot/warm from cold storage - Take precedence over other bucket controls
33
What is the bucket control precedence?
"Most restrictive rule wins" - Oldest bucket will be frozen first - Age determined by most recent event - Hot buckets are measured by size but are exempt from age controls
34
What are typical storage multiplication factors?
.35 SF and .15 RF Review formulas
35
What is the search head dispatch search sequence? (same with stats)
User Query -> SH -> Check search quota -> Check disk -> dispatch directory -> indexers
36
Sequence of events for searching for events in Splunk?
SH sends search request: 1) request is received by indexer 2) Indexer checks disk 3) Creates dispatch directory in var/run/spl/dispatch 4) configures subsystem - initializes configs (props, transforms etc) using bundle identified by SH 5) Implement time range - finds buckets in range 6) uses bloom filters to minimize resource usage 7) checks the lexicon- find events matching keywords within the lexicon (tsidx files) 8) Use results returned to find the event offsets within raw data from the values array 9) uncompresses raw data - uncompresses appropriate raw data to get the _raw event 10) Process field extractions 11) Send results to the search head
37
What metrics does the job inspector provide?
Time spent in search Time spent searching the index Time spent fetching data Workload undertaken by search peers
38
What is the REST endpoint to find job properties?
REST /services/search/jobs
39
Search job inspector components?
Header Execution costs - categories listed as command.* and command.search.* reflect various phases of the search process Search job properties - Bundle - Can summarize - Create time - Cursor time - diskUsage - dropCount
40
What are the type of search commands?
``` Generating Streaming Transforming Centralized (stateful) streaming Non-streaming ```
41
Characteristic of a generating search command?
Invoked at beginning of a search. Does not expect or require an input | search is implied
42
Characteristic of a transforming search command?
Generates a report data structure Operate on the entire event set ex. chart, timechart, stats
43
Characteristic of a streaming search command?
Operates on each event individually Distributable streaming - run on indexers ex. eval, fields, rename, regex
44
Characteristic of a centralized (stateful) streaming search command?
Runs on SH | ex. head, streamstats
45
Characteristic of a non-streaming search command?
Fore the entire set of events to the SH (sort, dedup, top
46
Examples of a generating search command?
``` search datamodel inputcsv metadata rest stats ```
47
Examples of a streaming search command?
``` fields lookup rex spath where ```
48
Examples of a centralized streaming search command?
``` head eventstats streamstats tail transaction lookup local=t ```
49
Examples of a transforming search command?
``` append chart join stats table timechart top ```
50
How does Splunk minimize searching?
Parsed map-reduce Job inspection -> Search job properties (2 parts): remoteSearch - done on indexer reportSearch - done on SH
51
Splunk tstats search sequence?
1) request received from SH on indexer 2) checks disk 3) creates remote dispatch folder 4) configs subsystem 5) checks time range 6) bloom filter 7) splunk lexicon 8) results to SH No raw data, no field extractions, doesn't use results to offset in array
52
When to use sub-searches?
- Small result sets. Max of 10000 events. Max runtime of 60 sec - Certain commands require (join, set) - Used to produce search terms fr outer search aka find subset of hosts, determine time, craft main search string dynamically *subsearches always run first before main
53
When not to use sub-searches?
For subsearches that return many results - better to use stats or eval - typically subsearches take longer than main - GUI provides no feedback when subsearch runs
54
Best practice for maximizing search efficiency:
- filter early - specify index - utilize indexed extractions where avail - use the TERM directive if applicable - place streaming/remote commands before non-streaming - avoid using table, except very end. Causes data to be pushed to SH - Remove unnecessary data using | fields
55
Describe a deployment app:
An arbitrary unit of content deployed by the DS to a group of deployment clients - Fully developed apps (Splunkbase) - Simple groups of configs - Usually focused on a specific type of data or business need
56
Where are deployment apps store on DS and client?
DS: /etc/deployment-apps ---> Client: etc/apps
57
What is the DS?
A centralized config manager that delivers updated content to deployment clients. - units of contents know as deployment apps - operates on a "pull" model - clients phone home - DS does not have SHC or IDXC clients
58
DS capacity
2000 polls/minute (Windows) 10000 polls/minute (Linux) - Utilize the phoneHomeIntervalInSecs attribute in deploymentclient.conf For more thank 50 clients the DS should be on its own server.
59
App deployment process from DS:
1) Client polls at certain interval: Client X, architecture 2) Determine apps for client using serverclasses 3) List of apps and checksums sent to client 4) Compares remote and local lists to determine updates 5) Client downloads now or updated apps from DS 6) Client restarts if necessary
60
What is a client state on the DS?
The client records its class membership, apps, checksums. Client caches the bundle (tar archive) with the app content
61
App update procedure from DS?
App removed from DS, app removed from client If app update is found on DS, client will delete and download a new copy - Apps that store user settings locally will have those settings "erased" - crossServerChecksum uses checksum rather than modtime
62
Describe deployment system configs
Use base configs to provide consistency
63
Benefit of base configs
Fast initial deployment time Reusable, predictable, supportable Faster troubleshooting Common naming scheme
64
Base configs by deployment type:
Review
65
How is forwarder management maintained?
Serverclass: - allows you to group Splunk instances by common characteristics and distribute content based on those characteristics blacklist (takes precedence), whitelist, filter by instance type
66
Editing the serverclass.conf - what are the stanza levels?
[global] - global level [serverClass:serverClassName] - Individual serverclass, can be multiple erverClass stanzas -one for each serverClass [serverClass: serverClassName:app:appname] - app within the server class. Used to specify apps the serverclass applies to - one for each app in the serverClass
67
Serverclass non-filtering attributes?
``` repositoryLocation stateOnClient - only thing enabled by default restartSplunkWeb restartSplunkd issueReload ```
68
Types of DS scaling
Horizontal - all DS are peers - all DS are on same level - all DS respond to clients Tiered - primary DS that servers other DS (parent/child) - any peers can respond to client
69
Why would you need more than 1 DS?
Multiple regions More than 2000/10000 clients Network Segregation HA is required Load balancer if too many clients are phoning home
70
What do you need to set in tiered DS? child and parent
child: deploymentclient.conf - repostitoryLocation= $SPL_HOME/etc/deployment-apps serverRepositoryLocationPolicy = rejectAlways child and parent: serverclass.conf - crossServerChecksum = true
71
What are the components of an indexer cluster?
Master node - a single node to manage the cluster Peer nodes - to index, maintain, and search the data Search heads - one or more to coordinate searches across peers
72
Purpose of the cluster master:
- Validates config settings before sending to the indexers - Monitors indexer peers and attempts to migrate node failures - Acts as a single point of contact for the SHs/MC
73
Describe the indexer cluster communication sequence:
1) Indexers stream copies of their data to other indexes 2) Master node coordinates activities involving search peers and search head 3) Forwarders send load-balanced data to peer nodes 4) Indexers send search results to SH
74
Steps in deploying indexer cluster:
``` 1 - identify requirement 2 - install Splunk Enterprise on instances 3 - enable clustering 4 - complete peer node configuration 5 - forward master node data to peers ```
75
What requirements are needed when determining your idxc?
``` DR and failover needs single site v multi site RF SF Quantity of data indexed and search load ```
76
Threshold on RF?
Do not set = # of indexers bc cluster will not be able to handle failures
77
Enable the master node CLI and .conf for IDXC:
./splunk edit cluster-config -mode master -replication_factor -search_factor -secret your_key -cluster_label cluster1 ``` server.conf [clustering] mode=master replication_factor = search_factor = pass4SymmKey = pwd cluster_label = label ```
78
Enable the peer node CLI and .conf for IDXC:
./splunk edit cluster-config -mode slave -master_uri https://:8089 -replication_port 9887 -secret your_key ``` server.conf [clustering] mode=slave master_uri = https://:8089 pass4SymmKey = pwd ``` [replication_port://9887] disabled = false
79
Enable the SH node CLI and .conf for IDXC:
./splunk edit cluster-config -mode searchead -master_uri https://:8089 -replication_port 9887 ``` server.conf [clustering] mode=searchhead master_uri = https://servername:8089 pass4SymmKey = pwd ```
80
Enable the SH node CLI and .conf for multi-site IDXC:
./splunk add cluster-master -master_uri https://:8089 -secret your_key server.conf [clustering] mode=searchhead master_uri = clustermaster:one, clustermaster:two [clustermaster:one] master_uri = https://:8089 pass4SymmKey = pwd [clustermaster:two] master_uri = https://:8089 pass4SymmKey = pwd
81
How can you guard against data loss?
Indexer acknowledgement - which retains a copy of the raw data until events are acknowledged
82
What is indexer discovery?
forwarders query the master node to get a list of all indexers in cluster
83
Where do apps to be pushed to indexers live on the master?
etc/master-apps --> etc/slave-apps
84
How to forward Splunk internal data to indexers?
outputs.conf [tcpout] defaultGroup = peer_nodes forwardedindex.filter.disable = true [tcpout:peer_nodes] server=server1:9997, server2:9997, etc
85
Index cluster upgrade procedure:
1 Cluster master 2 SH tier 3 Indexers Options: Tier by tier Site by site Rolling peer-by-peer (7.1.x+) Forwarders only should be upgraded opportunistically
86
Jobs of the CM
- Listens for cluster peers - adds to cluster when peer registers - Waits for RF number of peers to be satisfied before starting its functions - Listens for heartbeat of peers - if doesn't hear back for x amount of time peer it is marked down - Checks the manifest of buckets provided by all peers to determine if policy is met. If not fix up is triggered
87
Where does CM config bundle live?
SPL/var/run/splunk/cluster/remote-bundle bundle contains the contents from master-apps
88
What are the 3 parts of a bucket ID
index name local ID orig indexer GUID
89
Hot bucket lifecycle
1 idx notifies CM of new hot bucket 2 CM replies with list of streaming targets for replication 3 orig indexer begins replicating to new indexer
90
Warm bucket lifecycle
1 IDX notifies CM when bucket rolls to warm | 2 Rep target notified that bucket is complete and rolls to warm
91
Frozen bucket lifecycle
CM notified when freezes | CM stops doing fix up tasks
92
What happens if min free space is hit on indexer?
stops processing events | CM is notified and idx enters detention
93
What happens if indexer is in detention?
Auto: Stops indexing internal and external data. Stops replication. Doesn't participate in searches. Manual: Stops indexing external data (can be switched) Stops replication
94
What happens when CM goes down?
Cluster runs as normal as long as there are no other failures If peer creates hot bucket - it will try to contact master and fail. It will continue sending to previous peers SH will continue to function but will eventually begin to access incomplete data Forwarders continue to send to their list When it comes back up: Master starts fix up tasks Peers continue to send heartbeats and reconnect
95
How to replace CM?
No failover Must have standby Copy over server.conf and master/apps Ensure peer nodes can reach new master
96
What happens when indexer goes down?
Stops sending heartbeat. Master detects after 60s and starts fix up tasks Searches will continue but only provide partial results
97
What happens when indexer comes back up?
Starts sending heartbeat. Master detects and adds back to cluster Master rebalances cluster IDX downloads latest conf bundle from master (if necessary)
98
Multi-site clustering configs
A multi-site cluster requires additional configuration in the [clustering] stanza
 - The Cluster Master requires at least one host per site - Origin site is the site originating the data or the site where data first entered the cluster - origin : is minimum number of copies held on the origin site - site#: defines the minimum copies for that site - total: defines the total copies across all sites ``` server.conf [general]
 site = site2 [clustering]
 mode = master multisite = true available_sites = site2, site8, site44
 site_replication_factor = origin:2, site8:4, total : 8
site search factor = origin:1, site8:l, total : 4 constrain_singlesite_buckets=false ```
99
Single-site to multi-site characteristics:
After migration from single to multi: - cluster holds both sing and multi buckets - buckets created with marker in journal.gz indicating site origin - All SHs and CM are required to declare site membership - Indexers only return data if their "primary" status matches the requested site
100
Indexer migration to new indexers
1 install new indexers 2 add new indexers to cluster and ensure they receive common configs 3 Decommission and remove old indexers from master list by removing one at a time to allow bucket fix up to complete (selectively fail node) ./splunk offline --enforce-counts 1) Install new Indexers 2a) Bootstrap indexers to join the cluster as peers 2b) Ensure new indexers receive common configuration - Distribute files and apps with the configuration bundle - Common Files: indexes . conf, props . conf, transforms . conf 3a) Prepare to decommission old indexers - Point forwarders to new indexers - Put old indexers into detention 3b) Decommission old indexers (one at a time) - Run command splunk offline --enforce-counts - Wait for indexer status to show as GracefulShutdown in CM Ul - Repeat for remaining indexers - CM will fix / migrate buckets to new hardware 3c) Remove the old peer from the masters list
101
Upgrade CM
stop master upgrade using normal Splunk Enterprise procedure start the master
102
Upgrade search tier
stop all SHs Upgrade using normal procedures * if integrated with IDXC - upgrade one member and make it the captain - upgrade additional members one by one - upgrade deployer start SH
103
Upgrade IDX tier
- on CM enable maintenance mode to prevent unnecessary fix-ups - stop indexers - upgrade normally - start indexers - on CM disable maintenance-mode Or searchable rolling restart on indexers
104
Manage SHC
Deployer - pushes out apps | SHs replicate knowledge object
105
SHC Deployment
1) identity reqs 2) set up deployer 3) Install Splunk instance 4) initialize cluster members 5) bring up cluster captain 6) perform post-deployment set up
106
Deployer set up config:
server.conf [shclustering] pass4SymmKey = pwd shcluster_label = label
107
How to initialize SHC members
splunk init shcluster-config - auth usr:pwd -mgmet_uri servername:port -replication_port -replication_factor -conf_deploy_fetch_url :<8089> -secret key -cluster_label label
108
SHC benefits
- Horizontal scaling for increased capacity - HA for scheduled search activity - Centralized management of baseline configs - Replication of user generated content for consistent user experience
109
What does SHC captain do?
Coordinates replication of artifacts, maintains registry. - artifacts stored in /var/run/splunk/dispatch Pushes knowledge bundles to peers Replicates runtime config updates Assigns jobs to members based on relative current loads
110
What causes new captain election SHC?
Uses a dynamic election - election occurs: - current captain fails or restarts - network errors cause 1+ members to disconnect - current captain steps down after detecting that majority of members have stopped participating in the cluster - New captain is elected with majority vote
111
Captain Election Implications
Cluster consists of at least 3 members Captain election requires 51% If deploying across 2 sites - primary site must contain majority nodes bc network distruption will still allow election
112
Why/how to control captaincy SHC?
server.conf preferred_captain=true - to have one member always run as captain - you don't want captain performing ad hoc jobs - repair the cluster