Data Collection Flashcards
Describe different ways data can be ingested by an indexer
- Universal Forwarder, Heavy Forwarder
- Monitor Files
- Scripted Inputs
- Network Inputs (TCP, UDP)
- HEC
- WindowsEventLog,admon,perfmon,regmon
- FirstInFirstOut (FIFO)
What firewall rules needs to be defined for Splunks applicationserver (default port 8065)
None. It is a loopback port only for internal communication.
List the types of Forwarder
- Universal Forwarder (smallest footprint)
- Heavy Forwarder (medium footprint)
- Light Forwarder [deprecated since v6] (smaller footprint)
What is the chunk size of data send from an Universal Forwarder to an Indexer?
A forwarder sends data in 64kb blocks (unparsed)
How do you make sure that data from a Universal Forwarder does not arrive truncated or trashed on the indexer tier?
Always configure EVENT_BREAKER and EVENT_BREAKER_ENABLE. The regex can be copied from LINE_BREAKER. This makes sure that the the Universal Forwarder does send properly broken data chunks to the indexer tier.
List the most common pre-trained sourcetypes
- Application Server (log4j, Websphere)
- Mailserver (sendmail, postfix)
- OS (Linux, Windows, OSX)
- Network (Cisco)
- Datebases (DB2, mysql)
- Webserver (access_combined, apache)
What is the fishbucket?
Splunk remembers in here what it has read
Is the _fishbucket index an real index?
It used to be a real index until version 3.x. It still lives in the $SPLUNK_DB directory but it has its own structure, based on a btree database.
How can you reset a file to be re-indexed again?
Stop splunk ------------------- ./splunk cmd btprobe -d $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db --file --reset ------------------- Start splunk
Describe steps to troubleshoot data inputs?
- Checking splunkd log
- Using btool (splunk btool –debug input list)
- Activating debugging
- Check permissions on the source file
- Check input format (e.g. binaries, non-utf8 conform)
- Check if file exists
- Check network connection
- Check CRC of the file
- Tailing processor can be checked with ./splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus
How do you clean the eventtdata from an index?
Stop splunk ------------------- ./splunk clean eventdata -index ( to clean all indexes, just drop off -index ) ------------------- Start splunk
What happens if you remove the fishbucket index on a Universal Forwarder?
It kicks of a process to re-index all eventdata. Be very careful to proceed with this step. Only recommended in a well planned scenario.
What is a oneshot?
Copy the file directly into Splunk. This uploads the file once, but Splunk Enterprise does not continue to monitor it.
You cannot use the oneshot command against a remote Splunk Enterprise instance. You also cannot use the command with either recursive folders or wildcards as a source. Specify the exact source path of the file you want to monitor.
This is a common method for PS consultants to test and validate props/transforms configurations.
What are the great eight?
EVENT_BREAKER_ENABLE = true EVENT_BREAKER = \r\n MAX_TIMESTAMP_LOOKAHEAD = 30 TIME_PREFIX = ^ TIME_FORMAT = %Y-%m-%d %H-%M-%S SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+) TRUNCATE = 100000
Why the great eight are recommended to use?
Because it improves the data process significant. Splunk gets everything what it needs to know to parse data. There is no need to detect the settings by itself, which redues the load. It is best practise to always use the great eight.
