EX#2&3

Question 1

Tell me about yourself

Answer 1

x

Question 2

How big is my current environment?

Answer 2

x

Question 3

What is your experience with Regex?

Answer 3

x

Question 4

What resources do you use to write your Regex?

Answer 4

x

Question 5

How would you write a Regex statement to match an IP address?

Answer 5

x

Question 6

Explain how you exclude something from Regex?

Answer 6

x

Question 7

What do you know about props.conf?

Answer 7

x

Question 8

How many prop stanzas are needed if you have 8 data sources with 4 different sourcetypes and why?

Answer 8

32

Question 9

When onboarding data how would you bring the data in DEV into PROD?

Answer 9

Identify the data sources and data types that need to be migrated. This includes identifying the formats of the data, the locations of the data, and the frequency of the data updates.

Develop a migration plan. This plan should include the following steps:
Extracting the data from the DEV environment.
Transforming the data as needed. This may include converting the data to a different format, filtering the data, or enriching the data with additional data.
Loading the data into the PROD environment. This may involve creating new tables or indexes, or updating existing tables or indexes.
Validating the data in the PROD environment. This involves checking the data for accuracy and completeness.

Test the migration plan in a staging environment. This will help to identify any potential problems with the migration plan before it is executed in the PROD environment.

Execute the migration plan in the PROD environment. This should be done during a maintenance window to minimize the impact on users.

Monitor the migration process to ensure that it is completed successfully.

Question 10

Do you have any experience building technical add-ons or apps?

Answer 10

x

Question 11

Need upgrade Splunk to version 8-how would you upgrade

Answer 11

See EXAM 3 NOTES

Question 12

How would you troubleshoot Splunk configuration files?

Answer 12

Identify the configuration file that is causing the problem. You can do this by looking at the Splunk logs for errors.
Check the syntax of the configuration file. Make sure that all of the syntax is correct and that there are no missing or extraneous characters.
Verify the permissions on the configuration file. Make sure that the Splunk process has permission to read and write to the configuration file.
Check for duplicate entries in the configuration file. Make sure that there are no duplicate entries for any of the configuration settings.
Check for conflicting configuration settings. Make sure that there are no conflicting configuration settings in the configuration file.
Restart the Splunk service. Once you have made changes to the configuration file, you need to restart the Splunk service for the changes to take effect.

Use the Splunk btool command to validate your configuration files. The btool command will check your configuration files for errors and warn you of any potential problems.
Use the Splunk debug mode to troubleshoot configuration problems. The debug mode will provide you with more information about the Splunk configuration process.
Search the Splunk documentation and Splunk community forums for help with troubleshooting configuration problems.

Question 13

What tools have you integrated with Splunk?

Answer 13

Application performance monitoring (APM) tools: APM tools can be integrated with Splunk to collect and analyze application performance data. This data can be used to identify and troubleshoot performance problems, and to optimize application performance.
Security information and event management (SIEM) tools: SIEM tools can be integrated with Splunk to collect and analyze security event data. This data can be used to detect and respond to security threats.
IT infrastructure monitoring (ITIM) tools: ITIM tools can be integrated with Splunk to collect and analyze IT infrastructure data. This data can be used to monitor the health and performance of IT infrastructure, and to troubleshoot problems.
Business intelligence (BI) tools: BI tools can be integrated with Splunk to create dashboards and reports that provide insights into business data. This data can be used to make better business decisions.

APM tools: New Relic, Dynatrace, AppDynamics
SIEM tools: Splunk Enterprise Security, IBM QRadar, ArcSight ESM
ITIM tools: Nagios, Zabbix, Datadog
BI tools: Tableau, Qlik Sense, Microsoft Power BI

Question 14

How do you onboard data in your environment?

Answer 14

x

Question 15

Is your environment running onPREM or onCloud?

Answer 15

On-prem deployments give you full control over your data and infrastructure. You can choose the hardware and software that you want to use, and you can customize your Splunk deployment to meet your specific needs. However, on-prem deployments can be complex and expensive to manage.

Cloud deployments offer a number of benefits, including scalability, flexibility, and ease of management. Cloud providers handle the hardware and software maintenance, so you can focus on using Splunk to analyze your data. However, cloud deployments can be more expensive than on-prem deployments, and you may not have as much control over your data and infrastructure.

Question 16

Base apps vs Custom TAs used for onboarding vs Splunk based TAs/Apps-explain use and difference

Answer 16

Splunk Base apps
Base apps are apps that are included with the Splunk installation. They provide a set of common functionality, such as data parsing, searching, and reporting. Base apps are a good starting point for new Splunk users, as they provide a foundation that can be built upon to create custom apps and dashboards.

Custom Technical Add-ons (TAs)
Custom TAs are add-ons that are created by Splunk or by third-party vendors. They provide additional functionality to Splunk, such as the ability to collect data from new sources, parse data in new ways, or generate new reports. Custom TAs can be used to extend the functionality of Splunk to meet the specific needs of an organization.

Splunk-based TAs/Apps
Splunk-based TAs/Apps are apps that are created using Splunk’s built-in development tools. They provide a way to customize Splunk to meet the specific needs of an organization. Splunk-based TAs/Apps can be used to collect data from new sources, parse data in new ways, generate new reports, and create custom dashboards.

best practices for using base apps, custom TAs, and Splunk-based TAs/Apps:

Use base apps as a starting point. Base apps provide a good foundation that can be built upon to create custom apps and dashboards.
Use custom TAs to extend the functionality of Splunk. Custom TAs can be used to collect data from new sources, parse data in new ways, or generate new reports.
Use Splunk-based TAs/Apps to customize Splunk to meet your specific needs. Splunk-based TAs/Apps can be used to collect data from new sources, parse data in new ways, generate new reports, and create custom dashboards.
Test all TAs and apps before deploying them to production. This will help to prevent problems and ensure that your data is being processed correctly.
Monitor your Splunk environment for problems. This will help you to identify any problems with TAs and apps early on.

Question 17

How would you check the storage on your sever on the CLI?

Answer 17

x

Question 18

Check running processes on server, how?

Answer 18

x

Question 19

What are some methods you use to troubleshoot and solve issues in your environment?

Answer 19

x

Question 20

How would you download a TA from Splunkbase that you intend to deploy?

Answer 20

x

Question 21

What is your process for using Splunkbase-when you find and app what is your process for assessing the app and then using it?

Answer 21

My process for using Splunkbase is as follows:

Find an app. I can do this by browsing the Splunkbase catalog or by searching for a specific keyword or feature.
Assess the app. I review the app’s description, screenshots, and reviews to get a sense of what it does and how well it is rated. I also check the app’s compatibility with my version of Splunk and my operating system.
Install the app. If I am satisfied with the app, I install it on my Splunk environment.
Configure the app. Once the app is installed, I configure it to meet my specific needs. This may involve setting up data inputs, outputs, and transformations.
Test the app. Once the app is configured, I test it to make sure that it is working as expected.
Deploy the app. If I am satisfied with the app, I deploy it to production.
Here are some additional things that I keep in mind when using Splunkbase:

Only install apps from trusted sources. Splunkbase has a reputation system that can help you to identify trusted sources.
Read the app’s documentation carefully. This will help you to understand how to use the app and how to troubleshoot any problems that you may encounter.
Keep your apps up to date. Splunkbase apps are updated regularly to fix bugs and add new features.

Question 22

When using a Splunkbase TA or app how do you customize it?

Answer 22

Modify the configuration files. Splunkbase TAs and apps typically come with a number of configuration files that control how the TA or app behaves. You can modify these configuration files to meet your specific needs. For example, you can modify the configuration files to specify different data inputs, outputs, or transformations.
Create custom transforms. Splunk transforms can be used to modify data before it is indexed. You can create custom transforms to meet your specific needs. For example, you could create a custom transform to extract a specific field from a data source or to convert a field to a different format.
Write custom scripts. You can write custom scripts to automate tasks and extend the functionality of Splunkbase TAs and apps. For example, you could write a custom script to collect data from a new source or to generate a custom report.

Question 23

Talk to me about summary indexing and what you used it for?

Answer 23

x

Question 24

Where does splunk buckets reside?

Answer 24

x

Question 25

What attributes do you use to configure retention?

Answer 25

x

Question 26

Which takes precedence local or default? why?

Answer 26

The local configuration file takes precedence over the default configuration file in Splunk. This is because the local configuration file is more specific to the current environment. The default configuration file is a general configuration file that is used by all Splunk environments.

When Splunk starts, it loads the default configuration file first. It then loads the local configuration file, if it exists. The local configuration file overrides any settings in the default configuration file.

This allows you to customize the Splunk configuration for your specific environment.

Question 27

Greedy vs Lazy Regex

Answer 27

x

Question 28

Can you list the precedence order? For local.

Answer 28

x

Question 29

Name 4 internal logs and their uses

Answer 29

x

Question 30

Key differences between a TA and a Splunk App

Answer 30

x

Question 31

Which components of Splunk commonly share the same instance?

Answer 31

In a Splunk deployment, the components that often share the same instance are:
1. Search Head and Search Peer: Search Heads can be clustered and configured to share the same instance to distribute search workloads and provide redundancy. These are collectively known as a Search Head Cluster.
2. Indexer and Peer Nodes: Indexers can also be configured in a cluster with peer nodes. This is done for scalability, data replication, and fault tolerance. Indexers in a cluster share the same instance for these purposes.
3. License Master and License Slave: In a distributed Splunk environment, a License Master can be used to manage licenses centrally. License Slave instances can share this configuration with the License Master to ensure proper license management across the deployment.
4. Deployment Server and Deployment Clients: A Deployment Server is used to manage configurations across multiple Splunk instances. Deployment Clients share the same instance as the Deployment Server to receive and apply configurations. These configurations are often employed in larger Splunk deployments to enhance performance, availability, and manageability.

Question 32

What is multi-site clustering? And your experience with it?

Answer 32

Multi-site clustering is a Splunk feature that allows you to deploy Splunk search heads and indexers across multiple sites. This can improve the performance and reliability of your Splunk deployment, especially if you have a large amount of data to process or if your data is located across multiple geographic regions.

With multi-site clustering, you can create a single Splunk cluster that spans multiple sites. Each site has its own set of indexers, and each indexer replicates its data to the other indexers in the cluster. This ensures that your data is always available, even if one site experiences an outage.

You can also use multi-site clustering to load balance search traffic across the different sites. This can improve the performance of your Splunk deployment by distributing the search load across multiple search heads.

I have experience with multi-site clustering in a production environment. I have used it to deploy Splunk clusters across multiple geographic regions. I have found that multi-site clustering can be a very effective way to improve the performance, reliability, and scalability of Splunk deployments.

Here are some tips for using multi-site clustering effectively:

Carefully plan your deployment. Consider the location of your data, the performance requirements of your deployment, and your budget.
Configure your Splunk cluster correctly. Follow the Splunk documentation to configure your Splunk cluster for multi-site clustering.
Monitor your Splunk cluster. Use the Splunk Web UI or the Splunk CLI to monitor the performance of your Splunk cluster and to identify any problems.
Manage the replication of data between sites. You can use the Splunk Web UI or the Splunk CLI to manage the replication of data between sites.

Question 33

Name two ways you can filter out unwanted data?

Answer 33

x

Question 34

Give examples of character types in Regex and what they do

Answer 34

x

Question 35

Difference between search time and index time field extraction ? which is better?

Answer 35

x

Question 36

Check if ports are open and listening for inbound data

Answer 36

x

Question 37

What is the REX command used for and how do you use it?

Answer 37

Regex is a general-purpose language that can be used in a variety of programming languages and applications. It is used to define patterns that can be matched in text. Regex is based on a set of rules and operators that define the patterns that can be matched.

REX is a Splunk-specific language that is used to search and analyze data in Splunk. It is based on regular expressions, but it extends regex with a number of features that make it more powerful and flexible for use in Splunk.

Lookaheads and lookbehinds: Lookaheads and lookbehinds allow you to match patterns that are adjacent to the current position, but not part of the match itself. This is useful for matching patterns that are surrounded by other text.
Subroutines: Subroutines allow you to define reusable patterns that can be used in multiple rex expressions. This can make your rex expressions more concise and easier to read.
Splunk-specific functions: REX includes a number of Splunk-specific functions that can be used in rex expressions. These functions allow you to perform tasks such as extracting data from fields, converting data types, and formatting data.

Question 38

Explain your architectural experience and what have you done in that realm?

Answer 38

I have experience with designing, implementing, and managing Splunk architectures in a variety of environments. I have worked with customers of all sizes, from small businesses to large enterprises.

Here are some of the Splunk architecture tasks that I have performed:

Designed and implemented Splunk clusters for high availability and performance.
Configured Splunk data forwarding, indexing, and search components to meet the specific needs of each customer.
Created and customized Splunk dashboards and reports.
Developed Splunk scripts and transforms to automate tasks and extend the functionality of Splunk.
Troubleshooted and resolved Splunk performance and functionality problems.
I have also worked on a number of Splunk architecture projects, including:

Designed and implemented a Splunk cluster for a large financial services company to monitor and analyze their security data.
Helped a retail company to implement a Splunk cluster to monitor their customer behavior and identify trends.
Worked with a healthcare company to implement a Splunk cluster to monitor their patient data and identify potential problems.
I am also familiar with the latest Splunk technologies and trends, such as Splunk Cloud and Splunk Enterprise Security. I am always looking for new ways to use Splunk to help customers solve their business problems.

Here are some of the best practices that I have learned from my experience with Splunk architecture:

Use a distributed architecture for large deployments. This will improve the performance and reliability of your Splunk deployment.
Configure Splunk components to meet the specific needs of your environment. There is no one-size-fits-all approach to Splunk architecture.
Monitor your Splunk deployment closely. This will help you to identify and resolve problems early.
Use Splunk best practices and documentation. Splunk provides a wealth of information on how to design, implement, and manage Splunk deployments.

Question 39

Tell me about yourself

Answer 39

(repeat)

Question 40

Filepath where deployment apps reside

Answer 40

x

Question 41

Filepath for where file configurations from cluster master reside

Answer 41

x

Question 42

Name 5 splunk ports and what they are used for

Answer 42

x

Question 43

In clustered environment what component manages which component?

Answer 43

x

Question 44

What are sourcetypes and give examples of some

Answer 44

x

Question 45

Your team need to monitor a file with the path app/log/test-how would you set that monitoring stanza up?

Answer 45

[monitor://app/log/test]
disabled=0
sourcetype = linux_secure
index=security

**restart splunk after editing configuration files

Question 46

Your co-workers made some unauthorized changes to the server and your boss wants to trace the steps and undo the damage-which index would you search and why?

Answer 46

x

Question 47

Your boss wants you to run a search over a large data set-how would you set that up so that it would be as efficient as possible.

Answer 47

x

Question 48

One of your servers in London went down for a couple of hours-which internal indexer saves your day and why?

Question 49

I am trying to connect my indexers together for data replication-what port should I use?

Answer 49

x

Question 50

Your co-worker is trying to set the search factor to 4 while keeping the replication factor at 3-why is he wrong? and how would you explain it to him?

Answer 50

x

Question 51

A boss told you check in files to make sure the indexers have the bundles you just pushed from the CM-where would you go to check this?

Answer 51

x

Question 52

In what scenario would you be creating BASE APPS?

Answer 52

Base apps in Splunk are reusable apps that provide a starting point for collecting, indexing, searching, and analyzing data. They are typically used for specific use cases, such as security monitoring, IT operations, or business intelligence.

Here are some scenarios in which you would create a Splunk base app:

To create a custom app that is tailored to the specific needs of your organization.
To create an app that integrates with other Splunk apps or third-party applications.
To create an app that extends the functionality of Splunk by adding new features or capabilities.
To create an app that packages and distributes existing Splunk content, such as dashboards, reports, and alerts.
For example, you might create a Splunk base app to:

Monitor the security of your network and applications.
Track the performance of your IT infrastructure.
Analyze customer behavior to improve your marketing campaigns.
Monitor the health of your production systems.
Identify and investigate fraud.

Question 53

How would you SCALE OUT your Splunk architecture-what are the considerations?

Answer 53

There are two main ways to scale out Splunk architecture: horizontally and vertically.

Horizontal scaling involves adding more Splunk components, such as indexers and search heads, to your cluster. This is the most common way to scale Splunk deployments because it allows you to scale linearly. In other words, the performance of your Splunk deployment will increase linearly as you add more components.

Vertical scaling involves adding more resources to your existing Splunk components, such as more CPU, memory, and disk space. This is a less common way to scale Splunk deployments because it is not as efficient as horizontal scaling. In addition, vertical scaling can be more expensive than horizontal scaling.

When scaling out your Splunk architecture, there are a few things you need to consider:

Data volume: How much data do you need to index and search?
Search load: How many concurrent searches do you need to support?
Budget: How much money do you have to spend on scaling your Splunk deployment?
Expertise: Do you have the expertise to manage a distributed Splunk deployment?
If you have a large data volume or a high search load, you will need to scale out your Splunk deployment horizontally. If you have a limited budget, you may need to scale out your Splunk deployment vertically.

Question 54

Your boss has determined that the company rarely searches data is over 1month and almost never looks at anything over a year-how would you set up the bucket system?

Answer 54

x

Question 55

What is the difference between monitoring a file in linux systems and in windows system when writing your monitoring stanza?

Answer 55

Monitoring stanza in Windows vs Linux

Windows = [monitor://C:\app\log\data\catalina.out]
Linux = [monitor:///another/random/path]

Question 56

What .conf file helps you collect data ?

Answer 56

x

Question 57

Where would you put that file in a clustered environment?

Answer 57

x

Question 58

Tell me the path of where you would put this .conf file?

Answer 58

CM filepath

Question 59

TA vs Splunk App

Answer 59

(repeat)

Question 60

What is indexes.conf for?

Answer 60

x

Question 61

Bucket lifecycle in full detail

Answer 61

x

Question 62

How are buckets configured in your environment? What are your rotation policies in your environment?

Answer 62

x

Question 63

Filepath for the cold bucket

Answer 63

x

Question 64

What is metadata? and describe its component in full detail

Answer 64

Metadata in Splunk is data that describes data. It is used to provide additional information about events, such as the source of the event, the time the event occurred, and the type of event. Splunk uses metadata to index and search events, and to generate reports and dashboards.

Splunk metadata is divided into two types:

System-generated metadata: This metadata is generated by Splunk automatically, such as the event time and the source of the event.
User-defined metadata: This metadata is added by the user, such as the sourcetype and the index.
System-generated metadata includes the following fields:

_time: The time at which the event occurred.
_source: The source of the event, such as the hostname or the IP address.
host: The hostname of the system that generated the event.
sourcetype: The type of event, such as a log message or a system event.
index: The index in which the event is stored.
User-defined metadata can include any field that is defined in the Splunk schema. For example, you could define a field to store the severity of the event, or the application that generated the event.

Splunk uses metadata to:

Index and search events: Splunk uses metadata to index events so that they can be quickly and easily searched. For example, you could search for all events that occurred within a certain time period, or for all events that were generated by a specific application.
Generate reports and dashboards: Splunk uses metadata to generate reports and dashboards that provide insights into your data. For example, you could generate a report that shows the number of events that occurred each day, or a dashboard that shows the top 10 busiest hosts.

Question 65

Heavy forwarder vs UF

Answer 65

x

Question 66

What are two requirements a forwarder will need in order to receive apps from the DS?

Answer 66

x

Question 67

Explain what a load balancer does and where it sits

Answer 67

Load balancer = sit between users and our SH (only)=does a health check of SH-if SH is bad LB will move end user to another SH

given URL of load balancer in work environment (accessing main domain)jpmorgan.splunk.com

Question 68

Explain the process of creating a custom app

Answer 68

(repeat)

Question 69

What is clustering and what are the benefits of a clustered environment ?

Answer 69

x

Question 70

what does it mean to scale up or scale out?

Answer 70

x

Question 71

why would you scale your searchhead up?

Answer 71

x

Question 72

Name the two types of files indexes consist of

Answer 72

x

Question 73

Difference between the duties of the deployer and the duties of the captain in a SH cluster

Answer 73

x

Question 74

what is RF and SF? and what is their default setting?

Answer 74

x

Question 75

how are forwarders typically grouped in a clustered environment ?

Answer 75

x

Question 76

What is multisite clustering

Answer 76

(repeat)

Question 77

That place you store your data what is called-where orgs store their data

Question 78

what are the functions of a cluster master

Question 79

what port does the master node use to communicate with its peer nodes

Question 80

what happens when SH becomes clusters-and what are the benefits

Question 81

Static captain vs dynamic captain

Question 82

what is a distributed search ?

Question 83

Name 3 jobs of cluster peers

Answer 83

Here are three jobs of cluster peers in Splunk:

Indexing: Cluster peers are responsible for indexing data. This process involves breaking the data into events and storing the events in a highly compressed format.

Replication: Cluster peers replicate indexed data to other cluster peers. This process can be done in real time or on a schedule. Replication helps to ensure that all of the cluster peers have the same data available for searching and analysis.

Search: Cluster peers can be used to search data. When you search for data in a Splunk cluster, the search head distributes the search to the cluster peers. The cluster peers then search the data and return the results to the search head. The search head then aggregates the results and displays them to the user.

In addition to these three jobs, cluster peers can also perform other tasks, such as:

Load balancing: Cluster peers can be used to load balance traffic across the cluster. This can help to improve the performance and reliability of the cluster.
Failover: Cluster peers can be configured to fail over to other cluster peers in the event of a failure. This helps to ensure that the cluster remains available even if one or more cluster peers fail.

Question 84

How would forwarder distribute the data it has collected? How would it write them in indexes?

Answer 84

Universal forwarder

A universal forwarder can distribute data to any number of Splunk indexers, either in real time or on a schedule. The universal forwarder uses a variety of mechanisms to distribute data, such as TCP/IP, UDP, and HTTPS.

The universal forwarder can also write data to indexes in a variety of ways. The default setting is to index the data before sending it to the indexer. However, you can also configure the universal forwarder to send the data unindexed or to use a hybrid approach.

Heavy forwarder

A heavy forwarder can send data to a single Splunk indexer. The heavy forwarder can also perform some pre-processing on the data, such as parsing and filtering. This can reduce the load on the indexer and improve the performance of searches and reports.

The heavy forwarder can write data to indexes in the same way as a universal forwarder. However, the heavy forwarder typically indexes the data before sending it to the indexer. This is because the heavy forwarder has more resources available than a universal forwarder.

Question 85

how does replication work/explain what happens once data comes in

Answer 85

Splunk replication is the process of copying data from one Splunk instance to another. This can be done for a variety of reasons, such as to improve performance, reliability, and scalability.

Once data comes into Splunk, it is indexed and then replicated to other Splunk instances in the cluster. This process is transparent to users, so they do not need to worry about how their data is being replicated.

Here is a more detailed explanation of how Splunk replication works:

Data is received by a Splunk indexer. Splunk indexers are responsible for receiving data from sources and indexing it.
The data is indexed by the Splunk indexer. The indexing process involves breaking the data into events and storing the events in a highly compressed format.
The indexed data is replicated to other Splunk indexers in the cluster. This is done using a variety of mechanisms, such as TCP/IP or UDP.
The replicated data is stored on the other Splunk indexers. The replicated data is stored in the same highly compressed format as the original data.
The replicated data is available for searching and analysis. Once the data has been replicated to other Splunk indexers, it is available for searching and analysis by users.

Question 86

explain round robin

Question 87

what is a license master

Answer 87

A license master is a central license server for a Splunk deployment. It is responsible for managing and distributing licenses to Splunk instances. The license master can be installed on any Splunk instance, but it is typically installed on the search head or deployment server.

Question 88

I want to make a change to an indexer in a clustered environment-where would I go to do this?

Question 89

what is etc/system local used for and what is a better alternative ?

Answer 89

The etc/system/local directory in Splunk is used to store configuration files that are specific to the local Splunk instance. These files can override the default configuration files that are shipped with Splunk.

Here are some examples of configuration files that are typically stored in the etc/system/local directory:

inputs.conf: This file configures Splunk to collect data from a variety of sources.
outputs.conf: This file configures Splunk to send data to a variety of destinations.
props.conf: This file defines custom properties that can be used in Splunk searches and reports.
transforms.conf: This file defines custom transforms that can be used to modify data before it is indexed.
The etc/system/local directory is a convenient place to store configuration files that are specific to the local Splunk instance. However, it can also be difficult to manage when there are multiple Splunk instances in a deployment.

A better alternative to using the etc/system/local directory is to use the Splunk Deployment Server. The Deployment Server allows you to centralize the management of Splunk configuration files. This can make it easier to manage and maintain Splunk deployments with multiple instances.

Question 90

what index would you use to troubleshoot errors on your server?

Question 91

Explain capture groups in Regex

Answer 91

A capture group is a regular expression that matches a specific part of a string. Capture groups are used to extract data from events, such as IP addresses, hostnames, and dates.