Splunk Day-1a

CURL NEW
Edit HEX and add tutorial_index

STOP COLLECTD SERVER
docker compose down collectd

curl -k https://35.91.217.173:8088/services/collector/event -H "Authorization: Splunk edc2b152-2e32-41db-ad62- "test event", "sourcetype": "manualevent", "index": "tutorial_index"}'

{"text":"Success","code":0}

curl -k https://35.91.217.173:8088/services/collector/event -H "Authorization: Splunk edc2b152-2e32-41db-ad62- "test event 124 New", "sourcetype": "manualevent", "index": "tutorial_index"}'

{"text":"Success","code":0}


index="tutorial_index"

Lab

2024 AUG FINAL SCHEDULE SPLUNK + LAB SETUP

ssh -i ~/vms/aws/SreeOregonOct24.pem ubuntu@44.243.99.69
ssh -i ~/vms/aspe-sree.pem ubuntu@54.167.126.67
git clone https://github.com/schogini/splunk-lab.git
schogini
cd splunk-lab
cd ~;cd splunk-lab/SETUP2024/

sudo sh ssh.sh or sudo sh ssh-abrar.sh
sudo sh pwd.sh
sudo sh usr.sh
ssh micro@44.243.99.69
aphp4..

git clone https://github.com/schogini/splunk-lab.git
..

cd splunk-lab/SETUP2024/

sudo sh docker.sh

exit and login

ssh micro@44.243.99.69
aphp4..

cd splunk-lab
sh start.sh

sudo sh delete.sh on all except sree's
sudo rm -fr /home/ubuntu/splunk-lab on all

SLIDES

AIOps (Artificial Intelligence for IT Operations) and IT Ops (IT Operations) focus on managing and optimizing an organization’s IT infrastructure. Here’s a brief breakdown:

IT Ops (IT Operations)

IT Ops involves the management, maintenance, and administration of all IT services, infrastructure, and applications within an organization. Key tasks include: - Monitoring and Incident Management: Keeping systems and applications running smoothly and addressing issues when they arise. - Maintenance: Regular updates, backups, and patching of systems. - Resource Allocation: Ensuring resources (e.g., storage, computing power) are efficiently distributed.

AIOps (Artificial Intelligence for IT Operations)

AIOps enhances IT Ops by integrating machine learning and data analytics to handle complex IT operations tasks. Key features include: - Anomaly Detection: Using AI to detect unusual patterns in real-time, identifying potential issues before they impact users. - Root Cause Analysis: Automating analysis of vast datasets to quickly pinpoint the cause of incidents. - Predictive Insights: Forecasting issues and optimizing resource usage to prevent downtime.

AIOps enables faster and smarter decision-making, reducing human intervention and increasing operational efficiency. It’s especially useful in modern IT environments with high complexity and large data volumes.

Authentication Methods

EVENTS AND METRICS

The primary difference between events and metrics in IT monitoring and observability is how they represent and analyze data.

  1. Events:

  2. Metrics:

In summary: Events are detailed logs for in-depth analysis, while metrics are numerical values ideal for monitoring and trend analysis over time.

MAJOR MINOR BREAKERS

In Splunk, indexing rules rely on understanding how the system breaks down and tokenizes event data for searching, impacting how data can be searched and analyzed. Splunk uses major breakers and minor breakers to create tokens (smaller searchable chunks) from the data, which directly influences how search terms behave. Here’s an explanation with examples:

Major Breakers

Minor Breakers

Using TERM() to Isolate Specific Searchable Terms

The TERM() function in Splunk allows you to search for exact terms without breaking down by major or minor breakers.

Understanding these breakers and the TERM() function is essential to ensure accurate searches, especially when dealing with structured data or specific terms that should not be divided.

TYPES OF SEARCHES

Here are examples of searches in Splunk for each type of processing category:

1. Distributable Streaming

2. Centralized Streaming

3. Transforming

4. Generating

5. Orchestrating

6. Dataset Processing

EFFICIENT SPL

Here are five quizzes to help understand inefficient versus efficient Splunk Search Processing Language (SPL) use cases:

Quiz 1

Question:
Which of the following is an inefficient way to retrieve events with the status=404?

Options: 1. index=web_logs | search status=404 2. index=web_logs status=404 3. search status=404 | index=web_logs 4. index=web_logs | where status=404

Answer:
Option 3 (search status=404 | index=web_logs) is inefficient because it places index=web_logs after the search term, which may slow down performance by not filtering at the index level first.

Quiz 2

Question:
When searching for events from the past 24 hours with status=500, which is more efficient?

Options: 1. index=app_logs | search _time>now()-86400 status=500 2. index=app_logs earliest=-24h@h status=500 3. index=app_logs | where _time>relative_time(now(), "-24h") status=500 4. index=app_logs | search earliest=-24h status=500

Answer:
Option 2 (index=app_logs earliest=-24h@h status=500) is efficient because using earliest=-24h@h optimizes time filtering at the index level.

Quiz 3

Question:
For a query that finds unique users in the users.csv lookup file and performs calculations, which is more efficient?

Options: 1. | inputlookup users.csv | dedup user_id | eval total_purchases=... 2. | inputlookup users.csv | eval total_purchases=... | dedup user_id 3. | inputlookup users.csv | dedup user_id 4. | inputlookup users.csv | search user_id=*

Answer:
Option 1 (| inputlookup users.csv | dedup user_id | eval total_purchases=...) is efficient because dedup user_id removes duplicates before performing calculations.

Quiz 4

Question:
Which command sequence is more efficient for displaying the top 10 values of host with the highest count?

Options: 1. index=web_logs | top limit=10 host 2. index=web_logs | stats count by host | sort - count | head 10 3. index=web_logs | head 10 | stats count by host 4. index=web_logs | sort limit=10 host

Answer:
Option 1 (index=web_logs | top limit=10 host) is efficient because top directly retrieves and limits results without extra processing steps.

Quiz 5

Question:
For searching errors in large datasets, which command sequence would generally be inefficient?

Options: 1. index=app_logs | search error 2. index=app_logs | regex _raw="error" 3. index=app_logs error 4. index=app_logs | where match(_raw, "error")

Answer:
Option 4 (index=app_logs | where match(_raw, "error")) is inefficient because where evaluates events after they’re retrieved, unlike direct filters at the index level.

Searching

index="tutorial_data"
index="tutorial_data" clientip="*"
index="tutorial_data" sourcetype="vendor_sales"
index="tutorial_data” 5036
index="tutorial_data" 5036 Code
index="tutorial_data" 5036 AND Code
index="tutorial_data" 5036 OR Code
index="tutorial_data" 5036 NOT Failed
index="tutorial_data" sourcetype="access*" action="purchase" status=200
index="tutorial_data" sourcetype="access_*"
index="tutorial_data" sourcetype="access_*" status=200 action=purchase

Regex Based Filtering

index="tutorialdata" sourcetype="accesscombined_wcookie" | regex _raw!="Apple"

The regex command in Splunk is used to filter events or extract data based on specific patterns. It's a powerful tool for working with textual data and finding patterns using regular expressions. Regular expressions (regex) allow you to define a search pattern, which is very useful for extracting, matching, or excluding specific values in log files or event data.

Here are various examples and use cases of the regex command:

1. Basic Filtering with regex

You can use the regex command to filter events based on a specific pattern. Only events matching the regex pattern will be returned.

Example (Filter Events by IP Address):

index=web sourcetype=access_logs | regex ip="^192\.168\."

2. Filtering Events with Exclusion

You can exclude events that match a certain pattern using the !~ operator within the regex command.

Example (Exclude IP Addresses from a Specific Subnet):

index=web sourcetype=access_logs | regex ip!="^192\.168\."

Head/Tail

index="tutorial_data" VendorID=* | head
index="tutorial_data" VendorID=* | tail

Exercise-1: Top/Rare

index="tutorial_data" sourcetype="vendor_sales" | top VendorID
index="tutorial_data" sourcetype="vendor_sales" | top 5 VendorID
index="tutorial_data" sourcetype="vendor_sales" | rare VendorID
index="tutorial_data" sourcetype="vendor_sales" | rare limit=5 VendorID

Exercise-3: Stat Sum

index="tutorial_data" sourcetype="access_*" | top limit=100 referer_domain
index="tutorial_data" sourcetype="access_*" | top limit=100 referer_domain | stats sum(count)
index="tutorial_data" sourcetype="access_*" | stats count by status, host

Exercise-4: Table

index="tutorial_data" sourcetype="access_*" status=200 action=purchase | top limit=1 clientip
index="tutorial_data" sourcetype="access_*" status=200 action=purchase | top limit=1 clientip | table clientip
index="tutorial_data" sourcetype="access_*" action=purchase status=200 | top limit=1 clientip showperc=false showcount=false

Exercise-5: Top Buyer/Subsearch

SUBSCEARCH

https://blog.schogini.com/html_files/splunk-subsearch.html

index="tutorial_data" sourcetype="access_*" status=200 action=purchase clientip=<IP we got from above> ===> will give me the purchases this customer has made
index="tutorial_data" sourcetype="access_*" status=200 action=purchase clientip=<IP> | stats count(productId)
index="tutorial_data" sourcetype="access_*" status=200 action=purchase clientip=<IP> | stats count by productId
OR
index="tutorial_data" sourcetype="access_*" status=200 action=purchase clientip=<IP> | stats count, values(productId)
index="tutorial_data" sourcetype="access_*"  status=200  action=purchase [search index="tutorial_data" sourcetype="access_*"  status=200  action=purchase | top 1 clientip showperc=false showcount=false] | stats count by productId
OR
index="tutorial_data" sourcetype="access_*" status=200 action=purchase [search index="tutorial_data" sourcetype="access_*" status=200 action=purchase | top limit=1 clientip | table clientip] | stats count by productId

Exercise-6: Eval

index="tutorial_data" sourcetype="access_*" | eval error=if(status == 200, "OK", "Problem")
index=web-uf_index status=200 | eval A=status+100
index=web-uf_index status=200 | eval A=status*100
index="tutorial_data" VendorID=* | eval NEW_FIELD=VendorID+"_"+Code

Exercise-6: Chart

index="tutorial_data" sourcetype="access_*" status=200 | chart count as "views"
index="tutorial_data" sourcetype="access_*" status=200 | chart count as "views" by productId
index="tutorial_data" sourcetype="access_*" status=200 | chart count as "views" by productId, action
index="tutorial_data" sourcetype="access_*" status=200 | chart count as "views" count(eval(action="addtocart")) as "addtocart" by productId
index="tutorial_data" sourcetype="access_*" status=200 | chart count as "views" count(eval(action="addtocart")) as "addtocart" count(eval(action="purchase")) as "purchase" by productId

Exercise-7: TimeChart

index="tutorial_data" sourcetype="access_*" action=purchase | timechart span=1d count by categoryId
index="tutorial_data" sourcetype="access_*" action=purchase | timechart span=1d count by categoryId usenull=false

Exercise-8: Rex (Regex to extract fields on the fly)

Regex Example: Extract Browser Name from Access lOg
index="tutorial_data" sourcetype="access_combined_wcookie" | rex field=useragent "(?<browser>[a-zA-Z]+)/"


from (?P<IP>\d+\.\d+\.\d+\.\d+)

(?P<datetime_usSREE>GET) 
(?P<datetime_usSREE>”GET) 
(?P<datetime_usSREE>\[.+\]) ==> [21/Aug/2024:18:22:16]
"(?P<datetime_usSREE>.+)” ==> GET /oldlink?itemId=EST-14&JSESSIONID=S

\[(?P<datetime_usSREE>.+)] => 21/Aug/2024:18:22:16

? = NOT GREEDY
\[(?P<datetime_usSREE>.+)].*(?P<urlSree>http.*?)”\s


CHATGPT
^(?P<client_ip>\d+\.\d+\.\d+\.\d+) - - \[(?P<datetime>[^\]]+)\] "(?P<http_method>\w+) (?P<http_uri>[^"]+)" (?P<http_status_code>\d+) \d+ "[^"]*" "(?P<http_user_agent>[^"]+)" (?P<http_response_time>\d+)



index="tutorial_data" sourcetype="access_combined_wcookie" status=200| rex field=EXISTING-FIELD "(?<browser>[a-zA-Z]+)/"
index="tutorial_data" sourcetype="access_combined_wcookie" status=200| rex field=useragent "(?<browser>[a-zA-Z]+)/"
index="tutorial_data" sourcetype="access_combined_wcookie" status=200| rex field=useragent "(?<useragent>[a-zA-Z]+)/"
index="tutorial_data" sourcetype="access_combined_wcookie" | rex field=useragent "(?<BROWSER>[a-zA-Z]+)/"

index="tutorial_data" sourcetype="access_combined_wcookie" status=200| rex field=useragent mode=sed "s/^M/N/g"
index="tutorial_data" sourcetype="access_combined_wcookie" status=200| rex field=useragent mode=sed "s/Mozilla/Godzilla/g"

index="tutorial_data" sourcetype="access_combined_wcookie" | regex _raw="Mozilla"

3. Extracting Data with rex

If you want to extract data from events using regex, use the rex command, which allows for field extraction.

Example (Extract Domain from URL):

index=web sourcetype=access_logs | rex field=url "(?<domain>https?://[^/]+)"

4. Case-Insensitive Matching

You can perform case-insensitive matches by adding the (?i) flag to the regex pattern.

Example (Case-Insensitive Match for HTTP Methods):

index=web sourcetype=access_logs | regex method="(?i)post"

5. Using Multiple Patterns

You can use multiple patterns in your regex to filter events that match any of the patterns.

Example (Match Multiple Status Codes):

index=web sourcetype=access_logs | regex status_code="(200|404|500)"

6. Extracting Multiple Fields with rex

You can extract multiple fields from a single event using rex with multiple capture groups.

Example (Extract IP and Port from a Log Entry):

index=web sourcetype=firewall_logs | rex field=_raw "(?<src_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}):(?<src_port>\d+)"

7. Matching Entire Event (regex _raw)

You can apply a regex pattern to the entire event (_raw) to filter based on raw event data.

Example (Filter Events with URLs Containing ".com"):

index=web sourcetype=access_logs | regex _raw="\.com"

8. Extracting Strings with Fixed Patterns

You can use rex to extract fixed-length or patterned strings from events.

Example (Extract Dates from Logs):

index=web sourcetype=access_logs | rex field=_raw "(?<date>\d{4}-\d{2}-\d{2})"

9. Replacing Text with eval and replace()

If you need to replace part of a field’s value based on a regex pattern, you can use eval with the replace() function.

Example (Anonymize IP Addresses):

index=web sourcetype=access_logs | eval ip=replace(ip, "\d{1,3}\.\d{1,3}\.\d{1,3}\.(\d{1,3})", "xxx.xxx.xxx.xxx")

10. Extracting Field Values Using Backreferences in rex

You can use backreferences (like \1, \2, etc.) to refer to previously matched groups in your regex pattern.

Example (Extract Usernames from Email Addresses):

index=web sourcetype=email_logs | rex field=email "(?<username>[^@]+)@"

11. Using regex to Filter by Number Range

You can use regex to filter numeric fields based on a range by crafting specific patterns.

Example (Filter IPs in 192.168.x.x Subnet):

index=web sourcetype=access_logs | regex ip="^192\.168\.\d{1,3}\.\d{1,3}$"

Common regex Patterns and Use Cases:

Pattern Description
\d Matches any digit (0-9).
\w Matches any word character (alphanumeric + underscore).
. Matches any character except newline.
^ Matches the start of the string.
$ Matches the end of the string.
\. Matches a literal period (dot).
.* Matches any sequence of characters (wildcard).
\d{1,3} Matches a number between 1 and 3 digits.
[A-Za-z0-9] Matches any alphanumeric character.
(?<fieldname>...) Named capture group to extract a specific field.
(value1|value2|value3) Matches any of the values inside the parentheses (OR operator).

Summary:

By mastering the regex command and its functions, you can perform advanced pattern matching, data extraction, and filtering in Splunk searches, making it easier to focus on key data points within your logs.

FIELD EXTRACTION

The data you provided has a slightly different format compared to the regular expression we discussed earlier. I’ll modify the regular expression to match the provided log entry format and then verify it.

Log Entry:

190.113.128.150 - - [17/Aug/2024:23:59:01] "POST /cart/success.do?JSESSIONID=SD8SL1FF2ADFF27563 HTTP 1.1" 200 2392 "http://www.buttercupgames.com/cart.do?action=purchase&itemId=EST-16" "Mozilla/5.0 (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3" 228

Adjusted Regular Expression:

Here’s a regex to match and extract the necessary fields from this specific log format:

^(?P<client_ip>\d+\.\d+\.\d+\.\d+) - - \[(?P<datetime>[^\]]+)\] "(?P<http_method>\w+) (?P<http_uri>[^"]+)" (?P<http_status_code>\d+) \d+ "[^"]*" "(?P<http_user_agent>[^"]+)" (?P<http_response_time>\d+)

Explanation:

  1. ^(?P<client_ip>\d+\.\d+\.\d+\.\d+): Matches the client IP address at the start of the line.
  2. - - \[(?P<datetime>[^\]]+)\]: Matches the datetime field within square brackets.
  3. "(?P<http_method>\w+) (?P<http_uri>[^"]+)": Matches the HTTP method and URI.
  4. (?P<http_status_code>\d+) \d+: Matches the HTTP status code and skips the byte size field.
  5. "[^"]*": Skips the referrer URL (not captured).
  6. "(?P<http_user_agent>[^"]+)": Matches the user agent string.
  7. (?P<http_response_time>\d+): Matches the response time at the end.

Verification with the Provided Data:

Applying the regular expression to the log entry:

This regular expression should successfully capture all relevant fields from the provided log entry in Splunk.

CLIENT IP FIELD EXTRACTION
^(?P<clientIP>\d+\.\d+\.\d+\d+)

EXTRACT-MYIP    Inline  ^(?P<MYIP>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) 

clientIP    Inline  ^(?P<clientIP>\d+\.\d+\.\d+\d+) 

secure-2 : EXTRACT-ACCESS_IP    Inline  from (?P<ACCESS_IP>\d{0,3}\.\d{0,3}\.\d{0,3}\.\d{0,3}) 



access_combined_wcookie : EXTRACT-MYIP  Inline  ^(?P<MYIP>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})   
admin
    
search
    Private | Permissions   Enabled     Move | Delete
access_combined_wcookie : EXTRACT-clientIP  Inline  ^(?P<clientIP>\d+\.\d+\.\d+\d+)     
admin
    
search
    Private | Permissions   Enabled     Move | Delete
secure-2 : EXTRACT-ACCESS_IP    Inline  from (?P<ACCESS_IP>\d{0,3}\.\d{0,3}\.\d{0,3}\.\d{0,3}) 


splunk_web_service : EXTRACT-useragent  Inline  userAgent=(?P<browser>[^ (]+)

ADVANCED SEARCH - TRANSACTIONS

https://blog.schogini.com/html_files/splunk-advanced-search.html

index="tutorial_data" sourcetype="access_*" status=200 | regex SD7SL10FF6ADFF48967

index="tutorial_data" sourcetype="access_*" status=200 | transaction JSESSIONID

GET SEARCH HISTORY

GET SEARCH HISTORY

index=_audit action=search info=granted search=* NOT "search_id='scheduler" NOT "search='|history" NOT "user=splunk-system-user" NOT "search='typeahead" NOT "search='| metadata type=* | search totalCount>0" | stats count by user search _time | sort _time | convert ctime(_time) | stats list(_time) as time list(search) as search by user

SEE LOOKUP DATA

| inputlookup prices.csv
| inputlookup prices_lookup

MULTI IP SUB QUERY

index="tutorial_index" sourcetype="access_combined_wcookie" action=purchase status=200 
[search index="tutorial_index" sourcetype="access_combined_wcookie" action=purchase status=200 
| top limit=5 clientip 
| fields clientip 
| format]