index="tutorialdata" sourcetype="access*" status=200 action=purchase [search index="tutorialdata" sourcetype="access*" status=200 action=purchase | top limit=1 clientIP | table clientIP] | stats count by productId

A subsearch in Splunk is a search within a search. Subsearches allow you to run a secondary search and use the results of that search as input for the main (outer) search. Subsearches are enclosed in square brackets ([...]), and their results are passed to the outer search for further processing.

Here’s a breakdown of how subsearches work, some common use cases, and examples:

How Subsearches Work:

1. Subsearch Execution: Splunk executes the subsearch first and gathers the results.
2. Passing Results: The results from the subsearch are passed as a filter or input to the main (outer) search. These results can be passed as field-value pairs or directly depending on the search.
3. Subsearch Result Limit: By default, a subsearch returns a maximum of 10,000 results or runs for a maximum of 60 seconds, whichever comes first. You can modify these limits if needed using subsearch_maxout.

Subsearch Syntax:

Subsearches are enclosed within square brackets [...], and they are placed within the main search.

Basic Syntax:

<main_search> [<subsearch>]

Examples of Subsearch Usage:

1. Simple Subsearch for Filtering Results

This is one of the most common subsearch use cases. A subsearch is used to generate a set of field-value pairs that are passed to the main search as filters.

Example (Find Events Related to Specific IPs):

index=web sourcetype=access_logs [search index=security sourcetype=firewall_logs | fields ip | dedup ip]

• Description: The subsearch (search index=security sourcetype=firewall_logs | fields ip | dedup ip) retrieves unique ip addresses from firewall logs. The results are passed to the main search, which then filters access logs (index=web sourcetype=access_logs) to show only events from those specific IPs.

2. Subsearch with return

The return command inside a subsearch allows you to format the results in a specific way (as a list of field-value pairs). This is useful when you need to pass specific fields to the outer search.

Example (Search for Specific Usernames):

index=web sourcetype=access_logs [search index=security sourcetype=user_logs "login_failed" | fields user | dedup user | return 10 user]

• Description: The subsearch finds the top 10 unique users with login failures from the user_logs (index=security sourcetype=user_logs "login_failed"). The return 10 user command formats the output as user="user1" OR user="user2" OR .... This result is passed to the outer search, which filters access_logs to show only the logs for those specific users.

3. Using Subsearch for Dynamic Field Matching

A subsearch can be used to match dynamic values in the outer search.

Example (Find Logs from Hosts with High Error Rates):

index=web sourcetype=access_logs [search index=web sourcetype=error_logs error_type=high | stats count by host | where count > 100 | fields host]

• Description: The subsearch finds hosts that have more than 100 high error events (index=web sourcetype=error_logs error_type=high). The main search (index=web sourcetype=access_logs) then retrieves logs from those hosts.

4. Subsearch with format

The format command is used within a subsearch to format the results as a list of OR’ed conditions. This is useful when you want to pass a list of field-value pairs to the outer search.

Example (Search for IPs with Multiple Failed Logins):

index=web sourcetype=access_logs [search index=web sourcetype=auth_logs "login_failed" | stats count by ip | where count > 10 | fields ip | format]

• Description: The subsearch finds IPs with more than 10 failed login attempts (login_failed). The format command transforms the list of IPs into a condition like ip="192.168.0.1" OR ip="192.168.0.2" OR ..., and the main search filters access logs for those IPs.

5. Subsearch for Exclusion

Subsearches can be used for exclusion as well. You can use a subsearch to exclude specific results from the outer search.

Example (Exclude Suspicious IPs from Search):

index=web sourcetype=access_logs NOT [search index=security sourcetype=suspicious_ips | fields ip]

• Description: The subsearch finds all suspicious IPs from suspicious_ips logs, and the main search excludes events from the access_logs that contain those IPs.

6. Subsearch with Joins (append)

You can use subsearches to append or join data from two different searches, allowing you to merge different datasets.

Example (Join Two Searches on a Common Field):

index=web sourcetype=access_logs | append [search index=web sourcetype=user_logs | fields user, action]

• Description: This command appends the results of the user_logs search to the access_logs search. The fields user, action ensures that only the relevant fields from the subsearch are appended.

Key Subsearch Commands:

Subsearch Limitations:

1. Result Limits: By default, subsearches return a maximum of 10,000 results or run for 60 seconds. You can modify this using subsearch_maxout in the limits.conf file if necessary, though increasing this can impact performance.
2. Performance: Since subsearches run before the main search, they can introduce performance overhead, especially when handling large datasets.
3. Chaining Subsearches: Subsearches cannot be nested or chained together.

Summary:

• Basic Subsearch: [search <subsearch>] runs a secondary search and uses its results to filter or manipulate the outer search.
• Field Matching: Use subsearches to pass field-value pairs (e.g., IPs, usernames) dynamically to the main search.
• Exclusions: Subsearches can be used to exclude results (e.g., NOT [<subsearch>]).
• Joining Data: Use append or similar commands to join results from different searches.

Subsearches are a powerful tool in Splunk, enabling complex searches by passing results between searches. However, care should be taken to manage performance, especially with large datasets.