Splunk Day-1b Collectd-Hec-Metric

The difference between collectd and Splunk's HTTP Event Collector (HEC) lies in their functionality, purpose, and the way they integrate with Splunk for data collection. Here's a detailed comparison:

1. Purpose:

2. Data Collection and Usage:

3. Deployment and Configuration:

4. Data Format:

5. Use Cases:

6. Scalability:

Summary:

Both tools serve different purposes, and your choice between them depends on the specific requirements of your monitoring and data ingestion strategy.

ENABLE

docker-compose exec collectd bash

nano /etc/collectd/collectd.conf

Hostname "dd15a3fc994c"

FQDNLookup false
Interval 10
Timeout 2
ReadThreads 5
WriteThreads 5

#Sree enabled these
LoadPlugin cpu
LoadPlugin interface
LoadPlugin load
LoadPlugin memory

TypesDB "/usr/share/collectd/types.db"
Include "/etc/collectd/collectd.d/*.conf"

<Plugin write_http>
    <Node "node1">
        URL "https://splunk:8088/services/collector/raw?channel=4609eb39-f258-435c-9a75-cfc2ea1303d4"
        Header "Authorization: Splunk edc2b152-2e32-41db-ad62-72f3a9ae7c5b"
        Format "JSON"
        VerifyPeer false
        VerifyHost false
        Metrics true
        StoreRates true
    </Node>
</Plugin>

# SEE ALREADY INCLUDING THIS
LoadPlugin write_http
<Plugin write_http>
  <Node "node-http-1">
    URL "http://splunk:8088/services/collector/raw?channel={{SPLUNK_TOKEN}}"
    Header "Authorization: Splunk {{SPLUNK_TOKEN}}"
    Format "JSON"
    Metrics true
    StoreRates true
  </Node>
</Plugin>

service collectd restart 
service collectd status

SEARCH METRICS

New Metrics India Collectd_httpd

| msearch index="collectd_index"
| mcatalog values(metric_name) WHERE index="collectd_index"
| mstats avg(_value) WHERE index="collectd_index" metric_name=cpu.idle.valu
| mstats avg(_value) WHERE index="collectd_index" metric_name=cpu.idle.value span=5s
| mstats avg(_value) where index="collectd_index" metric_name=cpu.idle.value span=10m

| mstats avg(_value) where index="collectd_index" metric_name="cpu.*" span=10m  prestats=true | stats avg(_value) by metric_name

| mstats avg(_value) WHERE index="collectd_index" metric_name=memory.free.value span=1d

msearch index="collectd_metrics"  =>>> search msearch index="collectd_metrics"
(DEPRECATED) | msearch index="collectd_metrics"
| mpreview index="collectd_metrics"

| mpreview index="collectd_index"

| mstats avg(_value) WHERE index="collectd_index" metric_name=cpu.idle.value span=5s

find / -type f -exec grep -l abcd {} \;

Metrics in Splunk

Explanation: - Metrics are a type of data in Splunk that is specifically optimized for high-volume, time-series data, such as performance data from servers, applications, and networks. Unlike event data, which is typically unstructured text, metrics data is structured, meaning it has defined fields like metric name, value, and timestamp. - Metrics are stored in metric indexes (metrics type index), which are optimized for storage and retrieval of time-series data, allowing for faster search and analysis.

Key Concepts:

  1. Metric Name: The name of the measurement being tracked (e.g., cpu.usage, memory.used).
  2. Dimensions: Additional fields that provide context to the metric, like host, region, service.
  3. Value: The actual numeric value of the metric.
  4. Timestamp: The time when the metric was recorded.

SPL Commands for Metrics:

1. mstats Command

2. mcatalog Command

3. msearch Command

4. timechart with Metrics

5. metasearch Command

6. mcollect Command

Best Practices with Metrics:

By understanding and using these SPL commands, you'll be able to effectively work with metrics data in Splunk, enabling high-performance monitoring and analysis of time-series data.