Sunday, March 6, 2016

Rest API calls in Splunk (Frequently used)

I used to work in a access segregated environment where being a power user, i need some means to find things that are typically available to admin's only or multiple clicks in GUI. Splunk's REST API calls comes in handy.

Below are some of my collection of frequently used REST API calls in Splunk with their use case.


Splunk REST Search

Indexer  Status
| rest /services/server/introspection/indexer | table title splunk_server status updated
List of Lookup Files
| rest /services/data/transforms/lookups | table eai:appName filename title fields_list updated id
List of Commands
| rest /services/data/commands | table title type filename updated
List of Inputs
| rest /services/data/inputs/all | convert ctime(starttime) AS "Start Time"  | convert ctime(endtime) AS "End Time" | table index interval source sourcetype title updated starttime endtime "Start Time" "End Time"
List of Field Extractions
| rest /services/data/props/extractions | table title type value attribute
| rest /services/data/transforms/extractions | table title eai:appName REGEX FORMAT updated
List of Field Aliases
| rest /services/data/props/fieldaliases | table title type value attribute stanza updated
List of Saved event Types
| rest /services/saved/eventtypes | table title tags search
List of Saved Searches
| rest /services/saved/searches | table title search updated
List of jobs
| rest /services/search/jobs | table  author label title
List of Dashboards
| rest /servicesNS/-/-/data/ui/views | search"$app_name$" | table id label title
List of Fired Alerts
| rest /services/alerts/fired_alerts splunk_server=local| table eai:acl.owner id title triggered_alert_count
List of Saved Searches
| rest /servicesNS/-/-/saved/searches splunk_server=local | eval state=if(disabled=0,"Enabled","Disabled") | eval is_scheduled=if(is_scheduled=1,"Yes","No") | rename title AS savedsearch_name | fields cron_schedule state  is_scheduled savedsearch_name search dispatch.earliest_time dispatch.latest_time | search state=Enabled is_scheduled=Yes | table savedsearch_name, search

| rest /services/saved/searches | table title search

List of Searches ran by the user (‘s)
| rest /services/search/jobs |rename as customSearch|search NOT author="splunk-system-user" | eval SearchString=if(isnotnull(customSearch),customSearch,eventSearch)| addtotals fieldname=duration *duration_secs| table author,SearchString , earliestTime,latestTime,request.earliest_time, request.latest_time, eventCount,duration

| rest /services/search/jobs |rename as customSearch|search NOT author="splunk-system-user" | eval SearchString=if(isnotnull(customSearch),customSearch,eventSearch)| addtotals fieldname=duration *duration_secs  | eval search_time=tostring(duration, "duration")| table author,SearchString , earliestTime,latestTime,request.earliest_time, request.latest_time, eventCount, search_time | sort -search_time

| rest /services/search/jobs |rename as customSearch|search NOT author="splunk-system-user" | eval SearchString=if(isnotnull(customSearch),customSearch,eventSearch)| addtotals fieldname=duration *duration_secs  | eval search_time=tostring(duration, "duration") | stats sum(eventCount) AS Count dc(author) | fieldformat Count=tostring(Count, "commas")
Splunk Server Lookup
 | rest splunk_server=* /services/server/info | mvexpand server_roles | search server_roles!=search_peer | rename server_roles AS role splunk_server AS host | table host guid role version

Currently Logged in USers
| rest /services/authentication/current-context | search NOT username="splunk-system-user" | table username roles updated

One or more of your indexers is reporting an abnormal state.
rest splunk_server=local /services/search/distributed/peers/
| where status!="Up"
| fields peerName, status
| rename peerName as Instance, status as Status

| rest /services/server/introspection/indexer | search NOT splunk_server="*SearchHead*" | fields splunk_server, title, average_KBps, status, reason
| eval average_KBps = round(average_KBps, 0)
| eval status= if(status=="normal", status, status." - ".reason)
| fields - reason
| rename splunk_server as Instance, average_KBps as "Average KB/s (last 30s)", status as Status

|rest splunk_server=local /services/search/distributed/peers/ | table peerName host title  numberOfCores os_build os_name os_version physicalMemoryMB status version updated

Critical System Physical Memory Usage
One or more instances has exceeded 90% memory usage
| rest splunk_server_group=* /services/server/status/resource-usage/hostwide
| eval percentage=round(mem_used/mem,3)*100
| where percentage > 90
| fields splunk_server, percentage, mem_used, mem
| rename splunk_server AS Instance, mem AS "Physical memory installed (MB)", percentage AS "Memory used (%)", mem_used AS "Memory used (MB)"

Near Critical Disk Usage
You have used 80% of your disk capacity
| rest splunk_server_group=* /services/server/status/partitions-space
| eval free = if(isnotnull(available), available, free)
| eval usage = capacity - free
| eval pct_usage = floor(usage / capacity * 100)
| where pct_usage > 80
| stats first(fs_type) as fs_type first(capacity) AS capacity first(usage) AS usage first(pct_usage) AS pct_usage by splunk_server, mount_point
| eval usage = round(usage / 1024, 2)
| eval capacity = round(capacity / 1024, 2)
| rename splunk_server AS Instance mount_point as "Mount Point", fs_type as "File System Type", usage as "Usage (GB)", capacity as "Capacity (GB)", pct_usage as "Usage (%)"

Saturated Event-Processing Queues
One or more of your indexer queues is reporting a fill percentage, averaged over the last 15 minutes, of 90% or more.
| rest splunk_server_group=*    /services/server/introspection/queues
| search title=tcpin_queue OR title=parsingQueue OR title=aggQueue OR title=typingQueue OR title=indexQueue
| eval 15min_fill_perc = round(value_cntr3_size_bytes_lookback / max_size_bytes * 100,2)
| fields title 15min_fill_perc splunk_server
| where '15min_fill_perc' > 90
| rename splunk_server as Instance, title AS "Queue name", 15min_fill_perc AS "Average queue fill percentage (last 15min)"

Total License Usage Near Daily Quota - You have used 90% of your total daily license quota
| rest splunk_server_group=* /services/licenser/pools
| join type=outer stack_id splunk_server [rest splunk_server_group=* /services/licenser/groups | search is_active=1 | eval stack_id=stack_ids | fields splunk_server stack_id is_active]
| search is_active=1
| fields splunk_server, stack_id, used_bytes
| join type=outer stack_id splunk_server [rest splunk_server_group=* /services/licenser/stacks | eval stack_id=title | eval stack_quota=quota | fields splunk_server stack_id stack_quota]
| stats sum(used_bytes) as used_bytes max(stack_quota) as stack_quota by splunk_server
| eval usedGB=round(used_bytes/1024/1024/1024,3)
| eval totalGB=round(stack_quota/1024/1024/1024,3)
| eval percentage=round(usedGB / totalGB, 3)*100
| fields splunk_server, percentage, usedGB, totalGB
| where percentage > 90
| rename splunk_server AS Instance, percentage AS "License quota used (%)", usedGB AS "License quota used (GB)", totalGB as "Total license quota (GB)"

Memory Usage
| rest  /services/server/status/resource-usage/hostwide | eval cpu_count = if(isnull(cpu_count), "N/A", cpu_count) | eval cpu_usage = cpu_system_pct + cpu_user_pct | eval mem_used_pct = round(mem_used / mem * 100 , 2) | eval mem_used = round(mem_used, 0) | eval mem = round(mem, 0) | fields splunk_server, cpu_count, cpu_usage, mem, mem_used, mem_used_pct | sort - cpu_usage, -mem_used | rename splunk_server AS Instance, cpu_count AS "CPU Cores", cpu_usage AS "CPU Usage (%)", mem AS "Physical Memory Capacity (MB)", mem_used AS "Physical Memory Usage (MB)", mem_used_pct AS "Physical Memory Usage (%)"

User Search quoto usage
| rest  /services/search/jobs | eval diskUsageMB=diskUsage/1024/1024 | rename eai:acl.owner as UserName | stats sum(diskUsageMB) as totalDiskUsage_mb by UserName

Other REST API Calls
| rest /services/authentication/users
| rest splunk_server=local /services/alerts/reviewstatuses/
| rest splunk_server=local count=0 /services/alerts/correlationsearches
| rest splunk_server=local count=0 /services/saved/searches | table author title search updated dispatch.earliest_time dispatch.latest_time
| rest /services/saved/searches | table title search
| rest /services/server/info
| rest /services/server/status/resource-usage/splunk-processes | table process fd_used mem_used read_mb written_mb normalized_pct_cpu page_faults pct_cpu pct_memory pid ppid search_props.type search_props.user splunk_server
| rest /services/server/status/resource-usage/hostwide | table splunk_server forks  mem mem_used cpu_count cpu_idle_pct cpu_idle_pct cpu_user_pct normalized_load_avg_1min
| rest /services/server/status/partitions-space | table splunk_server fs_type title capacity available free mount_point updated
| rest /services/server/status/dispatch-artifacts | search splunk_server="SearchHead*" | transpose
| rest /services/server/settings | table splunk_server host httpport kvStorePort mgmtHostPort minFreeSpace sessionTimeout updated
| rest /services/server/logger | table splunk_server title level updated
| rest /services/data/ui/views  | table eai:appName label title eai:data
| rest /services/search/distributed/peers | table title splunk_server numberOfCores replicationStatus searchable_indexes server_roles cpu_arch status is_https physicalMemoryMB updated
| rest /services/search/jobs  | dedup label | table author label keywords normalizedSearch
| rest /services/properties/| dedup title  | table title
| rest /services/datamodel/model | table title  displayName
| rest /services/data/inputs/tcp/ssl | table host sslVersions cipherSuite rootCA serverCert sslVersions
| rest /services/apps/local | table title label description eai:acl.perms.write