Tuesday, May 10, 2016

Splunk Frequently Troubleshooting Commands - Installation and Configuration

As part of Engineering role, i administer, manage and support bunch of Splunk platform infrastructure like forwarders, Deployment servers , Searchhead and  indexers. Below is a list of frequently used troubleshooting commands.

Post Installation Checks
  Splunk version: ./splunk version
  Splunk running status: ./splunk status
  Splunk management (splunkd) port: ./splunk show splunkd-port returns 8089
  Splunk Web port:  ./splunk show web-port returns 8000
  Splunk server name:  ./splunk show servername returns splunk01
  Default host name:  ./splunk show default-hostname returns splunk01
  Assuming you are already in the $SPLUNK_HOME/bin directory, run the commands shown above.

Post Configuration
  Configure the forwarder to send event data to your receiver
  splunk add forward-server
  splunk remove forward-server

  splunk list monitor (need to be splunk Admin to see logs monitored)
  splunk set deploy-poll -auth admin:password
  splunk enable listen port -auth username:password
  splunk enable boot-start -user siem ( as root)

  splunk enable deploy-client -auth admin:password
  splunk list deploy-clients (on deployment server - asks for admin PW)
  splunk reload deploy-server (after changing deployment server app)

  splunk list licenser-messages
  splunk list licenser-slaves

  splunk status
  splunk disable webserver
  splunk enable webserver

Is my receiver enabled and listening on the port I designated?
Execute this CLI command on the indexer: .
  /splunk display listen

Is my forwarder output setup active?
Execute this CLI command on the forwarder: .
  /splunk list forward-server

Is there any issues logged in splunkd.log on the forwarder:
  egrep 'ERROR|WARN' ~/splunkforwarder/var/log/splunk/splunkd.log

Is indexer getting any data from the forwarder?
Search with the time range set to Last 15 minutes:
  index=_internal ERROR OR host="forwarder_ip"  sourcetype=splunkd

Some useful Linux Commands

Useful Linux Commands
  ps -ef | grep splunk or | grep 8000 # will show siem users processes
  ps -ef | grep -i syslog # syslog process running
  netstat -an | grep 514 # is port 514 open?

  ps aux | grep -ie splunk | awk '{print "kill -9 " $2}'
  pkill -9 splunk
  ps -ef | grep "splunk"

Display all established, recently terminated, and listening TCP and UDP network connections along with the program name related to each socket:
  netstat -anp | grep -e tcp -e udp

  free -m # memory available
  df # free hard disc totals
  df - H # check mount points
  du IndexedData #directory size
  fuser . # shows all processes using file/dir
  find . -name outputs.conf # find all instances of outputs.conf from current dir down
  find . type f -exec grep -l "" {} \; # all instances of ip from here down