Splunk Admin Manual Version: 4.2.3 Generated: 10/06/2011 03:45 pm Copyright Splunk, Inc. All Rights Reserved Table of Contents Welcome to Splunk administration....................................................................................................1 What's in this manual.................................................................................................................1 What is Splunk?.........................................................................................................................2 What to do first.....................................................................................................................................3 Start Splunk................................................................................................................................3 Configure Splunk to start at boot time........................................................................................5 Find Splunk Manager in Splunk Web.........................................................................................6 Install your license......................................................................................................................6 Change default values...............................................................................................................7 Bind Splunk to an IP.................................................................................................................10 Specify a proxy server..............................................................................................................11 Functional differences between *nix and Windows in Splunk operations................................11 Getting started for Windows admins...............................................................................................13 Introduction for Windows admins.............................................................................................13 Try Splunk out..........................................................................................................................14 Learn what Splunk does...........................................................................................................15 Integrate Splunk into your enterprise.......................................................................................16 Get the most out of Splunk.......................................................................................................16 Optimize Splunk for peak performance....................................................................................16 Put Splunk onto system images...............................................................................................17 Integrate a universal forwarder onto a system image..............................................................19 Integrate full Splunk onto a system image...............................................................................20 Deploying Splunk on Windows.................................................................................................21 Manage Splunk licenses...................................................................................................................25 How Splunk licensing works.....................................................................................................25 Types of Splunk licenses.........................................................................................................26 Groups, stacks, pools, and other terminology..........................................................................27 Install a license.........................................................................................................................29 Configure a license master......................................................................................................30 Configure a license slave.........................................................................................................31 Create or edit a license pool....................................................................................................31 Add an indexer to a license pool..............................................................................................34 Manage your licenses..............................................................................................................34 About license violations............................................................................................................38 Migrate to the new Splunk licenser..........................................................................................41 More about Splunk Free...........................................................................................................43 Manage licenses from the CLI.................................................................................................45 Meet Splunk Web and Splunk apps..................................................................................................48 What's Splunk Web?................................................................................................................48 What are apps and add-ons?....................................................................................................49 Where to get more apps and add-ons......................................................................................52 i Table of Contents Meet Splunk Web and Splunk apps App architecture and object ownership....................................................................................53 Manage app and add-on objects..............................................................................................55 How to configure Splunk...................................................................................................................57 Splunk configuration methods..................................................................................................57 About Splunk Manager..............................................................................................................58 About configuration files...........................................................................................................60 Configuration file precedence..................................................................................................64 Attribute precedence within a single props.conf file.................................................................68 Configuration parameters and the data pipeline......................................................................70 Indexing with Splunk.........................................................................................................................73 What's a Splunk index?............................................................................................................73 How indexing works.................................................................................................................73 Index time versus search time..................................................................................................76 Advanced indexing strategy......................................................................................................77 Add and manage users.....................................................................................................................79 About users and roles..............................................................................................................79 Set up user authentication.......................................................................................................80 Set up user authentication with Splunk's built-in system..........................................................80 Set up user authentication with LDAP......................................................................................82 Set up user authentication with external systems....................................................................90 Use single sign-on (SSO) with Splunk.....................................................................................95 Delete user accounts using the CLI.........................................................................................99 User language and locale........................................................................................................99 Configure user session timeouts............................................................................................100 Add and edit roles..................................................................................................................102 Manage indexes...............................................................................................................................107 About managing indexes........................................................................................................107 Set up multiple indexes..........................................................................................................107 Set limits on disk usage.........................................................................................................113 How Splunk stores indexes....................................................................................................114 Configure segmentation to manage disk usage.....................................................................125 Configure custom segmentation for a host, source, or source type.......................................127 Move the index database.......................................................................................................128 Remove indexed data from Splunk........................................................................................130 Optimize indexes....................................................................................................................133 Define alerts.....................................................................................................................................134 How does alerting work in Splunk..........................................................................................134 Set up alerts in savedsearches.conf......................................................................................135 Configure scripted alerts........................................................................................................142 ii Table of Contents Define alerts Send SNMP traps to other systems.......................................................................................144 Set up backups and retention policies..........................................................................................147 What you can back up............................................................................................................147 How much space you will need..............................................................................................147 Back up indexed data.............................................................................................................148 Back up configuration information..........................................................................................151 Set a retirement and archiving policy.....................................................................................151 Archive indexed data..............................................................................................................153 Restore archived indexed data..............................................................................................156 Configure data security...................................................................................................................161 What you can secure with Splunk..........................................................................................161 Use SSL (HTTPS) for secure access to Splunk Web............................................................163 Use SSL for secure intra-Splunk communication...................................................................165 Use SSL to encrypt and authenticate data from forwarders...................................................172 Configure archive signing.......................................................................................................174 Configure IT data block signing..............................................................................................175 Cryptographically sign audit events........................................................................................178 Audit Splunk activity...............................................................................................................180 Configure event hashing........................................................................................................182 Hardening standards..............................................................................................................185 Manage search jobs.........................................................................................................................188 About jobs and job management...........................................................................................188 Manage jobs in Splunk Web..................................................................................................189 Manage jobs in the OS...........................................................................................................190 Use Splunk's command line interface (CLI)..................................................................................192 About the CLI.........................................................................................................................192 Get help with the CLI..............................................................................................................193 Use the CLI to administer a remote Splunk server.................................................................195 CLI admin commands............................................................................................................197 Configuration file reference............................................................................................................199 admon.conf.............................................................................................................................199 alert_actions.conf....................................................................................................................200 app.conf..................................................................................................................................205 audit.conf.................................................................................................................................209 authentication.conf..................................................................................................................212 authorize.conf..........................................................................................................................219 commands.conf.......................................................................................................................223 crawl.conf................................................................................................................................226 default.meta.conf.....................................................................................................................228 iii Table of Contents Configuration file reference deploymentclient.conf.............................................................................................................229 distsearch.conf........................................................................................................................232 eventdiscoverer.conf...............................................................................................................236 event_renderers.conf..............................................................................................................237 eventtypes.conf.......................................................................................................................239 fields.conf................................................................................................................................240 indexes.conf............................................................................................................................242 inputs.conf...............................................................................................................................251 limits.conf................................................................................................................................266 literals.conf..............................................................................................................................281 macros.conf.............................................................................................................................282 multikv.conf.............................................................................................................................284 outputs.conf.............................................................................................................................287 pdf_server.conf.......................................................................................................................300 perfmon.conf...........................................................................................................................304 procmon-filters.conf.................................................................................................................306 props.conf...............................................................................................................................307 pubsub.conf.............................................................................................................................323 regmon-filters.conf..................................................................................................................324 report_server.conf...................................................................................................................326 restmap.conf...........................................................................................................................330 savedsearches.conf................................................................................................................334 searchbnf.conf.........................................................................................................................341 segmenters.conf......................................................................................................................344 server.conf..............................................................................................................................346 serverclass.conf......................................................................................................................354 serverclass.seed.xml.conf.......................................................................................................358 setup.xml.conf.........................................................................................................................360 source-classifier.conf..............................................................................................................363 sourcetypes.conf.....................................................................................................................364 sysmon.conf............................................................................................................................366 tags.conf..................................................................................................................................367 tenants.conf.............................................................................................................................369 times.conf................................................................................................................................370 transactiontypes.conf..............................................................................................................373 transforms.conf.......................................................................................................................375 user-seed.conf........................................................................................................................383 web.conf..................................................................................................................................384 wmi.conf..................................................................................................................................392 workflow_actions.conf.............................................................................................................396 viewstates.conf........................................................................................................................400 iv Table of Contents Troubleshooting...............................................................................................................................403 What Splunk logs about itself.................................................................................................403 Work with metrics.log.............................................................................................................406 Contact Support.....................................................................................................................409 Anonymize data samples to send to support.........................................................................413 Not finding the events you're looking for?..............................................................................415 SuSE Linux: unable to get a properly formatted response from the server............................416 Command line tools for use with Support's direction.............................................................417 Troubleshoot configurations with btoo.l..................................................................................419 v Welcome to Splunk administration What's in this manual What's in this manual This manual contains information and procedures for the Splunk administrator. If you're responsible for configuring, running, and maintaining Splunk as a service for yourself or other users, this manual is for you. For example, learn how to: • add users and set up roles • configure data security • back up index and user data • manage the search jobs your users run and much more. Where is all the information about forwarding, distributed search, and other deployment strategies? Starting with the 4.2 release, we've moved that information into a separate manual, Distributed Deployment. There's a lot of new stuff there, as well. Check it out and let us know what you think! How about the data inputs and event processing stuff? Yep, we've moved that, too -- to a second new manual, Getting Data In. There you'll find all the great information previously in the Admin manual, along with a lot of new topics. That manual covers the entire process of getting data into Splunk -- from setting up your data inputs to configuring event processing. And the information about event types and source types etc? We moved that information into a new manual a while back, actually, with the 4.1 release. For information about Splunk knowledge, new and old, look in the Knowledge Manager Manual. Looking for help with searching in Splunk? Check out the User Manual and the Search Reference Manual for all things search. In particular, you might want to check out the Search Cheatsheet if you're looking for a quick list of common examples. Make a PDF If you'd like a PDF of any version of this manual, click the pdf version link above the table of contents bar on the left side of this page. A PDF version of the manual is generated on the fly for you, and you can save it or print it out to read later. 1 What is Splunk? What is Splunk? Splunk is an IT search engine. • You can use Splunk to search and navigate IT data from applications, servers, and network devices in real-time. • Data sources include logs, configurations, messages, alerts, scripts, code, metrics, etc. • Splunk lets you search, navigate, alert, and report on all your IT data in real-time using Splunk Web. Learn more about what Splunk is, what it does, and how it's different. 2 What to do first Start Splunk Start Splunk This topic provides brief instructions for starting Splunk. If you are new to Splunk, we recommend reviewing the User Manual first. Start Splunk on Windows On Windows, Splunk is installed by default into C:\Program Files\Splunk. Many examples in the Splunk documentation use $SPLUNK_HOME to indicate the Splunk installation, or home, directory. You can replace the string $SPLUNK_HOME (and the Windows variant %SPLUNK_HOME%) with C:\Program Files\Splunk if you installed Splunk into the default directory. You can start and stop Splunk on Windows in one of the following ways: 1. Start and stop Splunk processes via the Windows Services control panel (accessible from Start -> Control Panel -> Administrative Tools -> Services) • Server daemon: splunkd • Web interface: splunkweb 2. Start and stop Splunk services from a command prompt by using the NET START <service> or NET STOP <service> commands: • Server daemon: splunkd • Web interface: splunkweb 3. Start, stop, and restart both processes at once by going to %SPLUNK_HOME%\bin and typing > splunk [start|stop|restart] Start Splunk on UNIX Start Splunk From a shell prompt on the Splunk sever host, run this command: # splunk start This starts both splunkd (indexer and other back-end processes) and splunkweb (the Splunk Web interface). To start them individually, type: # splunk start splunkd or 3 # splunk start splunkweb Note: If startwebserver is disabled in web.conf, manually starting splunkweb does not override that setting. If it is disabled in the configuration file, it will not start. To restart Splunk (splunkd or splunkweb) type: # splunk restart # splunk restart splunkd # splunk restart splunkweb Stop Splunk To shut down Splunk, run this command: # splunk stop To stop splunkd and Splunk Web individually, type: # splunk stop splunkd or # splunk stop splunkweb Check if Splunk is running To check if Splunk is running, type this command at the shell prompt on the server host: # splunk status You should see this output: splunkd is running (PID: 3162). splunk helpers are running (PIDs: 3164). splunkweb is running (PID: 3216). Note: On Unix systems, you must be logged in as the user who runs Splunk to run the splunk status command. Other users cannot read the necessary files to report status correctly. You can also use ps to check for running Splunk processes: # ps aux | grep splunk | grep -v grep Solaris users, type -ef instead of aux: # ps -ef | grep splunk | grep -v grep 4