Oracle® Trace File Analyzer Collecting and Analyzing Oracle Database Diagnostic Data 18c E90669-07 May 2018 Oracle Trace File Analyzer Collecting and Analyzing Oracle Database Diagnostic Data, 18c E90669-07 Copyright © 2017, 2018, Oracle and/or its affiliates. All rights reserved. Primary Author: Nirmal Kumar Contributing Authors: Mark Bauer, Doug Williams Contributors: Gareth Chapman, Bill Burton This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable: U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agency- specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S. Government. This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group. This software or hardware and documentation may provide access to or information about content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services unless otherwise set forth in an applicable agreement between you and Oracle. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services, except as set forth in an applicable agreement between you and Oracle. Contents Preface Audience xii Documentation Accessibility xii Related Documentation xii Conventions xiii Third-Party License Information xiii Changes in this Release for Oracle Trace File Analyzer User’s Guide 18.2.0 REST Service xiv Oracle Cluster Health Advisor Integration xiv New SRDCs xv Metadata Search Capability xv 1 Oracle Trace File Analyzer 2 Getting Started with Oracle Trace File Analyzer 2.1 Supported Environments 2-1 2.2 Installing Oracle Trace File Analyzer on Linux or UNIX as root User in Daemon Mode 2-2 2.3 Installing Oracle Trace File Analyzer on Linux or UNIX as Non-root User in Non-Daemon Mode 2-3 2.4 Installing Oracle Trace File Analyzer on Microsoft Windows 2-3 2.5 Installing Oracle Trace File Analyzer on Microsoft Windows in Non-Daemon Mode 2-4 2.6 Oracle Trace File Analyzer Key Directories 2-4 2.7 Oracle Trace File Analyzer Command Interfaces 2-5 2.8 Masking Sensitive Data 2-5 2.9 Securing Access to Oracle Trace File Analyzer 2-6 2.10 Uninstalling Oracle Trace File Analyzer 2-7 iii 3 Automatic Diagnostic Collections 3.1 Collecting Diagnostics Automatically 3-1 3.2 Configuring Email Notification Details 3-2 4 On-demand Analysis and Diagnostic Collection 4.1 Collecting Diagnostics and Analyzing Logs On-Demand 4-1 4.2 Viewing System and Cluster Summary 4-2 4.3 Investigating Logs for Errors 4-2 4.4 Analyzing Logs Using the Included Tools 4-4 4.5 Searching Oracle Trace File Analyzer Metadata 4-6 4.6 Collecting Diagnostic Data and Using One Command Service Request Data Collections 4-6 4.7 Uploading Collections to Oracle Support 4-11 4.8 Changing Oracle Grid Infrastructure Trace Levels 4-13 4.8.1 tfactl dbglevel 4-13 5 Maintaining Oracle Trace File Analyzer to the Latest Version 6 Performing Custom Collections 6.1 Adjusting the Diagnostic Data Collection Period 6-1 6.2 Collecting from Specific Nodes 6-2 6.3 Collecting from Specific Components 6-2 6.4 Collecting from Specific Directories 6-3 6.5 Changing the Collection Name 6-4 6.6 Preventing Copying Zip Files and Trimming Files 6-5 6.7 Performing Silent Collection 6-6 6.8 Preventing Collecting Core Files 6-6 6.9 Collecting Incident Packaging Service (IPS) Packages 6-6 7 Managing and Configuring Oracle Trace File Analyzer 7.1 Querying Oracle Trace File Analyzer Status and Configuration 7-1 7.2 Managing the Oracle Trace File Analyzer Daemon 7-3 7.3 Managing the Repository 7-4 7.3.1 Purging the Repository Automatically 7-4 7.3.2 Purging the Repository Manually 7-5 7.4 Managing Collections 7-5 7.4.1 Including Directories 7-5 iv 7.4.2 Managing the Size of Collections 7-6 7.5 Configuring the Host 7-7 7.6 Configuring the Ports 7-7 7.7 Configuring SSL and SSL Certificates 7-8 7.7.1 Configuring SSL/TLS Protocols 7-8 7.7.2 Configuring Self-Signed Certificates 7-9 7.7.3 Configuring CA-Signed Certificates 7-10 7.7.4 Configuring SSL Cipher Suite 7-11 7.8 Configuring and Using REST 7-11 7.9 REST Authentication 7-17 7.10 Configuring Email Notification Details 7-18 8 Managing Oracle Database and Oracle Grid Infrastructure Diagnostic Data 8.1 Managing Automatic Diagnostic Repository Log and Trace Files 8-1 8.2 Managing Disk Usage Snapshots 8-2 8.3 Purging Oracle Database and Oracle Grid Infrastructure Logs 8-2 9 Troubleshooting Oracle Trace File Analyzer 9.1 Cluster Nodes are Not Showing As One Cluster When Viewed by Running the tfactl status Command 9-1 9.2 Oracle Trace File Analyzer is Not Starting and the init.tfa script is Missing After Reboot 9-2 9.3 Error Message Similar to "Can't locate **** in @inc (@inc contains:....)" 9-2 9.4 Non-Release Update Revisions (RURs) Oracle Trace File Analyzer Patching Fails on Remote Nodes 9-3 9.5 Non-Root Access is Not Enabled After Installation 9-3 9.6 TFA_HOME and Repository Locations are Moved After Patching or Upgrade 9-4 9.7 Oracle Trace File Analyzer Fails with TFA-00103 After Applying the July 2015 Release Update Revision (RUR) or Later 9-4 9.8 OSWatcher Parameters are Different After a Reboot or Otherwise Unexpectedly Different 9-10 9.9 Oracle Trace File Analyzer Installation or Oracle Trace File Analyzer Discovery (tfactl rediscover) Fails on Linux 7 9-11 9.10 OSWatcher Analyzer Fails When OSWatcher is Not Running from the TFA_HOME 9-12 9.11 Oracle Trace File Analyzer Fails to Start with com.sleepycat.je.EnvironmentLockedException Java Exception 9-12 9.12 Oracle Trace File Analyzer Startup Fails When Solution-Soft Time Machine Software is Installed, but Not Running on the System 9-13 9.13 Non-privileged User is Not Able to Run tfactl Commands? 9-13 v 9.14 Oracle Trace File Analyzer Daemon is Not Starting or Not Running? 9-14 A Oracle Trace File Analyzer Command-Line and Shell Options A.1 Running Administration Commands A-2 A.1.1 tfactl diagnosetfa A-3 A.1.2 tfactl host A-3 A.1.3 tfactl set A-4 A.1.4 tfactl access A-5 A.2 Running Summary and Analysis Commands A-7 A.2.1 tfactl summary A-7 A.2.2 tfactl changes A-9 A.2.3 tfactl events A-10 A.2.4 tfactl analyze A-11 A.2.5 tfactl run A-14 A.2.6 tfactl toolstatus A-15 A.3 Running Diagnostic Collection Commands A-16 A.3.1 tfactl diagcollect A-17 A.3.2 tfactl directory A-20 A.3.3 tfactl ips A-22 A.3.3.1 tfactl ips ADD A-25 A.3.3.2 tfactl ips ADD FILE A-26 A.3.3.3 tfactl ips COPY IN FILE A-26 A.3.3.4 tfactl ips REMOVE A-27 A.3.3.5 tfactl ips REMOVE FILE A-27 A.3.3.6 tfactl ips ADD NEW INCIDENTS PACKAGE A-27 A.3.3.7 tfactl ips GET REMOTE KEYS FILE A-28 A.3.3.8 tfactl ips USE REMOTE KEYS FILE A-28 A.3.3.9 tfactl ips CREATE PACKAGE A-28 A.3.3.10 tfactl ips FINALIZE PACKAGE A-30 A.3.3.11 tfactl ips GENERATE PACKAGE A-30 A.3.3.12 tfactl ips DELETE PACKAGE A-30 A.3.3.13 tfactl ips GET MANIFEST FROM FILE A-31 A.3.3.14 tfactl ips GET METADATA A-31 A.3.3.15 tfactl ips PACK A-31 A.3.3.16 tfactl ips SET CONFIGURATION A-33 A.3.3.17 tfactl ips SHOW CONFIGURATION A-33 A.3.3.18 tfactl ips SHOW PACKAGE A-33 A.3.3.19 tfactl ips SHOW FILES PACKAGE A-34 A.3.3.20 tfactl ips SHOW INCIDENTS PACKAGE A-34 A.3.3.21 tfactl ips SHOW PROBLEMS A-34 vi A.3.3.22 tfactl ips UNPACK FILE A-35 A.3.3.23 tfactl ips UNPACK PACKAGE A-35 A.3.4 tfactl collection A-35 A.3.5 tfactl print A-35 A.3.6 tfactl purge A-38 A.3.7 tfactl managelogs A-38 Index vii List of Examples 3-1 tfactl set smtp 3-3 4-1 Analyzing logs 4-2 4-2 Diagnostic Collection 4-10 4-3 One command SRDC 4-10 6-1 Show Incidents 6-7 6-2 Show Problems 6-8 6-3 Show Packages 6-9 6-4 IPS Collect 6-9 7-1 Print Configuration 7-2 7-2 tfactl set smtp 7-19 viii List of Figures 3-1 Automatic Diagnostic Collections 3-1 4-1 On-Demand Collections 4-2 ix List of Tables 2-1 Key Oracle Trace File Analyzer Directories 2-4 2-2 Oracle Trace File Interfaces 2-5 3-1 Log Entries that Trigger Automatic collection 3-2 3-2 tfactl diagnosetfa Command Parameters 3-3 4-1 Tools included in Linux and UNIX 4-4 4-2 Tools included in Microsoft Windows 4-5 4-3 One Command Service Request Data Collections 4-7 4-4 SRDC collections 4-9 4-5 tfactl dbglevel Command Parameters 4-14 6-1 Ways to Specify the Collection Period 6-1 6-2 Component Options 6-3 6-3 tfactl ips Command Parameters 6-6 7-1 Configuration Listing and Descriptions 7-2 7-2 REST Command Parameters 7-12 7-3 Print API 7-13 7-4 Diagcollect API 7-16 7-5 Download API 7-17 7-6 tfactl diagnosetfa Command Parameters 7-19 A-1 Basic TFACTL commands A-2 A-2 tfactl diagnosetfa Command Parameters A-3 A-3 tfactl set Command Parameters A-4 A-4 tfactl access Command Parameters A-6 A-5 tfactl analyze Command Parameters A-12 A-6 tfactl analyze -type Parameter Arguments A-13 A-7 tfactl run Command Parameters A-14 A-8 tfactl run Analysis Tools Parameters A-14 A-9 tfactl run Profiling Tools Parameters A-15 A-10 tfactl toolstatus Output A-15 A-11 tfactl directory Command Parameters A-21 A-12 tfactl ips Command Parameters A-23 A-13 tfactl ips ADD Command Parameters A-25 A-14 tfactl ips ADD FILE Command Parameters A-26 A-15 tfactl ips COPY IN FILE Command Parameters A-26 A-16 tfactl ips REMOVE Command Parameters A-27 A-17 tfactl ips ADD NEW INCIDENTS PACKAGE Command Parameters A-28 x