Cisco APIC Troubleshooting Guide, Release 4.1(x) FirstPublished:2019-01-30 AmericasHeadquarters CiscoSystems,Inc. 170WestTasmanDrive SanJose,CA95134-1706 USA http://www.cisco.com Tel:408526-4000 800553-NETS(6387) Fax:408527-0883 THESPECIFICATIONSANDINFORMATIONREGARDINGTHEPRODUCTSINTHISMANUALARESUBJECTTOCHANGEWITHOUTNOTICE.ALLSTATEMENTS, INFORMATION,ANDRECOMMENDATIONSINTHISMANUALAREBELIEVEDTOBEACCURATEBUTAREPRESENTEDWITHOUTWARRANTYOFANYKIND, EXPRESSORIMPLIED.USERSMUSTTAKEFULLRESPONSIBILITYFORTHEIRAPPLICATIONOFANYPRODUCTS. THESOFTWARELICENSEANDLIMITEDWARRANTYFORTHEACCOMPANYINGPRODUCTARESETFORTHINTHEINFORMATIONPACKETTHATSHIPPEDWITH THEPRODUCTANDAREINCORPORATEDHEREINBYTHISREFERENCE.IFYOUAREUNABLETOLOCATETHESOFTWARELICENSEORLIMITEDWARRANTY, CONTACTYOURCISCOREPRESENTATIVEFORACOPY. TheCiscoimplementationofTCPheadercompressionisanadaptationofaprogramdevelopedbytheUniversityofCalifornia,Berkeley(UCB)aspartofUCB'spublicdomainversionof theUNIXoperatingsystem.Allrightsreserved.Copyright©1981,RegentsoftheUniversityofCalifornia. NOTWITHSTANDINGANYOTHERWARRANTYHEREIN,ALLDOCUMENTFILESANDSOFTWAREOFTHESESUPPLIERSAREPROVIDED“ASIS"WITHALLFAULTS. CISCOANDTHEABOVE-NAMEDSUPPLIERSDISCLAIMALLWARRANTIES,EXPRESSEDORIMPLIED,INCLUDING,WITHOUTLIMITATION,THOSEOF MERCHANTABILITY,FITNESSFORAPARTICULARPURPOSEANDNONINFRINGEMENTORARISINGFROMACOURSEOFDEALING,USAGE,ORTRADEPRACTICE. INNOEVENTSHALLCISCOORITSSUPPLIERSBELIABLEFORANYINDIRECT,SPECIAL,CONSEQUENTIAL,ORINCIDENTALDAMAGES,INCLUDING,WITHOUT LIMITATION,LOSTPROFITSORLOSSORDAMAGETODATAARISINGOUTOFTHEUSEORINABILITYTOUSETHISMANUAL,EVENIFCISCOORITSSUPPLIERS HAVEBEENADVISEDOFTHEPOSSIBILITYOFSUCHDAMAGES. AnyInternetProtocol(IP)addressesandphonenumbersusedinthisdocumentarenotintendedtobeactualaddressesandphonenumbers.Anyexamples,commanddisplayoutput,network topologydiagrams,andotherfiguresincludedinthedocumentareshownforillustrativepurposesonly.AnyuseofactualIPaddressesorphonenumbersinillustrativecontentisunintentional andcoincidental. Allprintedcopiesandduplicatesoftcopiesofthisdocumentareconsidereduncontrolled.Seethecurrentonlineversionforthelatestversion. Ciscohasmorethan200officesworldwide.AddressesandphonenumbersarelistedontheCiscowebsiteatwww.cisco.com/go/offices. CiscoandtheCiscologoaretrademarksorregisteredtrademarksofCiscoand/oritsaffiliatesintheU.S.andothercountries.ToviewalistofCiscotrademarks,gotothisURL:www.cisco.com gotrademarks.Third-partytrademarksmentionedarethepropertyoftheirrespectiveowners.TheuseofthewordpartnerdoesnotimplyapartnershiprelationshipbetweenCiscoandany othercompany.(1721R) ©2019CiscoSystems,Inc.Allrightsreserved. CONTEN TS PREFACE Preface xiii Audience xiii DocumentConventions xiii RelatedDocumentation xv DocumentationFeedback xv CHAPTER 1 NewandChanged 1 NewandChangedInformation 1 CHAPTER 2 TroubleshootingOverview 3 TroubleshootingBasics 4 CHAPTER 3 TroubleshootingAPICCrashScenarios 7 CiscoAPICClusterFailureScenarios 7 ClusterTroubleshootingScenarios 7 ClusterFaults 10 TroubleshootingApplicationCentricInfrastructureCrashScenarios 11 TroubleshootingFabricNodeandProcessCrash 11 APICProcessCrashVerificationandRestart 13 TroubleshootinganAPICProcessCrash 15 CHAPTER 4 RecoveringCiscoAPICPasswordsandAccessingSpecialLogins 17 RecoveringtheAPICPassword 17 UsingtheRescue-userAccounttoErasetheCiscoAPICConfigurationUsingtheNX-OSStyleCLI 18 UsingtheFallbackLoginDomaintoLogintotheLocalDatabase 18 CiscoAPICTroubleshootingGuide,Release4.1(x) iii Contents CHAPTER 5 CiscoAPICTroubleshootingOperations 21 ShuttingDowntheAPICSystem 21 ShuttingDowntheAPICControllerUsingtheGUI 21 UsingtheAPICReloadOptionUsingtheGUI 22 ControllingtheLEDLocatorUsingtheGUI 23 CHAPTER 6 UsingtheCiscoAPICTroubleshootingTools 25 EnablingandViewingACLContractandDenyLogs 26 AboutACLContractPermitandDenyLogs 26 EnablingACLContractPermitandDenyLoggingUsingtheGUI 27 EnablingACLContractPermitLoggingUsingtheNX-OSCLI 28 EnablingACLContractPermitLoggingUsingtheRESTAPI 28 EnablingTabooContractDenyLoggingUsingtheGUI 29 EnablingTabooContractDenyLoggingUsingtheNX-OSCLI 30 EnablingTabooContractDenyLoggingUsingtheRESTAPI 30 ViewingACLPermitandDenyLogsUsingtheGUI 31 ViewingACLPermitandDenyLogsUsingtheRESTAPI 32 ViewingACLPermitandDenyLogsUsingtheNX-OSCLI 33 UsingAtomicCounterPoliciesforGatheringStatistics 35 AtomicCounters 35 AtomicCountersGuidelinesandRestrictions 36 ConfiguringAtomicCounters 37 EnablingAtomicCounters 37 TroubleshootingUsingAtomicCounterswiththeRESTAPI 38 EnablingandViewingDigitalOpticalMonitoringStatistics 39 EnablingDigitalOpticalMonitoringUsingtheGUI 39 EnablingDigitalOpticalMonitoringUsingtheRESTAPI 40 ViewingDigitalOpticalMonitoringStatisticsWiththeGUI 41 TroubleshootingUsingDigitalOpticalMonitoringWiththeRESTAPI 41 ViewingandUnderstandingHealthScores 42 HealthScoreTypes 42 FilteringbyHealthScore 43 ViewingTenantHealth 43 CiscoAPICTroubleshootingGuide,Release4.1(x) iv Contents ViewingFabricHealth 43 ViewingMOHealthinVisore 43 DebuggingHealthScoresUsingLogs 44 ViewingFaults 44 EnablingPortTrackingforUplinkFailureDetection 45 PortTrackingPolicyforFabricPortFailureDetection 45 ConfiguringPortTrackingUsingtheGUI 46 PortTrackingUsingtheNX-OSCLI 46 PortTrackingUsingtheRESTAPI 47 ConfiguringSNMPforMonitoringandManagingDevices 47 AboutSNMP 47 SNMPAccessSupportinACI 48 ConfiguringtheSNMPPolicyUsingtheGUI 48 ConfiguringanSNMPTrapDestinationUsingtheGUI 50 ConfiguringanSNMPTrapSourceUsingtheGUI 50 MonitoringtheSystemUsingSNMP 51 ConfiguringSPANforTrafficMonitoring 51 AboutSPAN 51 MultinodeSPAN 52 SPANGuidelinesandRestrictions 53 ConfiguringSPANUsingtheGUI 56 ConfiguringaTenantSPANSessionUsingtheCiscoAPICGUI 56 ConfiguringaSPANFilterGroupUsingtheAPICGUI 57 ConfiguringanAccessSPANPolicyUsingtheCiscoAPICGUI 58 ConfiguringaFabricSPANPolicyUsingtheCiscoAPICGUI 59 ConfiguringaLayer3EPGSPANSessionforExternalAccessUsingtheAPICGUI 60 ConfiguringaDestinationGroupforanAccessSPANPolicyUsingtheCiscoAPICGUI 61 ConfiguringaDestinationGroupforaFabricSPANPolicyUsingtheCiscoAPICGUI 61 ConfiguringSPANUsingtheNX-OSStyleCLI 62 ConfiguringLocalSPANinAccessMode 62 ConfiguringaSPANFilterGroup 65 AssociatingaSPANFilterGroup 66 ConfiguringERSPANinAccessMode 68 ConfiguringERSPANinFabricMode 71 CiscoAPICTroubleshootingGuide,Release4.1(x) v Contents ConfiguringERSPANinTenantMode 73 ConfiguringaGlobalSPAN-On-DropSessionUsingtheCLI 75 ConfiguringSPANUsingtheRESTAPI 77 ConfiguringaFabricDestinationGroupforanERSPANDestinationUsingtheRESTAPI 77 ConfiguringaGlobalDropSourceGroupUsingtheRESTAPI 77 ConfiguringaLeafPortasaSPANDestinationUsingtheRESTAPI 77 ConfiguringaSPANAccessSourceGroupUsingtheRESTAPI 78 ConfiguringaSPANFabricSourceGroupUsingtheRESTAPI 78 ConfiguringanAccessDestinationGroupforanERSPANDestinationUsingtheRESTAPI 79 UsingStatistics 79 ViewingStatisticsintheGUI 80 SwitchStatisticsCommands 80 ManagingStatisticsThresholdsUsingtheGUI 81 StatisticsTroubleshootingScenarios 82 StatisticsCleanup 83 SpecifyingSyslogSourcesandDestinations 84 AboutSyslog 84 CreatingaSyslogDestinationandDestinationGroup 85 CreatingaSyslogSource 86 EnablingSyslogtoDisplayinNX-OSCLIFormat,UsingtheRESTAPI 87 DiscoveringPathsandTestingConnectivitywithTraceroute 88 AboutTraceroute 88 AboutWindowsandLinuxTraceroute 89 TracerouteGuidelinesandRestrictions 90 PerformingaTracerouteBetweenEndpoints 91 UsingtheTroubleshootingWizard 92 GettingStartedwiththeTroubleshootingWizard 92 GeneratingTroubleshootingReports 94 TopologyintheTroubleshootingWizard 95 UsingtheFaultsTroubleshootingScreen 96 UsingtheDrop/StatisticsTroubleshootingScreen 97 UsingtheContractsTroubleshootingScreen 99 UsingtheEventsTroubleshootingScreen 100 UsingtheTracerouteTroubleshootingScreen 100 CiscoAPICTroubleshootingGuide,Release4.1(x) vi Contents UsingtheAtomicCounterTroubleshootingScreen 102 UsingtheSPANTroubleshootingScreen 102 CreatingaSPANSessionUsingtheCiscoAPICTroubleshootingCLI 102 L4-L7ServicesValidatedScenarios 103 ListofAPIsforEndpointtoEndpointConnections 104 interactiveAPI 105 createsessionAPI 106 modifysessionAPI 107 atomiccounterAPI 107 tracerouteAPI 107 spanAPI 108 generatereportAPI 109 schedulereportAPI 109 getreportstatusAPI 110 getreportslistAPI 110 getsessionslistAPI 111 getsessiondetailAPI 111 deletesessionAPI 111 clearreportsAPI 112 contractsAPI 112 ListofAPIsforEndpointtoLayer3ExternalConnections 112 interactiveAPI 113 createsessionAPI 113 modifysessionAPI 114 atomiccounterAPI 115 tracerouteAPI 116 spanAPI 117 generatereportAPI 118 schedulereportAPI 119 getreportstatusAPI 120 getreportslistAPI 120 getsessionslistAPI 120 getsessiondetailAPI 121 deletesessionAPI 122 CiscoAPICTroubleshootingGuide,Release4.1(x) vii Contents clearreportsAPI 123 contractsAPI 123 ratelimitAPI 124 13extAPI 124 CheckingforConfigurationSynchronizationIssues 125 ViewingUserActivities 125 AccessingUserActivities 126 EmbeddedLogicAnalyzerModule 126 AbouttheEmbeddedLogicAnalyzerModule 126 GeneratinganELAMReportintheSimplifiedOutputforModularSwitches 126 GeneratinganELAMReportintheSimplifiedOutputforFixedForm-FactorSwitches 128 CHAPTER 7 ManuallyRemovingDisabledInterfacesandDecommissionedSwitchesfromtheGUI 129 ManuallyRemovingDisabledInterfacesandDecommissionedSwitchesfromtheGUI 129 CHAPTER 8 DecommissioningandRecommissioningSwitches 131 DecommissioningandRecommissioningSwitches 131 CHAPTER 9 TroubleshootingStepsforEndpointConnectivityProblems 133 TroubleshootingEndpointConnectivity 133 InspectingEndpointandTunnelInterfaceStatus 134 InspectingtheEndpointStatus 134 InspectingtheTunnelInterfaceStatus 135 ConnectinganSFPModule 135 CHAPTER 10 TroubleshootingEVPNType-2RouteAdvertisement 137 TroubleshootingEVPNType-2RouteDistributiontoaDCIG 137 CHAPTER 11 PerformingaRebuildoftheFabric 141 RebuildingtheFabric 141 CHAPTER 12 VerifyingIP-BasedEPGConfigurations 143 VerifyingIP-BasedEPGConfigurationsUsingtheGUI 143 CiscoAPICTroubleshootingGuide,Release4.1(x) viii Contents VerifyingIP-EPGConfigurationsUsingSwitchCommands 144 CHAPTER 13 RecoveringaDisconnectedLeaf 147 RecoveringaDisconnectedLeafUsingtheRESTAPI 147 CHAPTER 14 TroubleshootingaLoopbackFailure 149 IdentifyingaFailedLineCard 149 CHAPTER 15 DeterminingWhyaPIMInterfaceWasNotCreated 151 APIMInterfaceWasNotCreatedForanL3OutInterface 151 APIMInterfaceWasNotCreatedForaMulticastTunnelInterface 152 APIMInterfaceWasNotCreatedForaMulticast-EnabledBridgeDomain 152 CHAPTER 16 ConfirmingthePortSecurityInstallation 153 ConfirmingYourPortSecurityInstallationUsingVisore 153 ConfirmingYourHardwarePortSecurityInstallationUsingtheCiscoNX-OSCLI 153 CHAPTER 17 TroubleshootingQoSPolicies 157 TroubleshootingCiscoAPICQoSPolicies 157 CHAPTER 18 DeterminingtheSupportedSSLCiphers 159 AboutSSLCiphers 159 DeterminingtheSupportedSSLCiphersUsingtheCLI 160 CHAPTER 19 RemovingUnwanted_ui_Objects 161 RemovingUnwanted_ui_ObjectsUsingtheRESTAPI 163 CHAPTER 20 TroubleshootingMultipodandMulti-SiteIssues 165 TroubleshootingMultipodandMulti-Site 165 APPENDIX A acidiagCommand 167 APPENDIX B ConfiguringExportPoliciesforTroubleshooting 175 CiscoAPICTroubleshootingGuide,Release4.1(x) ix Contents AboutExportingFiles 175 FileExportGuidelinesandRestrictions 175 ConfiguringaRemoteLocation 176 ConfiguringaRemoteLocationUsingtheGUI 176 ConfiguringaRemoteLocationUsingtheRESTAPI 176 ConfiguringaRemoteLocationUsingtheNX-OSStyleCLI 177 SendinganOn-DemandTechSupportFile 178 SendinganOn-DemandTechSupportFileUsingtheGUI 178 SendinganOn-DemandTechSupportFileUsingtheRESTAPI 178 APPENDIX C FindingtheSwitchInventory 181 FindingYourSwitchInventoryUsingtheGUI 181 FindingYourSwitchInventoryUsingtheNX-OSCLI 181 FindingYourSwitchInventoryUsingtheRESTAPI 184 APPENDIX D CiscoAPICClusterManagement 187 ExpandingtheCiscoAPICCluster 187 ContractingtheCiscoAPICCluster 187 ClusterManagementGuidelines 188 ExpandingtheAPICClusterSize 189 ReducingtheAPICClusterSize 189 ReplacingCiscoAPICControllersintheCluster 190 ExpandingtheClusterExamples 191 ExpandingtheAPICClusterUsingtheGUI 191 ExpandingtheAPICClusterUsingtheRESTAPI 192 ContractingtheClusterExamples 192 ContractingtheAPICClusterUsingtheGUI 192 ContractingtheAPICClusterUsingtheRESTAPI 193 CommissioningandDecommissioningCiscoAPICControllers 194 CommissioningaCiscoAPICintheClusterUsingtheGUI 194 DecommissioningaCiscoAPICControllerintheClusterUsingtheGUI 194 ReplacingaCiscoAPICinaClusterUsingtheCLI 195 APPENDIX E CiscoAPICSSDReplacement 197 CiscoAPICTroubleshootingGuide,Release4.1(x) x