Cisco APIC Troubleshooting Guide LastModified:2018-04-23 AmericasHeadquarters CiscoSystems,Inc. 170WestTasmanDrive SanJose,CA95134-1706 USA http://www.cisco.com Tel:408526-4000 800553-NETS(6387) Fax:408527-0883 THESPECIFICATIONSANDINFORMATIONREGARDINGTHEPRODUCTSINTHISMANUALARESUBJECTTOCHANGEWITHOUTNOTICE.ALLSTATEMENTS, INFORMATION,ANDRECOMMENDATIONSINTHISMANUALAREBELIEVEDTOBEACCURATEBUTAREPRESENTEDWITHOUTWARRANTYOFANYKIND, EXPRESSORIMPLIED.USERSMUSTTAKEFULLRESPONSIBILITYFORTHEIRAPPLICATIONOFANYPRODUCTS. THESOFTWARELICENSEANDLIMITEDWARRANTYFORTHEACCOMPANYINGPRODUCTARESETFORTHINTHEINFORMATIONPACKETTHATSHIPPEDWITH THEPRODUCTANDAREINCORPORATEDHEREINBYTHISREFERENCE.IFYOUAREUNABLETOLOCATETHESOFTWARELICENSEORLIMITEDWARRANTY, CONTACTYOURCISCOREPRESENTATIVEFORACOPY. TheCiscoimplementationofTCPheadercompressionisanadaptationofaprogramdevelopedbytheUniversityofCalifornia,Berkeley(UCB)aspartofUCB'spublicdomainversionof theUNIXoperatingsystem.Allrightsreserved.Copyright©1981,RegentsoftheUniversityofCalifornia. NOTWITHSTANDINGANYOTHERWARRANTYHEREIN,ALLDOCUMENTFILESANDSOFTWAREOFTHESESUPPLIERSAREPROVIDED“ASIS"WITHALLFAULTS. CISCOANDTHEABOVE-NAMEDSUPPLIERSDISCLAIMALLWARRANTIES,EXPRESSEDORIMPLIED,INCLUDING,WITHOUTLIMITATION,THOSEOF MERCHANTABILITY,FITNESSFORAPARTICULARPURPOSEANDNONINFRINGEMENTORARISINGFROMACOURSEOFDEALING,USAGE,ORTRADEPRACTICE. INNOEVENTSHALLCISCOORITSSUPPLIERSBELIABLEFORANYINDIRECT,SPECIAL,CONSEQUENTIAL,ORINCIDENTALDAMAGES,INCLUDING,WITHOUT LIMITATION,LOSTPROFITSORLOSSORDAMAGETODATAARISINGOUTOFTHEUSEORINABILITYTOUSETHISMANUAL,EVENIFCISCOORITSSUPPLIERS HAVEBEENADVISEDOFTHEPOSSIBILITYOFSUCHDAMAGES. AnyInternetProtocol(IP)addressesandphonenumbersusedinthisdocumentarenotintendedtobeactualaddressesandphonenumbers.Anyexamples,commanddisplayoutput,network topologydiagrams,andotherfiguresincludedinthedocumentareshownforillustrativepurposesonly.AnyuseofactualIPaddressesorphonenumbersinillustrativecontentisunintentional andcoincidental. Allprintedcopiesandduplicatesoftcopiesofthisdocumentareconsidereduncontrolled.Seethecurrentonlineversionforthelatestversion. Ciscohasmorethan200officesworldwide.AddressesandphonenumbersarelistedontheCiscowebsiteatwww.cisco.com/go/offices. CiscoandtheCiscologoaretrademarksorregisteredtrademarksofCiscoand/oritsaffiliatesintheU.S.andothercountries.ToviewalistofCiscotrademarks,gotothisURL:www.cisco.com gotrademarks.Third-partytrademarksmentionedarethepropertyoftheirrespectiveowners.TheuseofthewordpartnerdoesnotimplyapartnershiprelationshipbetweenCiscoandany othercompany.(1721R) ©2014–2018CiscoSystems,Inc.Allrightsreserved. CONTEN TS PREFACE Preface xi Audience xi DocumentConventions xi RelatedDocumentation xiii DocumentationFeedback xiii CHAPTER 1 NewandChanged 1 NewandChangedInformation 1 CHAPTER 2 TroubleshootingOverview 3 TroubleshootingBasics 4 CHAPTER 3 TroubleshootingtheCiscoAPICCluster 7 ClusterTroubleshootingScenarios 7 ClusterFaults 10 CHAPTER 4 RecoveringCiscoAPICPasswordsandAccessingSpecialLogins 13 RecoveringtheAPICPassword 13 UsingtheRescue-userAccounttoErasetheCiscoAPICConfigurationUsingtheNX-OSStyleCLI 14 UsingtheFallbackLoginDomaintoLogintotheLocalDatabase 14 CHAPTER 5 CiscoAPICTroubleshootingOperations 17 ShuttingDowntheAPICSystem 17 ShuttingDowntheAPICControllerUsingtheGUI 17 UsingtheAPICReloadOptionUsingtheGUI 18 ControllingtheLEDLocatorUsingtheGUI 19 CiscoAPICTroubleshootingGuide iii Contents CHAPTER 6 UsingtheCiscoAPICTroubleshootingTools 21 EnablingandViewingACLContractandDenyLogs 22 AboutACLContractPermitandDenyLogs 22 EnablingACLContractPermitandDenyLoggingUsingtheGUI 22 EnablingACLContractPermitLoggingUsingtheNX-OSCLI 23 EnablingACLContractPermitLoggingUsingtheRESTAPI 23 EnablingTabooContractDenyLoggingUsingtheGUI 24 EnablingTabooContractDenyLoggingUsingtheNX-OSCLI 25 EnablingTabooContractDenyLoggingUsingtheRESTAPI 25 ViewingACLPermitandDenyLogsUsingtheGUI 26 ViewingACLPermitandDenyLogsUsingtheRESTAPI 27 ViewingACLPermitandDenyLogsUsingtheNX-OSCLI 28 UsingAtomicCounterPoliciesforGatheringStatistics 29 AtomicCounters 29 AtomicCountersGuidelinesandRestrictions 30 ConfiguringAtomicCounters 31 EnablingAtomicCounters 31 TroubleshootingUsingAtomicCounterswiththeRESTAPI 32 EnablingandViewingDigitalOpticalMonitoringStatistics 33 EnablingDigitalOpticalMonitoringUsingtheGUI 33 EnablingDigitalOpticalMonitoringUsingtheRESTAPI 34 ViewingDigitalOpticalMonitoringStatisticsWiththeGUI 35 TroubleshootingUsingDigitalOpticalMonitoringWiththeRESTAPI 35 ViewingandUnderstandingHealthScores 36 HealthScoreTypes 36 FilteringbyHealthScore 37 ViewingTenantHealth 37 ViewingFabricHealth 37 ViewingMOHealthinVisore 37 DebuggingHealthScoresUsingLogs 38 ViewingFaults 38 EnablingPortTrackingforUplinkFailureDetection 39 PortTrackingPolicyforFabricPortFailureDetection 39 CiscoAPICTroubleshootingGuide iv Contents PortTrackingUsingtheGUI 40 PortTrackingUsingtheNX-OSCLI 40 PortTrackingUsingtheRESTAPI 41 ConfiguringSNMPforMonitoringandManagingDevices 41 AboutSNMP 41 SNMPAccessSupportinACI 42 ConfiguringtheSNMPPolicyUsingtheGUI 42 ConfiguringanSNMPTrapDestinationUsingtheGUI 44 ConfiguringanSNMPTrapSourceUsingtheGUI 44 MonitoringtheSystemUsingSNMP 45 ConfiguringSPANforTrafficMonitoring 45 AboutSPAN 45 SPANGuidelinesandRestrictions 46 ConfiguringaSPANSession 48 ConfiguringanLayer3EPGSPANSessionforExternalAccessUsingtheAPICGUI 48 ConfiguringSPANUsingtheNX-OSStyleCLI 49 ConfiguringLocalSPANinAccessMode 49 ConfiguringERSPANinAccessMode 51 ConfiguringERSPANinFabricMode 54 ConfiguringERSPANinTenantMode 56 UsingStatistics 58 ViewingStatisticsintheGUI 59 SwitchStatisticsCommands 60 ManagingStatisticsThresholdsUsingtheGUI 61 StatisticsTroubleshootingScenarios 61 StatisticsCleanup 63 SpecifyingSyslogSourcesandDestinations 63 AboutSyslog 63 CreatingaSyslogDestinationandDestinationGroup 64 CreatingaSyslogSource 65 EnablingSyslogtoDisplayinNX-OSCLIFormat,UsingtheRESTAPI 66 DiscoveringPathsandTestingConnectivitywithTraceroute 67 AboutTraceroute 68 TracerouteGuidelinesandRestrictions 68 CiscoAPICTroubleshootingGuide v Contents PerformingaTracerouteBetweenEndpoints 68 UsingtheTroubleshootingWizard 69 GettingStartedwiththeTroubleshootingWizard 69 GeneratingTroubleshootingReports 71 TopologyintheTroubleshootingWizard 72 UsingtheFaultsTroubleshootingScreen 73 UsingtheDrop/StatisticsTroubleshootingScreen 74 UsingtheContractsTroubleshootingScreen 76 UsingtheEventsTroubleshootingScreen 77 UsingtheTracerouteTroubleshootingScreen 77 UsingtheAtomicCounterTroubleshootingScreen 79 UsingtheSPANTroubleshootingScreen 79 L4-L7ServicesValidatedScenarios 79 ListofAPIsforEndpointtoEndpointConnections 80 interactiveAPI 81 createsessionAPI 82 modifysessionAPI 83 atomiccounterAPI 83 tracerouteAPI 84 spanAPI 84 generatereportAPI 85 schedulereportAPI 86 getreportstatusAPI 86 getreportslistAPI 86 getsessionslistAPI 87 getsessiondetailAPI 87 deletesessionAPI 87 clearreportsAPI 88 contractsAPI 88 ListofAPIsforEndpointtoLayer3ExternalConnections 89 interactiveAPI 89 createsessionAPI 89 modifysessionAPI 91 atomiccounterAPI 92 CiscoAPICTroubleshootingGuide vi Contents tracerouteAPI 92 spanAPI 93 generatereportAPI 94 schedulereportAPI 95 getreportstatusAPI 96 getreportslistAPI 96 getsessionslistAPI 96 getsessiondetailAPI 98 deletesessionAPI 99 clearreportsAPI 99 contractsAPI 99 ratelimitAPI 100 13extAPI 101 CHAPTER 7 ManuallyRemovingDisabledInterfacesandDecommissionedSwitchesfromtheGUI 103 ManuallyRemovingDisabledInterfacesandDecommissionedSwitchesfromtheGUI 103 CHAPTER 8 DecommissioningandRecommissioningSwitches 105 DecommissioningandRecommissioningSwitches 105 CHAPTER 9 TroubleshootingStepsforEndpointConnectivityProblems 107 TroubleshootingEndpointConnectivity 107 InspectingEndpointandTunnelInterfaceStatus 108 InspectingtheEndpointStatus 108 InspectingtheTunnelInterfaceStatus 109 ConnectinganSFPModule 109 CHAPTER 10 TroubleshootingEVPNType-2RouteAdvertisement 111 TroubleshootingEVPNType-2RouteDistributiontoaDCIG 111 CHAPTER 11 PerformingaRebuildoftheFabric 115 RebuildingtheFabric 115 CiscoAPICTroubleshootingGuide vii Contents CHAPTER 12 VerifyingIP-BasedEPGConfigurations 117 VerifyingIP-BasedEPGConfigurationsUsingtheGUI 117 VerifyingIP-EPGConfigurationsUsingSwitchCommands 118 CHAPTER 13 RecoveringaDisconnectedLeaf 121 RecoveringaDisconnectedLeafUsingtheRESTAPI 121 CHAPTER 14 TroubleshootingaLoopbackFailure 123 IdentifyingaFailedLineCard 123 CHAPTER 15 DeterminingWhyaPIMInterfaceWasNotCreated 125 APIMInterfaceWasNotCreatedForanL3OutInterface 125 APIMInterfaceWasNotCreatedForaMulticastTunnelInterface 126 APIMInterfaceWasNotCreatedForaMulticast-EnabledBridgeDomain 126 CHAPTER 16 ConfirmingthePortSecurityInstallation 127 ConfirmingYourPortSecurityInstallationUsingVisore 127 ConfirmingYourHardwarePortSecurityInstallationUsingtheCiscoNX-OSCLI 127 CHAPTER 17 TroubleshootingQoSPolicies 131 TroubleshootingCiscoAPICQoSPolicies 131 CHAPTER 18 DeterminingtheSupportedSSLCiphers 133 AboutSSLCiphers 133 DeterminingtheSupportedSSLCiphersUsingtheCLI 134 CHAPTER 19 RemovingUnwanted_ui_Objects 135 RemovingUnwanted_ui_ObjectsUsingtheRESTAPI 137 APPENDIX A acidiagCommand 139 APPENDIX B ConfiguringExportPoliciesforTroubleshooting 147 CiscoAPICTroubleshootingGuide viii Contents AboutExportingFiles 147 FileExportGuidelinesandRestrictions 147 ConfiguringaRemoteLocation 148 ConfiguringaRemoteLocationUsingtheGUI 148 ConfiguringaRemoteLocationUsingtheRESTAPI 148 ConfiguringaRemoteLocationUsingtheNX-OSStyleCLI 149 SendinganOn-DemandTechSupportFile 150 SendinganOn-DemandTechsupportFileUsingtheGUI 150 SendinganOn-DemandTechSupportFileUsingtheRESTAPI 150 APPENDIX C FindingtheSwitchInventory 153 FindingYourSwitchInventoryUsingtheGUI 153 FindingYourSwitchInventoryUsingtheNX-OSCLI 153 FindingYourSwitchInventoryUsingtheRESTAPI 156 APPENDIX D CiscoAPICClusterManagement 159 ExpandingtheCiscoAPICCluster 159 ContractingtheCiscoAPICCluster 159 ClusterManagementGuidelines 160 ExpandingtheAPICClusterSize 161 ReducingtheAPICClusterSize 161 ReplacingCiscoAPICControllersintheCluster 162 ExpandingtheClusterExamples 163 ExpandingtheAPICClusterUsingtheGUI 163 ExpandingtheAPICClusterUsingtheRESTAPI 164 ContractingtheClusterExamples 164 ContractingtheAPICClusterUsingtheGUI 164 ContractingtheAPICClusterUsingtheRESTAPI 165 CommissioningandDecommissioningCiscoAPICControllers 166 CommissioningaCiscoAPICControllerintheClusterUsingtheGUI 166 DecommissioningaCiscoAPICControllerintheClusterUsingtheGUI 166 ReplacingaCiscoAPICinaClusterUsingtheCLI 167 APPENDIX E CiscoAPICSSDReplacement 169 CiscoAPICTroubleshootingGuide ix Contents ReplacingtheSolid-StateDriveinCiscoAPIC 169 CiscoAPICTroubleshootingGuide x