Ananta: Cloud Scale Load Balancing Parveen Patel, Deepak Bansal, Lihua Yuan, Ashwin Murthy, Albert Greenberg, David A. Maltz, Randy Kern, Hemant Kumar, Marios Zikos, Hongyu Wu, Changhoon Kim, Naveen Karri Microsoft ABSTRACT Layer-4 load balancing is fundamental to creating scale-out web services.WedesignedandimplementedAnanta,ascale-outlayer-4 loadbalancerthatrunsoncommodityhardwareandmeetstheper- formance,reliabilityandoperationalrequirementsofmulti-tenant cloud computing environments. Ananta combines existing tech- niquesinroutinganddistributedsystemsinauniquewayandsplits thecomponentsofaloadbalancerintoaconsensus-basedreliable controlplaneandadecentralizedscale-outdataplane.Akeycom- ponentofAnantaisanagentineveryhostthatcantakeoverthe packet modification function from the load balancer, thereby en- ablingtheloadbalancertonaturallyscalewiththesizeofthedata center. Duetoitsdistributedarchitecture, Anantaprovidesdirect Figure1:AnantaDataPlaneTiers serverreturn(DSR)andnetworkaddresstranslation(NAT)capa- bilities across layer-2 boundaries. Multiple instances of Ananta havebeendeployedintheWindowsAzurepubliccloudwithcom- tobeatleastashighasapplications’SLA,butoftensignificantly bined bandwidth capacity exceeding 1Tbps. It is serving traffic highertoaccountforfailuresinotherinfrastructureservices. needsofadiversesetoftenants,includingtheblob,tableandrela- Asacloudprovider,wehaveseenthatcloudservicesputhuge tionalstorageservices. Withitsscale-outdataplanewecaneasily pressureontheloadbalancer’scontrolplaneanddataplane. In- achievemorethan100GbpsthroughputforasinglepublicIPad- boundflowscanbeintense—greaterthan100Gbpsforasingle dress. Inthispaper,wedescribetherequirementsofacloud-scale IPaddress—witheverypackethittingtheloadbalancer.Thepay- loadbalancer,thedesignofAnantaandlessonslearntfromitsim- as-you-gomodelandlargetenantdeploymentsputextremelyhigh plementationandoperationintheWindowsAzurepubliccloud. demands on real-time load balancer configuration — in a typical CategoriesandSubjectDescriptors:C.2.4[Computer-Communi- environmentof1000hostsweseesixconfigurationoperationsper cationNetworks]:DistributedSystems minuteonaverage,peakingatoneoperationpersecond.Inourex- GeneralTerms:Design,Performance,Reliability perience,thedataplaneandcontrolplanedemandsdroveourhard- wareloadbalancersolutionintoanuntenablecornerofthedesign Keywords: Software Defined Networking; Distributed Systems; space,withhighcost,withSLAviolationsandwithloadbalancing ServerLoadBalancing devicefailuresaccountingfor37%ofalllivesiteincidents. Thedesignproposedinthispaper,whichwecallAnanta(mean- 1. INTRODUCTION inginfiniteinSanskrit),resultedfromexaminingthebasicrequire- The rapid rise of cloud computing is driving demand for large ments, and taking an altogether different approach. Ananta is a scalemulti-tenantclouds. Amulti-tenantcloudenvironmenthosts scalablesoftwareloadbalancerandNATthatisoptimizedformulti- manydifferenttypesofapplicationsatalowcostwhileproviding tenant clouds. It achieves scale, reliability and any service any- highuptimeSLA—99.9%orhigher[3,4,19,10].Amulti-tenant where(§2)viaanoveldivisionofthedataplanefunctionalityinto loadbalancerserviceisafundamentalbuildingblockofsuchmulti- three separate tiers. As shown in Figure 1, at the topmost tier tenantcloudenvironments.Itisinvolvedinalmostallexternaland routersprovideloaddistributionatthenetworklayer(layer-3)based halfofintra-DCtraffic(§2)andhenceitsuptimerequirementsneed ontheEqualCostMultiPath(ECMP)routingprotocol[25].Atthe secondtier,ascalablesetofdedicatedserversforloadbalancing, calledmultiplexers(Mux),maintainconnectionflowstateinmem- Permissiontomakedigitalorhardcopiesofallorpartofthisworkfor oryanddolayer-4loaddistributiontoapplicationservers. Athird personalorclassroomuseisgrantedwithoutfeeprovidedthatcopiesarenot madeordistributedforprofitorcommercialadvantageandthatcopiesbear tierpresentinthevirtualswitchoneveryserverprovidesstateful thisnoticeandthefullcitationonthefirstpage.Copyrightsforcomponents NATfunctionality. Usingthisdesign, nooutboundtrafffichasto ofthisworkownedbyothersthanACMmustbehonored.Abstractingwith pass through the Mux, thereby significantly reducing packet pro- creditispermitted.Tocopyotherwise,orrepublish,topostonserversorto cessing requirement. Another key element of this design is the redistributetolists,requirespriorspecificpermissionand/orafee.Request ability to offload multiplexer functionality down to the host. As [email protected]. discussedin§2, thisdesignenablesgreaterthan80%oftheload SIGCOMM’13,August12–16,2013,HongKong,China. Copyright2013ACM978-1-4503-2056-6/13/08...$15.00. balancedtraffictobypasstheloadbalancerandgodirect,thereby 207 eliminatingthroughputbottleneckandreducinglatency. Thisdivi- sionofdataplanescalesnaturallywiththesizeofthenetworkand introducesminimalbottlenecksalongthepath. Ananta’sapproachisanexampleofSoftwareDefinedNetwork- ing (SDN) as it uses the same architectural principle of manag- ingaflexibledataplaneviaacentralizedcontrolplane. Thecon- trollermaintainshighavailabilityviastatereplicationbasedonthe Paxos[14]distributedconsensusprotocol. Thecontrolleralsoim- plementsreal-timeportallocationforoutboundNAT,alsoknown asSourceNAT(SNAT). AnantahasbeenimplementedasaserviceintheWindowsAzure cloudplatform.WeconsideredimplementingAnantafunctionality inhardware.However,withthisinitialAnantaversioninsoftware, wewereabletorapidlyexplorevariousoptionsinproductionand determine what functions should be built into hardware, e.g., we Figure2: FlatDataCenterNetworkoftheCloud. Allnetwork realizedthatkeepingper-connectionstateisnecessarytomaintain devicesrunasLayer-3devicescausingalltrafficexternaltoarack application uptime due to the dynamic nature of the cloud. Sim- to be routed. All inter-service traffic — intra-DC, inter-DC and ilarly, weightedrandomloadbalancingpolicy, whichreducesthe Internet—goesviatheloadbalancer. need for per-flow state synchronization among load balancer in- stances,issufficientfortypicalcloudworkloads. Weconsiderthe evaluationofthesemechanisms,regardlessofhowtheyareimple- mented,tobeakeycontributionofthiswork. Morethan100instancesofAnantahavebeendeployedinWin- dows Azure since September 2011 with a combined capacity of 1Tbps. Ithasbeenserving100,000VIPswithvaryingworkloads. IthasprovenveryeffectiveagainstDoSattacksandminimizeddis- ruptionduetoabusivetenants. Comparedtotheprevioussolution, Ananta costs one order of magnitude less; and provides a more scalable,flexible,reliableandsecuresolutionoverall. There has been significant interest in moving middlebox func- tionalitytosoftwarerunningongeneral-purposehardwareinboth research[23,24,5]andindustry[8,2,27,21,31]. Mostofthese architecturesproposeusingeitherDNSorOpenFlow-enabledhard- Figure 3: Internet and inter-service traffic as a percentage of wareswitchesforscaling. TothebestofourknowledgeAnantais totaltrafficineightdatacenters. thefirstmiddleboxarchitecturethatrefactorsthemiddleboxfunc- tionalityandmovespartsofittothehosttherebyenablinguseof network routing technologies — ECMP and BGP — for natural isacollectionofvirtualornativemachinesthatismanagedasone scalingwiththesizeofthenetwork.Themaincontributionsofthis entity. Weusethetermstenantandserviceinterchangeablyinthis papertotheresearchcommunityare: paper.EachmachineisassignedaprivateDirectIP(DIP)address. Typically,aserviceisassignedonepublicVirtualIP(VIP)address • Identifying the requirements and design space for a cloud- andalltrafficcrossingtheserviceboundary,e.g.,totheInternetor scalesolutionforlayer-4loadbalancing. toback-endserviceswithinthesamedatacentersuchasstorage, • Providingdesign,implementationandevaluationofAnanta usestheVIPaddress.Aserviceexposeszeroormoreexternalend- thatcombinestechniquesinnetworkinganddistributedsys- pointsthateachreceiveinboundtrafficonaspecificprotocoland temstorefactorloadbalancerfunctionalityinanovelwayto port on the VIP. Traffic directed to an external endpoint is load- meetscale,performanceandreliabilityrequirements. balanced to one or more machines of the service. All outbound trafficoriginatingfromaserviceisSourceNAT’ed(SNAT)using • Providing measurements and insights from running Ananta the same VIP address as well. Using the same VIP for all inter- inalargeoperationalCloud. service traffic has two important benefits. First, it enables easy upgradeanddisasterrecoveryofservicessincetheVIPcanbedy- 2. BACKGROUND namically mapped to another instance of the service. Second, it Inthissection,wefirstconsiderourdatacenternetworkarchi- makesACLmanagementeasiersincetheACLscanbeexpressed tectureandthenatureoftrafficthatisservicedbytheloadbalancer. intermsofVIPsandhencedonotchangeasservicesscaleupor Wethenderiveasetofrequirementsfortheloadbalancer. downorgetredeployed. 2.1 DataCenterNetwork 2.2 NatureofVIPTraffic Figure2showsthenetworkofatypicaldatacenterinourcloud. Publiccloudservices[3,4,19]hostadiversesetofworkloads, Amediumsizeddatacenterhosts40,000servers, eachwithone suchasstorage,webanddataanalysis.Inaddition,thereareanin- 10Gbps NIC. This two-level Clos network architecture [11] typi- creasingnumberofthird-partyservicesavailableinthecloud.This callyhasanover-subscriptionratioof1:4atthespinelayer. The trendleadsustobelievethattheamountofinter-servicetrafficin border routers provide a combined capacity of 400Gbps for con- thecloudwillcontinuetoincrease. Weexaminedthetotaltraffic nectivitytootherdatacentersandtheInternet. Acloudcontroller in eight data centers for a period of one week and computed the managesresourcesinthedatacenterandhostsservices. Aservice ratio of Internet traffic and inter-service traffic to the total traffic. 208 The result is shown in Figure 3. On average, about 44% (with a minimumof18%andmaximumof59%)ofthetotaltrafficisVIP traffic—iteitherneedsloadbalancingorSNATorboth. Outof this,about14%trafficonaverageistotheInternetandtheremain- ing30%isintra-DC.Theratioofintra-DCVIPtraffictoInternet VIPtrafficis2 : 1. Overall,wefindthat70%oftotalVIPtraffic isinter-servicewithinthesamedatacenter. Wefurtherfoundthat on average the ratio of inbound traffic to outbound traffic across ourdatacentersis1 : 1. Majorityofthistrafficisread-writetraf- ficandcross-datacenterreplicationtraffictoourstorageservices. Insummary,greaterthan80%ofVIPtrafficiseitheroutboundor containedwithinthedatacenter.Asweshowinthispaper,Ananta offloadsallofthistraffictothehost,therebyhandlingonly20%of thetotalVIPtraffic. 2.3 Requirements Figure4:Componentsofatraditionalloadbalancer.Typically, Scale,ScaleandScale: Themoststringentscalerequirements theloadbalancerisdeployedinanactive-standbyconfiguration. canbederivedbyassumingthatalltrafficinthenetworkiseither Theroutemanagementcomponentensuresthatthecurrentlyactive loadbalancedorNAT’ed. Fora40,000servernetwork,builtus- instancehandlesallthetraffic. ingthearchitectureshowninFigure2,400Gbpsofexternaltraffic and100Tbpsofintra-DCtrafficwillneedloadbalancingorNAT. itisimportantthattheloadbalancerprovideseachserviceitsfair Basedonthetrafficratiospresentedin§2.2,at100%networkuti- shareofresources. lization, 44Tbps traffic will be VIP traffic. A truly scalable load balancerarchitecturewouldsupportthisrequirementwhilemain- 3. DESIGN taininglowcost.Whilecostisasubjectivemetric,inourcloud,less than1%ofthetotalservercostwouldbeconsideredlowcost,so 3.1 DesignPrinciples anysolutionthatwouldcostmorethan400general-purposeservers istooexpensive.AtthecurrenttypicalpriceofUS$2500perserver, Scale-outIn-networkProcessing:Figure4illustratesthemain thetotalcostshouldbelessthanUS$1,000,000. Traditionalhard- componentsofatraditionalloadbalancer. Foreachnewflow,the ware load balancers do notmeet this requirement as their typical load balancer selects a destination address (or source for SNAT) list price is US$80,000 for 20Gbps capacity without considering dependingonthecurrentlyactiveflowsandremembersthatdeci- bulkdiscounts,supportcostsorredundancyrequirements. sioninaflowtable. Subsequentpacketsforthatflowusethestate Therearetwomoredimensionstothescalerequirement. First, createdbythefirstpacket. TraditionalNATandloadbalancingal- thebandwidthandnumberofconnectionsservedbyasingleVIP gorithms(e.g.,round-robin)requireknowledgeofallactiveflows, arehighlyvariableandmaygoupto100Gbpsand1millionsimul- hencealltrafficforaVIPmustpassthroughthesameloadbalancer. taneousconnections. Second,therateofchangeinVIPconfigura- Thisforcestheloadbalancerintoascale-upmodel. Ascale-upor tiontendstobeverylargeandbursty,onaverage12000configura- verticalscalingmodelisonewherehandlingmorebandwidthfor tionsperdayfora1000nodecluster,withburstsof100sofchanges aVIPrequiresahighercapacitybox. perminuteascustomerservicesgetdeployed,deletedormigrated. Networkrouters,ontheotherhand,followascale-outmodel.A Reliability: The load balancer is a critical component to meet scale-outorhorizontalscalingmodelisonewheremorebandwidth theuptimeSLAofapplications.Servicesrelyontheloadbalancer can be handled by simply adding more devices of similar capac- tomonitorhealthoftheirinstancesandmaintainavailabilityduring ity. Routers scale out because they do not maintain any per-flow plannedandunplanneddowntime. Overmanyyearsofoperation state that needs synchronization across routers and therefore one ofourcloudwefoundthattraditional1+1redundancysolutions canaddorremoveadditionalrouterseasily.Anantadesignreduces deployedasactive/standbyhardwareloadbalancersareunableto thein-networkfunctionalityneededforloadbalancingtobesuch meetthesehighavailabilityneeds.Theloadbalancermustsupport thatmultiplenetworkelementscansimultaneouslyprocesspackets N +1redundancymodelwithauto-recovery,andtheloadbalanc- forthesameVIPwithoutrequiringper-flowstatesynchronization. ingservicemustdegradegracefullyinthefaceoffailures. Thisdesignchoiceisenabledbecausewecanmakecertainas- Any Service Anywhere: In our cloud, applications are gener- sumptionsaboutourenvironment. Oneofthekeyassumptionsis allyspreadovermultiplelayer-2domainsandsometimesevenspan that load balancing policies that require global knowledge, e.g., multiple data centers. The load balancer should be able to reach weightedroundrobin(WRR),arenotrequiredforlayer-4loadbal- DIPslocatedanywhereonthenetwork. Traditionalloadbalancers ancing. Instead,randomlydistributingconnectionsacrossservers providesomeoftheirfunctionality,e.g.,NAT,toDIPsonlywithin basedontheirweightsisareasonablesubstituteforWRR.Infact, a layer-2 domain. This fragments the load balancer capacity and weightedrandomistheonlyloadbalancingpolicyusedbyourload makesthemunusableinlayer-3basedarchitectures. balancerinproduction. Theweightsarederivedbasedonthesize Tenant Isolation: A multi-tenant load balancer is shared by oftheVMorothercapacitymetrics. thousands of services. It is critical that DoS attacks on one ser- Offload to End Systems: Hypervisors in end systems can al- vicedonotaffecttheavailabilityofotherservices. Similarly, an ready do highly scalable network processing, e.g., ACL enforce- abusiveserviceshouldnotbeabletoaffecttheavailabilityofNAT ment,ratelimitingandmetering. Anantaleveragesthisdistributed for other services by creating large number of outbound connec- scalable platform and offloads significant data plane and control tions. Inourcloudenvironment,weoftenseeserviceswithlarge planefunctionalitydowntothehypervisorinendsystems.Thehy- numberofoutboundNATconnectionsduetobugsandpoorappli- pervisorneedstohandlestateonlyfortheVMshostedonit. This cationdesign. Furthermore,whentheloadbalancerisunderload, designchoiceisanotherkeydifferentiatorfromexistingloadbal- 209 Figure6:JSONrepresentationofasimpleVIPConfiguration. Figure 5: The Ananta Architecture. Ananta consists of three components — Ananta Manager, Ananta Mux and Host Agent. Eachcomponentisindependentlyscalable. Managercoordinates stateacrossAgentsandMuxes. Muxisresponsibleforpacketfor- wardingforinboundpackets.AgentimplementsNAT,whichallows Figure7:LoadBalancingforInboundConnections. alloutboundtraffictobypassMux.Agentsareco-locatedwithdes- tinationservers. usingIP-in-IPprotocol[18]settingtheselectedDIPasthedestina- tionaddressintheouterheader(step2). Itthensendsitoutusing ancers.Whileononehanditenablesnaturalscalingwiththesizeof regularIProutingattheMux(step3).TheMuxandtheDIPdonot thedatacenter;ontheotherhand,itpresentssignificantchallenges needtobeonthesameVLAN,theyjustneedtohaveIP(layer-3) inmanagingdistributedstateacrossallhostsandmaintainingavail- connectivity between them. The HA, located on the same physi- abilityduringfailuresofcentralizedcomponents. calmachineasthetargetDIP,interceptsthisencapsulatedpacket, 3.2 Architecture removestheouterheader,andrewritesthedestinationaddressand port(step4)andremembersthisNATstate.TheHAthensendsthe Anantaisalooselycoupleddistributedsystemcomprisingthree rewrittenpackettotheVM(step5). maincomponents(seeFigure5)—AnantaManager(AM),Multi- WhentheVMsendsareplypacketforthisconnection,itisin- plexer(Mux)andHostAgent(HA).Tobetterunderstandthedetails terceptedbytheHA(step6).TheHAdoesareverseNATbasedon ofthesecomponents,wefirstdiscusstheloadbalancerconfigura- thestatefromstep4andrewritesthesourceaddressandport(step tion and the overall packet flow. All packet flows are described 7). Itthensendsthepacketouttotheroutertowardsthesourceof usingTCPconnectionsbutthesamelogicisappliedforUDPand thisconnection. ThereturnpacketdoesnotgothroughtheMuxat otherprotocolsusingthenotionofpseudoconnections. all,therebysavingpacketprocessingresourcesandnetworkdelay. Thistechniqueofbypassingtheloadbalanceronthereturnpathis 3.2.1 VIPConfiguration knownasDirectServerReturn(DSR).Notallpacketsofaconnec- The load balancer receives a VIP Configuration for every VIP tionwouldendupatthesameMux,howeverallpacketsforasingle thatitisdoingloadbalancingandNATfor. AsimplifiedVIPcon- connectionmustbedeliveredtothesameDIP.Muxesachievethis figuration is shown in Figure 6. An Endpoint refers to a specific viaacombinationofconsistenthashingandstatemanagementas transportprotocolandportontheVIPthatisloadbalancedtoaset explainedlaterinthissection. ofDIPs. PacketsdestinedtoanEndpointareNAT’edtotheDIP addressandport. SNATspecifiesalistofIPaddressesforwhich 3.2.3 OutboundConnections outboundconnectionsneedtobeSourceNAT’edwiththeVIPand A unique feature of Ananta is a distributed NAT for outbound anephemeralport. connections.EvenforoutboundconnectionsthatneedsourceNAT (SNAT), Ananta ensures that outgoing packets do not need to go 3.2.2 InboundConnections throughMux. Figure8showshowpacketsforanoutboundSNAT Figure 7 shows how packets destined for a VIP are load bal- anced and delivered to the DIP of a VM. When a VIP is con- figured on Ananta, each Mux advertises a route to its first-hop routerannouncingitselfasthenexthopforthatVIP1.Thiscauses the routers to distribute packets destined for the VIP across all the Mux nodes based on Equal Cost MultiPath Routing Protocol (ECMP)[25](step1). Uponreceivingapacket,theMuxchooses aDIPfortheconnectionbasedonitsloadbalancingalgorithm,de- scribedlaterinthissection.Itthenencapsulatesthereceivedpacket 1Inreality,routesareadvertisedforVIPsubnetsduetosmallrout- ingtablesincommodityroutersbutthesamelogicapplies. Figure8:HandlingOutboundSNATConnections. 210 connectionarehandled. AVMsendsapacketcontainingitsDIP asthesourceaddress,port astheportandanexternaladdressas d thedestinationaddress(step1).TheHAinterceptsthispacketand recognizes that this packet needs SNAT. It then holds the packet in a queue and sends a message to AM requesting an externally routableVIPandaportforthisconnection(step2). AMallocates a(VIP,port )fromapoolofavailableportsandconfigureseach s Muxwiththisallocation(step3).AMthensendsthisallocationto theHA(step4). TheHAusesthisallocationtorewritethepacket sothatitssourceaddressandportarenow(VIP,port ).TheHA s sendsthisrewrittenpacketdirectlytotherouter. Thereturnpack- ets from the external destination are handled similar to inbound connections. Thereturnpacketissentbytheroutertooneofthe Muxnodes(step6). TheMuxalreadyknowsthat DIP2should receivethispacket(basedonthemappinginstep3), soitencap- sulates the packet with DIP2 as the destination and sends it out (step7). TheHAinterceptsthereturnpacket, performsareverse Figure9: FastpathControlandDataPackets. Routersarenot translationsothatthepacket’sdestinationaddressandportarenow shown for brevity. Starting with step 8, packets flow directly (DIP,port ).TheHAsendsthispackettotheVM(step8). d betweensourceanddestinationhosts. 3.2.4 Fastpath jacktraffic. HApreventsthisbyvalidatingthatthesourceaddress Inordertoscaletothe100softerabitbandwidthrequirementof of redirect message belongs to one of the Ananta services in the intra-DCtraffic,Anantaoffloadsmostoftheintra-DCtraffictoend data center. This works in our environment since the hypervisor systems. This is done by a technique we call Fastpath. The key preventsIPspoofing. IfIPspoofingcannotbeprevented, amore ideaisthattheloadbalancermakesitsdecisionaboutwhichDIP dynamicsecurityprotocolsuchasIPSECcanbeemployed. a new connection should go to when the first packet of that con- nectionarrives.Oncethisdecisionismadeforaconnectionitdoes 3.3 Mux notchange. Therefore,thisinformationcanbesenttotheHAson thesourceanddestinationmachinessothattheycancommunicate TheMultiplexer(Mux)handlesallincomingtraffic.Itisrespon- directly. Thisresultsinthepacketsbeingdelivereddirectlytothe sibleforreceivingtrafficforalltheconfiguredVIPsfromtherouter DIP,bypassingMuxinbothdirections,therebyenablingcommu- andforwardingittoappropriateDIPs.EachinstanceofAnantahas nicationatfullcapacitysupportedbytheunderlyingnetwork.This oneormoresetsofMuxescalledMuxPool. AllMuxesinaMux changeistransparenttoboththesourceanddestinationVMs. Poolhaveuniformmachinecapabilitiesandidenticalconfiguration, ToillustratehowFastpathworks,considertwoservices1and2 i.e.,theyhandlethesamesetofVIPs.HavinganotionofMuxPool thathavebeenassignedvirtualaddressesVIP1andVIP2respec- allowsustoscalethenumberofMuxes(dataplane)independent tively.ThesetwoservicescommunicatewitheachotherviaVIP1 ofthenumberofAMreplicas(controlplane). andVIP2usingthealgorithmsforloadbalancingandSNATde- 3.3.1 RouteManagement scribedabove. Figure9showsasimplifiedversionofpacketflow foraconnectioninitiatedbyaVMDIP1(belongingtoservice1) EachMuxisaBGPspeaker[20]. WhenaVIPisconfiguredon toVIP2. ThesourcehostofDIP1SNATstheTCPSYNpacket Ananta,eachMuxstartsadvertisingarouteforthatVIPtoitsfirst- using VIP1 and sends it to VIP2 (step 1). This packet is de- hoprouterwithitselfasthenexthop.AllMuxesinaMuxPoolare liveredtoaMux2,whichforwardsthepackettowardsdestination equalnumberofLayer-3networkhopsawayfromtheentrypoint DIP2(step2). WhenDIP2repliestothispacket,itisSNAT’ed ofthedatacenter.Thisensuresthattheroutersdistributetrafficfor bythedestinationhostusingVIP2andsenttoMux1(step3).This agivenVIPequallyacrossallMuxesviatheEqualCostMultiPath MuxusesitsSNATstateandsendsthispackettoDIP1(step4). (ECMP)routingprotocol[25]. RunningtheBGPprotocolonthe Subsequentpacketsforthisconnectionfollowthesamepath. Muxprovidesautomaticfailuredetectionandrecovery. IfaMux For Fastpath, Ananta configures Mux with a set of source and failsorshutsdownunexpectedly,therouterdetectsthisfailurevia destination subnets that are capable of Fastpath. Once a connec- the BGP protocol and automatically stops sending traffic to that tionhasbeenfullyestablished(e.g.,TCPthree-wayhandshakehas Mux. Similarly,whentheMuxcomesupandithasreceivedstate completed)betweenVIP1andVIP2,Mux2sendsaredirectmes- fromAM,itcanstartannouncingtheroutesandtherouterwillstart sagetoVIP1,informingitthattheconnectionismappedtoDIP2 forwardingtraffictoit.MuxesusetheTCPMD5[13]protocolfor (step5).ThisredirectpacketgoestoaMuxhandlingVIP1,which authenticatingtheirBGPsessions. looksupitstabletoknowthatthisportisusedbyDIP1. Mux1 3.3.2 PacketHandling thensendsaredirectmessagetowardsDIP1andDIP2(steps6 and7respectively). HAonthesourcehostinterceptsthisredirect TheMuxmaintainsamappingtable,calledVIPmap,thatdeter- packetandremembersthatthisconnectionshouldbesentdirectly mineshowincomingpacketsarehandled. Eachentryinthemap- toDIP2.SimilarlyHAonthedestinationhostinterceptstheredi- pingtablemapsaVIPendpoint,i.e.,three-tuple(VIP,IPprotocol, rectmessageandremembersthatthisconnectionshouldbesentto port),toalistofDIPs.ThemappingtableiscomputedbyAMand DIP1.Oncethisexchangeiscomplete,anyfuturepacketsforthis senttoalltheMuxesinaMuxPool. WhenMuxreceivesapacket connectionareexchangeddirectlybetweenthesourceanddestina- fromtherouter,itcomputesahashusingthefive-tuplefrompacket tionhosts(step8). headerfields—(sourceIP,destinationIP,IPProtocol,sourceport, ThereisonesecurityconcernassociatedwithFastpath–arogue destinationport). ItthenusesthishashtolookupaDIPfromthe hostcouldsendaredirectmessageimpersonatingtheMuxandhi- listofDIPsintheassociatedmap. Finally,itencapsulates[18]the 211 packetwithanouterIPheader—withitselfasthesourceIPand theHostAgent,itdecapsulatesthemandthenperformsaNATas theDIPasthedestinationIP—andforwardsthispackettotheDIP. pertheNATrulesconfiguredbyAnantaManager. TheNATrules The encapsulation at the Mux preserves the original IP header describe the rewrite rules of type: (VIP,protocol ,port ) ⇒ v v andIPpayloadofthepacket,whichisessentialforachievingDi- (DIP,protocol ,port ). In this case, the Host Agent identifies v d rectServerReturn(DSR).AllMuxesinaMuxPoolusetheexact packetsthataredestinedto(VIP,protocol ,port ),rewritesthe v v samehashfunctionandseedvalue.SinceallMuxeshavethesame destinationaddressandportto(DIP,port )andcreatesbi-directional d mappingtable,itdoesn’tmatterwhichMuxagivennewconnec- flow state that is used by subsequent packets of this connection. tiongoesto,itwillbedirectedtothesameDIP. When a return packet for this connection is received, it does re- verseNATbasedonflowstateandsendsthepackettothesource 3.3.3 FlowStateManagement directlythroughtherouter,bypassingtheMuxes. Muxsupportstwotypesofmappingentries–statefulandstate- 3.4.2 SourceNATforOutboundConnections less. Statefulentriesareusedforloadbalancingandstatelessen- triesareusedforSNAT.Forstatefulmappingentries,onceaMux For outbound connections, the Host Agent does the following. haschosenaDIPforaconnection, itremembersthatdecisionin It holds the first packet of a flow in a queue and sends a mes- a flow table. Every non-SYN TCP packet, and every packet for sage to Ananta Manager requesting a VIP and port for this con- connection-lessprotocols, ismatchedagainstthisflowtablefirst, nection. Ananta Manager responds with a VIP and port and the and if a match is found it is forwarded to the DIP from the flow HostAgentNATsallpendingconnectionstodifferentdestinations table. ThisensuresthatonceaconnectionisdirectedtoaDIP,it usingthisVIPandport.Anynewconnectionstodifferentdestina- continuestogotothatDIPdespitechangesinthelistofDIPsinthe tions(remoteaddress,remoteport)canalsoreusethesameport mappingentry. Ifthereisnomatchintheflowtable,thepacketis as the TCP five-tuple will still be unique. We call this technique treatedasafirstpacketoftheconnection. portreuse. AMmayreturnmultipleportsinresponsetoasingle GiventhatMuxesmaintainper-flowstate,itmakesthemvulner- request. TheHAusestheseportsforanysubsequentconnections. abletostateexhaustionattackssuchastheSYN-floodattack. To Anyunusedportsarereturnedbackafteraconfigurableidletime- counterthistypeofabuse,Muxclassifiesflowsintotrustedflows out.IftheHAkeepsgettingnewconnections,theseportsarenever anduntrustedflows. AtrustedflowisoneforwhichtheMuxhas returnedbacktoAM,however,AMmayforceHAtoreleasethem seenmorethanonepacket. Theseflowshavealongeridletime- atanytime. Basedonourproductionworkload, wehavemadea out.UntrustedflowsaretheonesforwhichtheMuxhasseenonly numberofoptimizationstominimizethenumberofSNATrequests onepacket.Theseflowshaveamuchshorteridletimeout.Trusted aHostAgentneedstosend,includingpreallocationofports.These anduntrustedflowsaremaintainedintwoseparatequeuesandthey andotheroptimizationsarediscussedlaterinthispaper. havedifferentmemoryquotasaswell. OnceaMuxhasexhausted itsmemoryquota,itstopscreatingnewflowstatesandfallsbackto 3.4.3 DIPHealthMonitoring lookupinthemappingentry. ThisallowsevenanoverloadedMux AnantaisresponsibleformonitoringthehealthofDIPsthatare tomaintainVIPavailabilitywithaslightlydegradedservice. behindeachVIPendpointandtakeunhealthyDIPsoutofrotation. DIPhealthcheckrulesarespecifiedaspartofVIPConfiguration. 3.3.4 HandlingMuxPoolChanges Onfirstlook,itwouldseemnaturaltorunhealthmonitoringonthe WhenaMuxinaMuxPoolgoesdown,routerstakeitoutofro- Muxnodessothathealthmonitoringtrafficwould taketheexact tationonceBGPholdtimerexpires(wetypicallysetholdtimerto samenetworkpathastheactualdatatraffic.However,itwouldput 30seconds).WhenanychangetothenumberofMuxestakesplace, additionalloadoneachMux,couldresultinadifferenthealthstate ongoingconnectionswillgetredistributedamongthecurrentlylive on each mux and would incur additional monitoring load on the Muxes based on the router’s ECMP implementation. When this DIPsasthenumberofMuxescanbelarge.Guidedbyourprinciple happens,connectionsthatreliedontheflowstateonanotherMux ofoffloadingtoendsystems,wechosetoimplementhealthmon- maynowgetmisdirectedtoawrongDIPiftherehasbeenachange itoringontheHostAgents. AHostAgentmonitorsthehealthof in the mapping entry since the connection started. We have de- localVMsandcommunicatesanychangesinhealthtoAM,which signedamechanismtodealwiththisbyreplicatingflowstateon thenrelaysthesemessagestoallMuxesintheMuxPool. Perhaps twoMuxesusingaDHT.Thedescriptionofthatdesignisoutside surprisingtosomereaders,runninghealthmonitoringonthehost the scope of this paper as we have chosen to not implement this makesiteasytoprotectmonitoringendpointsagainstunwarranted mechanismyetinfavorofreducedcomplexityandmaintaininglow traffic–anagentintheguestVMlearnsthehostVM’sIPaddress latency.Inaddition,wehavefoundthatclientseasilydealwithoc- viaDHCPandconfiguresafirewallruletoallowmonitoringtraffic casional connectivity disruptions by retrying connections, which only from the host. Since a VM’s host address does not change happenforvariousotherreasonsaswell. (wedon’tdoliveVMmigration),migrationofaVIPfromonein- stanceofAnantatoanotherorscalingthenumberofMuxesdoes 3.4 HostAgent notrequirereconfigurationinsideguestVMs.Webelievethatthese AdifferentiatingcomponentoftheAnantaarchitectureisanagent, benefitsjustifythisdesignchoice.Furthermore,inafullymanaged calledHostAgent,whichispresentonthehostpartitionofevery cloudenvironmentsuchasours,out-of-bandmonitoringcandetect physicalmachinethatisservedbyAnanta. TheHostAgentisthe network partitions where HA considers a VM healthy but some keytoachievingDSRandSNATacrosslayer-2domains. Further- MuxesareunabletocommunicatewithitsDIPs,raiseanalertand more, the Host Agent enables data plane scale by implementing eventakecorrectiveactions. Fastpath and NAT; and control plane scale by implementing VM 3.5 AnantaManager healthmonitoring. TheAnantaManager(AM)implementsthecontrolplaneofAnanta. 3.4.1 NATforInboundConnections ItexposesanAPItoconfigureVIPsforloadbalancingandSNAT. Forloadbalancedconnections,theHostAgentperformsstateful BasedontheVIPConfiguration,itconfigurestheHostAgentsand layer-4NATforallconnections. Asencapsulatedpacketsarriveat Mux Pools and monitors for any changes in DIP health. Ananta 212 ManagerisalsoresponsibleforkeepingtrackofhealthofMuxes thenumberofVMs.Furthermore,therearelimitsonthenumberof andHostsandtakingappropriateactions.AMachieveshighavail- portsallocatedandrateofallocationsallowedforanygivenVM. abilityusingthePaxos[14]distributedconsensusprotocol. Each instanceofAnantarunsfivereplicasthatareplacedtoavoidcor- 3.6.2 PacketRateFairness related failures. Three replicas need to be available at any given MuxtriestoensurefairnessamongVIPsbyallocatingavailable time to make forward progress. The AM uses Paxos to elect a bandwidthamongallactiveflows. Ifaflowattemptstostealmore primary,whichisresponsibleforperformingallconfigurationand thanitsfairshareofbandwidth,Muxstartstodropitspacketswith statemanagementtasks. WenowlookatsomekeyAMfunctions aprobabilitydirectlyproportionaltotheexcessbandwidthitisus- indetail. ing. WhilebandwidthfairnessworksforTCPflowsthataresend- inglarge-sizedpackets, itdoesnotworkforshortpacketsspread 3.5.1 SNATPortManagement acrossflows, orflowsthatarenon-TCP(e.g., UDP)orflowsthat aremalicious(e.g.,aDDoSattack). Akeycharacteristicofthese AMalsodoesportallocationforSNAT(§3.2.3). WhenanHA flows is that they do not back off in response to packet drops, in makesanewportrequestonbehalfofaDIP,AMallocatesafree factwesometimesseetheexactoppositereaction. Sincedropping portfortheVIP,replicatestheallocationtootherAMreplicas,cre- packetsattheMuxisnotgoingtohelpandincreasesthechances atesastatelessVIPmapentrymappingtheporttotherequesting ofoverload,ourprimaryapproachhasbeentobuildarobustdetec- DIP,configurestheentryontheMuxPoolandthensendstheallo- tionmechanismforoverloadduetopacketrate. EachMuxkeeps cationtotheHA.TherearetwomainchallengesinservingSNAT trackofitstop-talkers–VIPswiththehighestrateofpackets.Mux requests — latency and availability. Since SNAT request is done continuouslymonitorsitsownnetworkinterfacesandonceitde- onthefirstpacketofaconnection,adelayinservingSNATrequest tectsthatthereispacketdropduetooverload,itinformsAMabout directlytranslatesintolatencyseenbyapplications.Similarly,AM the overload and the top talkers. AM then identifies the topmost downtimewouldresultinfailureofoutboundconnectionsforappli- top-talkerasthevictimofoverloadandwithdrawsthatVIPfrom cationsresultingincompleteoutageforsomeapplications.Ananta allMuxes,therebycreatingablackholefortheVIP.Thisensures employs a number of techniques to reduce latency and increase thatthereisminimalcollateraldamageduetoMuxoverload. De- availability. First, it allocates a contiguous port range instead of pending on the policy for the VIP, we then route it through DoS allocatingoneportatatime. Byusingfixedsizedportranges,we protectionservices(thedetailsareoutsidethescopeofthispaper) optimizestorageandmemoryrequirementsonboththeAMandthe and enable it back on Ananta. We evaluate the effectiveness of Mux.OntheMuxdriver,onlythestartportofarangeisconfigured overloaddetectionandroutewithdrawalin§5. andbymakingtheportrangesizeapowerof2,wecanefficiently maparangeofportstoaspecificDIP.Second,itpreallocatesaset 3.7 DesignAlternatives of port rangesto DIPs when it firstreceives a VIP configuration. Third,ittriestododemandpredictionandallocatesmultipleport 3.7.1 DNS-basedScaleOut rangesinasinglerequest. Weevaluatetheeffectivenessofthese techniquesin§5. AnantausesBGPtoachievescaleoutamongmultipleactivein- stancesofMux. Atraditionalapproachtoscalingoutmiddlebox 3.6 TenantIsolation functionalityisviaDNS.Eachinstanceofthemiddleboxdevice, e.g., load balancer, is assigned a public IP address. The author- Ananta is a multi-tenant load balancer and hence tenant isola- itative DNS server is then used to distribute load among IP ad- tionisanessentialrequirement. Thegoaloftenantisolationisto dresses of the instances using an algorithm like weighted round- ensurethattheQualityofService(QoS)receivedbyonetenantis robin. Whenaninstancegoesdown,theDNSserverstopsgiving independent of other tenants in the system. However, Ananta is outitsIPaddress. Thisapproachhasseverallimitations. First,it an oversubscribed resource and we do not guarantee a minimum ishardertogetgooddistributionofloadbecauseitishardtopre- bandwidthorlatencyQoS.Assuch,weinterpretthisrequirement dicthowmuchloadcanbegeneratedviaasingleDNSresolution asfollows–thetotalCPU,memoryandbandwidthresourcesare request. Forexample,loadfromlargeclientssuchasamegaproxy dividedamongalltenantsbasedontheirweights. Theweightsare is always sent to a single server. Second, it takes longer to take directlyproportionaltothenumberofVMsallocatedtothetenant. unhealthy middlebox nodes out of rotation due to DNS caching Wemakeanothersimplifyingassumptionthattraffic forallVIPs –manylocalDNSresolversandclientsviolateDNSTTLs. And is distributed equally among all Muxes, therefore, each Mux can third,itcannotbeusedforscaleoutofstatefulmiddleboxes,such independentlyimplementtenantisolation.Wefindthisassumption asaNAT. toholdtrueinourenvironment.Ifthisassumptionweretonothold trueinthefuture,aglobalmonitorcoulddynamicallyinformeach 3.7.2 OpenFlow-basedLoadBalancing Muxtoassigndifferentweightstodifferenttenants. Memoryfair- An alternative to implementing Mux functionality in general- ness is implemented at Muxes and AM by keeping track of flow purposeserversistouseOpenFlow-capableswitches[16]. How- stateandenforcinglimits. ever, currently available OpenFlow devices have insufficient sup- portforgeneral-purposelayer-4loadbalancing. Forexample,ex- 3.6.1 SNATFairness istingOpenFlowswitchesonlysupport2000to4000flows,whereas AM is a critical resource as it handles all SNAT port requests. Mux needs to maintain state for millions of flows. Another key ExcessiverequestsfromonetenantshouldnotslowdowntheSNAT primitive lacking in existing OpenFlow hardware is tenant isola- responsetimeofanothertenant.AMensuresthisbyprocessingre- tion.Finally,inapureOpenFlow-basednetwork,wewillalsoneed questsinafirst-come-first-serve(FCFS)orderanditensuresthatat toreplaceBGPwithcentralizedroutinginAnantaManager,which anygiventimethereisatmostoneoutstandingrequestfromaDIP. hascertaindrawbacksasdiscussedin§6. Otherresearchershave If a new request isreceived while another request from the same alsoattemptedtobuildloadbalancersusingOpenFlowswitches[28], DIPispending, thenewrequestisdropped. Thissimplemecha- however,thesesolutionsdonotyetmeetalltherequirements,e.g., nismensuresthatVIPsgetSNATallocationtimeinproportionto tenantisolationandscalablesourceNATacrosslayer-2domains. 213 Figure 10: Staged event-driven (SEDA) Ananta Manager. Anantamanagersharesthesamethreadpoolacrossmultiplestages and supports priority event queues to maintain responsiveness of VIPconfigurationoperationsunderoverload. 4. IMPLEMENTATION Figure11:CPUusageatMuxandHostswithandwithoutFast- We implemented all three Ananta components from Figure 5. path. OnceFastpathisturnedon, thehoststakeovertheencap- Ananta Manager and Mux are deployed as a tenant of our cloud sulationfunctionfromMux.ThisresultsinlowerCPUatMuxand platformitself. TheHostAgentisdeployedoneveryhostinour CPUincreaseateveryhostdoingencapsulation. cloudandisupdatedwheneverweupdatetheAnantatenant. Ina typicaldeploymentfivereplicasofAMmanageasingleMuxPool. initsVIPMapwith1GBofmemory.Eachmuxcanmaintainstate MostMuxPoolshaveeightMuxesinthembutthenumbercanbe formillionsofconnectionsandisonlylimitedbyavailablemem- basedonload. oryontheserver.CPUperformancecharacteristicsoftheMuxare AnantaManager: AMperformsvarioustime-criticaltasks— discussedin§5.2.3. configurationofVIPs,allocationofports,coordinationofDIPhealth HostAgent: TheHostAgentalsohasadrivercomponentthat across Muxes. Therefore, its responsiveness is very critical. To runsasanextensionoftheWindowsHyper-Vhypervisor’svirtual achieveahighdegreeofconcurrency,weimplementedAMusing switch. Beinginthehypervisorenablesustosupportunmodified alock-freearchitecturethatissomewhatsimilartoSEDA[29].As VMsrunningdifferentoperatingsystems.Fortenantsrunningwith showninFigure10,AMisdividedintothefollowingstages—VIP native(non-VM)configuration,werunastripped-downversionof validation, VIP configuration, Route Management, SNAT Man- thevirtualswitchinsidethehostnetworkingstack.Thisenablesus agement, Host Agent Management and Mux Pool Management. toreusethesamecodebaseforbothnativeandVMscenarios. Ananta implementation makes two key enhancements to SEDA. Upgrading Ananta: Upgrading Ananta is a complex process First, inAnanta, multiplestagessharethesamethreadpool. This that takes place in three phases in order to maintain backwards- allowsustolimitthetotalnumberofthreadsusedbythesystem. compatibility between various components. First, we update in- Second, Ananta supports multiple priority queues for each stage. stancesoftheAnantaManager,oneatatime. Duringthisphase, Thisisusefulinmaintainingresponsivenessduringoverloadcon- AMalsoadaptsitspersistentstatefrompreviousschemaversion ditions.Forexample,SNATeventstakelesspriorityoverVIPcon- to the new version. Schema rollback is currently not supported. figurationevents. ThisallowsAnantatofinishVIPconfiguration Second,weupgradetheMuxes;andthird,theHostAgents. tasksevenwhenitisunderheavyloadduetoSNATrequests. AnantamaintainshighavailabilityusingPaxos[14].Wetookan 5. MEASUREMENTS existingimplementationofPaxosandaddeddiscoveryandhealth monitoringusingtheSDKofourcloudplatform.TheSDKnotifies Inthissectionwefirstpresentafewmicro-benchmarkstoevalu- AMofaddition,removalormigrationofanyreplicasofthePaxos ateeffectivenessandlimitationsofsomeofourdesignchoicesand cluster. Thisallowsustoautomaticallyreconfiguretheclusteras implementation,andthensomedatafromrealworlddeployments. instancesareaddedorremoveddynamically.Thesefeaturesofthe 5.1 Micro-benchmarks platformSDKresultinsignificantlyreducedoperationaloverhead forAnanta. Theplatformalsoprovidesaguaranteethatnomore 5.1.1 Fastpath thanoneinstanceoftheAMroleisbroughtdownforOSorapplica- tionupgrade.Thisnotionofinstance-by-instanceupdatemaintains TomeasuretheeffectivenessofFastpathanditsimpactonhost availabilityduringupgrades. AnantausesPaxostoelectaprimary CPU,weconductedthefollowingexperiment. Wedeployeda20 andonlytheprimarydoesallthework. VM tenant as the server and two 10 VM tenants as clients. All Mux: Muxhastwomaincomponents—akernel-modedriver theclientVMscreateuptotenconnectionstotheserverandup- andauser-modeBGP[20]speaker. Thekernel-modedriverisim- load1MBofdataperconnection. WerecordedtheCPUusageat plementedusingtheWindowsFilteringPlatform(WFP)[30]driver thehostnodesandoneMux. TheresultsareshowninFigure11. model. ThedriverinterceptspacketsattheIPlayer,encapsulates We found the host CPU usage to be uniform across all hosts, so andthensendsthemusingthebuilt-inforwardingfunctionofthe weonlyshowmedianCPUobservedatarepresentativehost. As OS. Delegating routing to the OS stack has allowed Mux code expected, as soon as Fastpath is turned on, no new data transfers to remain simple, especially when adding support for IPv6, as it happenthroughtheMux. Itonlyhandlesthefirsttwopacketsof does not need to deal with IP fragmentation and next-hop selec- anynewconnection. OncetheMuxisoutoftheway,italsostops tion. Thedriverscalestomultiplecoresusingreceivesidescaling beingabottleneckfordatatransferandVMscanexchangedataat (RSS)[22].SincetheMuxdriveronlyencapsulatesthepacketwith thespeedallowedbytheunderlyingnetwork. anewIPheaderandleavestheinnerIPheaderanditspayloadin- 5.1.2 TenantIsolation tact,itdoesnotneedtorecalculateTCPchecksumandhenceitdoes not need any sender-side NIC offloads. For IPv4, each Mux can Tenantisolationisafundamentalrequirementofanymulti-tenant hold20,000loadbalancedendpointsand1.6millionSNATports service. In the following we present two different experiments 214 Figure 12: SYN-flood Attack Mitigation. Duration of impact showsthetimeAnantatakestodetectandblack-holetraffictothe Figure 13: Impact of heavy SNAT user H on a normal user victimVIPonallMuxes. N.HeavyuserHseeshigherlatencyandhigherSYNretransmits. NormaluserN’sperformanceremainsunchanged. thatshowAnanta’sabilitytoisolateinboundpacketandoutbound SNAT abuse. For these experiments, we deployed five different tenants,eachwithtenvirtualmachines,onAnanta. SYN-floodIsolation: TomeasurehowquicklyAnantacaniso- lateaVIPunderaSYN-floodattack,weranthefollowingexperi- ment(otherpacketratebasedattacks,suchasaUDP-flood,would showsimilarresult.) WeloadAnantaMuxeswithabaselineload andlaunchaSYN-floodattackusingspoofedsourceIPaddresses on one of the VIPs. We then measure if there is any connection lossobservedbyclientsoftheothertenants. Figure12showsthe maximumdurationofimpactobservedovertentrials. Asseenin the chart, Mux can detect and isolate an abusive VIP within 120 secondswhenitisrunningundernoload,minimumtimebeing20 seconds.However,undermoderatetoheavyloadittakeslongerto detectanattackasitgetshardertodistinguishbetweenlegitimate Figure14: Connectionestablishmenttimeexperiencedbyout- andattacktraffic.WeareworkingonimprovingourDoSdetection boundconnectionswithandwithoutportdemandprediction. algorithmstoovercomethislimitation. SNATPerformanceIsolation: SNATportallocationatAnanta Managercouldbeasubjectofabusebysometenants. Itispossi- dataispartitionedintobucketsof25ms.Theminimumconnection blethatatenantmakesalotofSNATconnectionscausingimpact establishmenttimetotheremoteservice(withoutSNAT)is75ms. on other tenants. To measure the effectiveness of per-VM SNAT Figure14showsconnectionestablishmenttimesforthefollowing isolationatAM,weconductedthefollowingexperiment. Asetof twooptimizationswhenthereisnootherloadonthesystem. normalusetenants(N)makeoutboundconnectionsatasteadyrate SinglePortRange: Inresponsetoaportrequest,AMallocates of150connectionsperminute. Whereas,aheavySNATuser(H) eightcontiguousportsinsteadofasingleportandreturnsthemto keepsincreasingitsSNATrequests. WemeasuretherateofSYN therequestingHA.HAusestheseportsforanypendingandsub- retransmits and the SNAT response time of Ananta at the corre- sequentconnections. TheHAkeepsanyunusedportsuntilapre- spondingHAs.Figure13showstheaggregateresultsovermultiple configured timeout before returning them to AM. By doing this trials. Asseeninthefigure,thenormaltenants’connectionskeep optimization, only one in every eight outbound connections ever succeeding at a steady rate without any SYN loss; and its SNAT resultsinarequesttoAM.ThisisevidentfromFigure14—88% port requests are satisfied within 55ms. The heavy user, on the connectionssucceedintheminimumpossibletimeof75ms. The otherhand,startstoseeSYNretransmitsbecauseAnantadelaysits remaining 12% connections take longer due to the round-trip to SNATrequestsinfavorofN.ThisshowsthatAnantarewardsgood AM.Withouttheportrangeoptimizationeverynewconnectionre- behaviorbyprovidingfasterSNATresponsetime. questthatcannotbesatisfiedusingexistingalreadyallocatedports willmakearound-triptoAM. 5.1.3 SNATOptimizations DemandPrediction: Whenthisoptimizationisturnedon,AM attemptstopredictportdemandofaDIPbasedonitsrecenthis- AkeydesignchoiceAnantamadeistoallocateSNATportsin tory. IfaDIPrequestsnewportswithinaspecifiedintervalfrom AMandthenreplicatetheportallocationstoallMuxesandother itspreviousrequests,AMallocatesandreturnsmultipleeight-port AMreplicastoenablescaleoutandhighavailabilityrespectively. portrangesinsteadofjustone.AsshowninFigure14,withthisop- Since port allocation takes place for the first packet of a connec- timization96%ofconnectionsaresatisfiedlocallyanddon’tneed tion, it can add significant latency to short connections. Ananta tomakearound-triptoAM.Furthermore,sinceAMhandlesfewer overcomes these limitations by implementing three optimizations requests,itsresponsetimeisalsobetterthanallocatingasingleport asmentionedin§3.5.1. Tomeasuretheimpactoftheseoptimiza- rangeforeachrequest. tions we conducted the following experiment. A client continu- ouslymakesoutboundTCPconnectionsviaSNATtoaremoteser- viceandrecordstheconnectionestablishmenttime. Theresulting 215 Figure15: CDFofSNATresponselatencyforthe1%requests handledbyAnantaManager. Figure17: DistributionofVIPconfigurationtimeovera24-hr period. over this period across all test tenants was 99.95%, with a mini- mum of 99.92% for one tenant and greater than 99.99% for two tenants.FiveofthelowavailabilityconditionsbetweenJan21and Jan26happenedduetoMuxoverload. TheMuxoverloadevents wereprimarilycausedbySYN-floodattacksonsometenantsthat arenotprotectedbyourDoSprotectionservice. Twoavailability drops were due to wide-area network issues while the rest were falsepositivesduetoupdateofthetesttenants. 5.2.3 Scale For control plane, Ananta ensures that VIP configuration tasks are not blocked behind other tasks. In a public cloud environ- Figure 16: Availability of test tenants in seven different data ment,VIPconfigurationtaskshappenatarapidrateascustomers centersoveronemonth. add, delete or scale their services. Therefore, VIP configuration timeisanimportantmetric. SinceAnantaneedstoconfigureHAs 5.2 RealWorldData andMuxesduringeachVIPconfigurationchange,itsprogramming timecouldbedelayedduetoslowHAsorMuxes.Figure17shows SeveralinstancesofAnantahavebeendeployedinalargepub- the distribution of time taken by seven instances Ananta to com- liccloudwithacombinedcapacityexceeding1Tbps. Ithasbeen pleteVIPconfigurationtasksovera24-hrperiod.Themediancon- servingInternetandintra-datacentertrafficneedsofverydiverse figuration time was 75ms, while the maximum was 200seconds. setoftenants,includingblob,tableandqueuestorageservicesfor These time vary based on the size of the tenant and the current overtwoyears.Herewelookatsomedatafromrealworld. healthofMuxes.ThesetimesfitwithinourAPISLAforVIPcon- 5.2.1 SNATResponseLatency figurationtasks. For data plane scale, Ananta relies on ECMP at the routers to Basedonproductiondata,Anantaserves99%oftheSNATre- spreadloadacrossMuxesandRSSattheNICtospreadloadacross questslocallybyleveragingportreuseandSNATpreallocationas multipleCPUcores.Thisdesignimpliesthatthetotaluploadthrough- describedabove.Theremainingrequestsfortenantsinitiatingalot putachievedbyasingleflowislimitedbywhattheMuxcanachieve of outbound requests to a few remote destinations require SNAT usingasingleCPUcore. Onourcurrentproductionhardware,the portallocationtohappenattheAM.Figure15showsthedistribu- Mux can achieve 800Mbps throughput and 220Kpps using a sin- tion of latency incurred for SNAT port allocation over a 24 hour glex64,2.4GHzcore. However,Muxescanachievemuchhigher windowinaproductiondatacenterfortherequeststhatgotoAM. aggregatethroughputacrossmultipleflows. Inourproductionde- 10%oftheresponsesarewithin50ms,70%within200msand99% ployment, wehavebeenabletoachievemorethan100Gbpssus- within2seconds. Thisimpliesthatintheworstcase,oneinevery taineduploadthroughputforasingleVIP.Figure18showsband- 10000connectionssuffersaSYNlatencyof2secondsorhigher; widthandCPUusageseenoveratypical24-hrperiodby14Muxes however,veryfewscenariosrequirethatnumberofconnectionsto deployedinoneinstanceofAnanta. HereeachMuxisrunningon thesamedestination. a 12-core 2.4Ghz intel(cid:13)R Xeon CPU. This instance of Ananta is 5.2.2 Availability serving12VIPsforblobandtablestorageservice. Asseeninthe figure,ECMPisloadbalancingflowsquiteevenlyacrossMuxes. Aspartofongoingmonitoringforourcloudplatform,wehave Each Mux is able to achieve 2.4Gbps throughput (for a total of multiple test tenants deployed in each data center. A monitoring 33.6Gbps)usingabout25%ofitstotalprocessingpower. serviceconnectstotheVIPofeverytesttenantfrommultiplegeo- graphiclocationsandfetchesawebpageonceeveryfiveminutes. 6. OPERATIONALEXPERIENCE Figure16showsaverageavailabilityoftesttenantsinsevendif- ferentdatacenters. Iftheavailabilitywaslessthan100%forany Wehavebeenofferinglayer-4loadbalancingaspartofourcloud five minute interval, it makes up a point in the above graph. All platformforthelastthreeyears.Theinabilityofhardwareloadbal- theotherintervalshad100%availability. Theaverageavailability ancerstodealwithDoSattacks,networkelasticityneeds,thegrow- 216