Apache Cassandra From The Ground Up Akhil Mehra Thisbookisforsaleathttp://leanpub.com/apachecassandrafromthegroundup Thisversionwaspublishedon2017-09-18 ThisisaLeanpubbook.LeanpubempowersauthorsandpublisherswiththeLeanPublishing process.LeanPublishingistheactofpublishinganin-progressebookusinglightweighttoolsand manyiterationstogetreaderfeedback,pivotuntilyouhavetherightbookandbuildtractiononce youdo. ©2015-2017AkhilMehra Contents AnIntroductionToNoSQL&ApacheCassandra . . . . . . . . . . . . . . . . . . . . . . . 1 DatabaseEvolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 NoSQLDatabase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 KeyFoundationalConcepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 ApacheCassandra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 An Introduction To NoSQL & Apache Cassandra WelcometoApacheCassandrafromTheGroupUp.Theprimarygoalofthisbooktohelpdevelopers and database administrators understand Apache Cassandra. We start off this chapter exploring database history. An overview of database history lays the foundation for understanding various types of databases currently available. This historical context enables a good understanding of the NoSQL ecosystem and Apache Cassandra’s place in this ecosystem. The chapter concludes by introducingApacheCassandra’sitskeyfeaturesandapplicableusecases.Thiscontextisinvaluable toevaluateandgettogripswithApacheCassandra. Database Evolution Thosewhoareunawareofhistoryaredestinedtorepeatit Let’s start with the basics. What is a database? According to Wikipedia, a database is an organized collectionofdata.Purelymathematicalcalculationsweretheprimaryuseofearlydigitalcomputers. Usingcomputersformathematicalcalculationswasshortlived.Applicationsgrewincomplexityand needed toread, writeand manipulatedata. Tocopewith thegrowingcomplexitycompanieswrote individual software applications that would enable users to read, write and manipulate data. Early databases stored data sequentially on media such as paper and magnetic tapes. Sequential access made fast retrieval of individual records impossible. The advent of magnetic spinning disk allowed randomaccesstoindividualrecords.Advancementinfilemanagementledtofurtherrandomaccess improvements.TheinventionoffilemanagementsystemssuchasIndexSequentialAccessMethod (ISAM) enabled sequential and random access to files. Improved random access led to the birth of Online Transaction Processing systems (OLTP). Initially, every application wrote its custom code for storing and retrieving data. Everyone writing custom code for data manipulation was an unproductiveapproach.DatabaseManagementSystems(DBMS)werecreatedtoaddressthisneed. DBMS is a software application/component responsible for storing, manipulating and retrieving data.1 Just like any technology databases have evolved over the past three decades. Database evolution, based on data models, can be broken up into three major eras, i.e., Navigational, SQL/Relational, andPostRelational.2 1NextGenerationDatabases:NoSQL,NewSQL,andBigData 2NextGenerationDatabases:NoSQL,NewSQL,andBigData AnIntroductionToNoSQL&ApacheCassandra 2 • Navigational Databases Era - Navigational database were popular in the 1960’s and early 1970’s. The primary goal of early DBMS was to provide concurrent data manipulation while maintaining the integrity of the database. It also optimized data retrieval via caching and sophisticated algorithms. Early DBMS ran exclusively on mainframe computer systems. These DBMS’s were called Navigational Databases because theymade heavy use of pointers and links. Finding data involved traversing these pointers and links. Two main types of navigationaldatamodelswerethehierarchicalmodelandthenavigationalmodel.3 • SQL/Relational Era - The seminal paper “A Relational Model of Data for Large Shared Data Banks” written by E. F. Codd in 1970 sparked the second database revolution 4. Codd believedthatexistingdatabase(NavigationalDB’s)weretoohardtouseandlackedtheoretical foundation. Codd advocated searching for data by its content instead of following links. His paperlaiddownthecoreideasfortherelationaldatamodel.Therelationalmodelfocussedon datapresentedtousersinsteadoffocusingonhowdatalayoutondisk.AlthoughCodd’spaper providedthefoundationfortherelationalmodel,itdidnotdefinewaysofhandlingconcurrent data modification and access. In late 1970’s Jim Gray established the most widely accepted transaction model in his paper “The Transaction Concept: Virtues and Limitations”5. A few years later Andreas Reuter and Theo Härder coined the term ACID6 (Atomic, Consistent, Independent, and Durable) that described Jim Gray’s set of properties. IBM built the first relationaldatabase SystemRin 1974.IBM’s SanJose ResearchLaboratorydeveloped System R as part of a research project. Initially, researches theorized that a database would struggle to provide both transaction processing and performance. System R was a seminal project whichbustedthismyth.SystemRalsoprovidedthefirstimplementationofStructuredQuery Language(SQL).ThesuccessofSystemRresultedinthedevelopmentofmanynewRDBMS inthesucceedingdecade.TheseincludeSybase,MicrosoftSQLServer,Informix,MySQL,and DB2. These databases relied on three fundamental principles, i.e., the relational model, SQL language,andtheACIDtransactionmodel.Relationaldatabaseswerethedefactochoicefor applicationstorageneedstillthelate2000’s7. • Post Relational Era - The massive explosion in data, i.e., Big Data drove the post relational database revolution. Big data is a broad term for large data sets. These data sets are often complicated and unprocessable by traditional data processing applications. In 2012 Gartner defined Big data as “high volume, high velocity, and/or high variety information assets that need new forms of processing to enable enhanced decision making, insight discovery and process optimization”8. Significant challenges around big data include capture, curation, storage, analysis, querying and visualization of these information assets. For over thirty years Relations Database Management Systems (RDBMS) has been the de facto choice for applications data storage needs. The Big Data revolution changed this. It challenged the RDBMS’s domination over the storage space. Databases were now required to store massive 3NextGenerationDatabases:NoSQL,NewSQL,andBigData 4ARelationalModelofDataforLargeSharedDataBanks 5TheTransactionConcept:VirtuesandLimitations 6Principlesoftransaction-orienteddatabaserecovery 7NextGenerationDatabases:NoSQL,NewSQL,andBigData 8GartnerSaysSolving‘BigData’ChallengeInvolvesMoreThanJustManagingVolumesofData AnIntroductionToNoSQL&ApacheCassandra 3 amounts of structured, semi-structured and unstructured data. The explosion of data, both structured and unstructured, has made the need to scale and handle non-relational data im- perative.InternationalDataCorporation(IDC)estimatesthattheworld’sdigitalinformation isdoublingeverytwoyears9,alargepartofwhichissemistructuredorunstructureddata.The explosion in big data led to the emergence of a vast number of open source and commercial RDBMS alternatives. These new breeds of databases were called NoSQL database. More on NoSQLdatabaselaterinthischapter. DatabaseEra Scaling Asestablishedintheprevioussectionthepostrelationalerawasdrivenbytheneedtoscaledatabase. Sowhatisscalability?Scalabilityistheabilitytohandleagrowingworkloadinanefficientandcost effectivemanner. Vertical vs. Horizontal Scaling Thereareessentiallytwowaystoscale: • VerticalScaling-Verticalscalingisalsoknownasscalingup.Verticalscalingreferstoadding moreresourcetoasinglenode,i.e.,addinginadditionalCPU,RAMandDisktoenableasingle 9ExtractingValuefromChaos AnIntroductionToNoSQL&ApacheCassandra 4 node to handle a growing workload. Vertical scaling has many limitations the most obvious one being outgrowing the largest available system. Vertical scaling is also more expensive as yourgrow.Costwisescalingverticallyisnotlinear. • Horizontal Scaling - Horizontal scaling is also called scaling out. Horizontal scaling is adding capacity by increasing the number of machines/nodes to a system so that each node can share the processing. Horizontal scaling is a cheaper and more flexible option. This flexibility does come at a cost. Sharing processing and storage amongst an army of nodes is complex. Horizontal scaling makes use of distributed computing to achieve scalability. AndrewS.Tanenbaumdefineddistributedsystemas“Acollectionofindependentcomputers that appears to its users as a single coherent system.”. There are three key aspects to a distributedsystem:Theseare: – Nodes/computersoperateconcurrently. – Nodes/computersfailindependently. – Computersdonotshareaglobalclock. Building and maintaining distributed systems is hard. Only use distributed systems when necessary. HorizontalvsVerticalScaling Newandemergingtechnologiesprefertoscalehorizontallybecause: • Increasecapacityonthefly. • Costeffectiveincomparisontoverticalscaling. • Moreover,intheory,itisinfinitelyscalablesinceaddingnodesincreasescapacityproportion- ally. Scaling Hardware To understand scaling it is important to grasp possible approaches to scaling hardware, i.e., possible hardware deployment architectures. The hardware deployment architecture chosen by a database dictates how it can scale. At a high level, there are three different hardware deployment architectures.Theseare: AnIntroductionToNoSQL&ApacheCassandra 5 • Shared Memory, i.e., Traditional Deployment Architecture - Shared memory is the standard traditional hardware architecture used by database systems. This architecture is characterizedbyhavingmanycoressharingablockofRAMviaacommoncacheandmemory bus. In other words, it is a single machine with many cores accessing a shared memory and singledisk.Scalingusingthisapproach(verticalscaling)isbuyingbiggerandbetterhardware, i.e., you scale by adding more CPU, RAM to your existing machine. Highly parallel shared memory machines are one of the last cash cows of the hardware industry. Traditionally RDBMSdatabasehasworkedwellonsharedmemoryarchitecture. AnIntroductionToNoSQL&ApacheCassandra 6 SharedMemoryArchitecture • Shared Disk - A shared disk approach is characterized by independent nodes which have theirownRAMandCPUbutshareacommondisk.ShareddiskfilesystemsuseStorageArea Network (SAN) to provide direct disk access from multiple computers at a block level. The Shared Disk architecture has gained traction with the rise in popularity of a Storage Area Networks(SAN).PopularRDBMSsuchasOracleandMSSQLuseashareddiskarchitecture toscalehorizontally. AnIntroductionToNoSQL&ApacheCassandra 7 SharedDiskArchitecture • Shared Nothing - A shared nothing architecture is characterized by having a cluster of independent machines communicating over a high-speed network. There is no way of accessingmemoryordiskofanothersystem.Itisuptothedatabaseimplementortocoordinate efficiently among various machines. Data storage is spread across the cluster as each part of a cluster stores a portion of the data. The main advantage of this approach is the ability to scale.Sharednothingarchitecturesarescalablelinearlybecausethereisnosinglebottleneck inthearchitectureandhavebeenproventoscalelinearly.