API Blender: A Uniform Interface to Social Platform APIs Georges Gouriten Pierre Senellart InstitutTélécom;TélécomParistech;CNRSLTCI InstitutTélécom;TélécomParistech;CNRSLTCI 46,rueBarrault,Paris,France 46,rueBarrault,Paris,France [email protected] [email protected] 3 ABSTRACT software projects, whetherit is for authentication purposes, 1 to collect information about a user, to present mash-ups 0 With the growing success of the social Web, most Web de- of popular social Web data, or for a myriad of other rea- 2 velopers have to interact with at least one social Web plat- sons. Our perspective comes from the need of archiving form, which implies studyingtherelated API specifications. n important social data for preservation purposes.1 Regular These are often only informally described, may contain er- a Web archives, such as those built by the Internet Archive2, rors, lack harmonization, and generally speaking make the J often include content from or pointers to social Web plat- developer’sworkdifficult. Mostattemptstosolvethisprob- 0 forms butdo not benefit from API data. As a consequence, lem,proposingformaldescriptionlanguagesforWebservice 1 the archives are either partial – Facebook disallows generic APIs, havehad limited success outsideof B2B applications; webelieveitisduetotheirtop-downnature. Inaddition,a crawlingofitspublicpages–orlacksomeextrainformation ] thattheAPIcan provide,forinstanceextracted entitieson E programmer dealing with one or several of these APIs has Twitter. Designinganarchivalcrawler forthesocialWebre- S to deal with a number of related tasks such as data inte- quiresinterfacingwiththemultiplesocialWebAPIs,aswell . gration, requests chaining, or policy management, that are s cumbersometoimplement. InspiredbytheSPOREproject, asrespectingthepolicies imposedbytheseservices,suchas c we present API Blender, an open-source solution to de- limiting thenumberof requests perhour. [ Many projects thus involve numerous interactions with scribe, interact with, and integrate the most common so- varioussocialplatforms,sometimeswithcomplexlogicssuch 1 cial Web APIs. In this perspective, we first introduce two v newlightweightdescriptionformatsforrequestsandservices asgettingthesocialgraphtillthethirdrankofusershaving 6 anddemonstratetheirrelevancewithrespecttocurrentplat- mentioned a specific keyword. Understanding the related 8 form APIs. WepresentourPythonimplementation of API API specifications can be challenging. There is no de facto 0 Blender and its features regarding authentication, policy standardtodescribethemandtheycancontainmistakesor 2 approximations. Thereisnoclearspecification,forinstance, management and multi-platform dataintegration. . of how many requests per hour are allowed on the Twitter 1 search API. For the most popular platforms, specific lan- 0 Categories andSubject Descriptors guage libraries sometimes exist but they often require the 3 H.3.5 [Information Storage and Retrieval]: Online In- same learning phase. 1 formationServices—Web-basedservices;D.3.2[Programming Having a unified description of the different social Web : v Languages]: Language Classifications—Python APIsis atechnicalchallenge. An early step was takenwith i WSDL [2], a Web Services Description Language standard- X General Terms ized by the W3C. WSDL has been heavily used in the in- r dustry and is at the core of many service-oriented software a Design, Standardization projects[5]. However,mostpopularsocialplatformsinclud- ing Facebook, Twitter, or Google+ and many other Web Keywords servicesarenot currentlyofferinganyWSDLdescription of social Web,API,REST, dataintegration theirAPIand donot seem to haveanyplansto doso. The reasons are manifold: WSDL-based services are often con- 1. INTRODUCTION sidered heavy machinery for such lightweight interfaces [1], WSDLhashistoricallyfocusedonSOAPmessageexchanges Interactingwith platforms like Facebook, Youtube,Twit- rather than on RESTful APIs though it can now express ter,Flickr,orGoogle+ becomesanimportantpartofmany both [10], WSDL has no support for important API meta- data such as policy management or the description of a se- quence of service calls3. In reaction to WSDL, some other approachestoWebservicesdescriptionhavebeenproposed, 1ARCOMEM project, http://www.arcomem.eu/ 2http://www.archive.org/ Copyrightisheldbytheauthor/owner(s). 3BPEL[8]istypicallyusedinB2Bprojectsthatneedservice WWW 2012 Developer Track,April18–20,2012,Lyon,France. orchestration, but leads to even heaviermachinery. a prime example being WADL [6] but they have not met tionofauthenticationandpolicyusuallyrequiredtointeract with more success on popularsocial Webplatforms. with social platform WebAPIs. Anotherperspectiveisnecessary. SpringSocial4 isaJava framework to interact with the different social platforms. Server Object We believe this bottom-up approach is a very promising "name": string, way to make the developers’ work simpler. Spring Social "host": string, implements a number of useful functionalities (authentica- "port": integer, tion, uniform interface to some of the API types, etc.) but "authentication": auth_object, does not fulfill our requirements. On the one hand, some "policy": policy_object, important features, especially for archival crawling, are not "interactions": [interaction_object] considered, such as limits on the number of requests. On Port, policy, and authentication are optional. The port de- the other hand, using Spring Social requires understanding faults to 80. an important amount of code before being able to interact Two authentication protocols are supported at the mo- withasocialplatform. Togiveanorderofmagnitudeofthe ment, one based on a unique authentication URL with pa- size of the software, the core v1.02 contains 405 files, with- out implementing any Web API.5 With API Blender, we rameters and the otheron the three-legged OAuth2[7]. aim at more simplicity and flexibility,as highlighted bythe example of use we give in Section 3. Simple Authentication Object OurmainsourceofinspirationhasbeentheSPOREproject[4]. "request_token_url": uri, It consists in a simple implementation-agnostic JSON for- "url_parameters": object mat allowing to describe Web APIs designed according to theRESTprinciples. Theproject hasbeenstartedrecently By simple authentication, we mean authentication with pa- and is still underdevelopment. rameters such as API key or login and password passed to With API Blender, we extend SPORE with the follow- a uniqueURL so as to receive theauthentication token. ing contributions: 1. twosimpledescriptionformatsattheAPIandrequest OAuth2 Authentication Object levels,adaptedtosocialplatforms,sortingSPOREout "consumer_key": string, and complementing it; "consumer_secret": string, 2. anopenPythonimplementation,allowing toeasily in- "request_token_url": uri, tegrate various platforms; "access_token_url": uri, 3. thefollowingfeatures,someofthemleftoutofexisting "authorize_url": uri tools or libraries: authentication, server policy man- Many social platforms (e.g., Twitter, Facebook, Google+) agement,multi-platformdataintegration,andrequest accept OAuth2authentication. chaining. WedesignedAPIBlenderinspiredbywhatweobserved on five prominent social platforms we identified: Twitter, Policy Object Facebook, Google+, FlickR and Youtube. However, we "requests_per_hour": integer, stroveatkeepingahighflexibilitysothatitcanbeextended "too_many_calls_response_code": integer, to many otherWeb APIs. "too_many_calls_waiting_seconds": integer Ourarticleisorganizedasfollows. InSection2,wepresent An overload can be detected by counting the requests or descriptions formats and discuss their relevance to social receivingatoo-many-callsresponse. Inthelattercase,API platforms. We then detail in Section 3 our implementation Blenderwillsnoozeforthespecifiedamountoftimebefore in Pythonand its features. testing if thecounterhas been reset. 2. DESCRIPTIONFORMATS Interactiondescriptionformat. An interaction is a class AWebAPIconsistsinasetofHTTPrequestmessagesas- of HTTPrequestswith acommon root pathandtheirasso- sociatedtoresponses,senttoaspecificHTTPserverhaving ciatedresponses. HerealsoweextendedSPOREandadded its own rules. Note that Twitter has different APIs corre- theresponse object. spondingtodifferenthosts: forinstance,api.twitter.com:80 or search.twitter.com:80.6 We describe a Web API with Interaction Object severalobjectsthatallowtodescribetheserveranditsrules "name": string, (with respect to access policies) as well as the interactions "description": string, it offers. We find JSON [3] light and readable and have "request": request_object, chosen touseit as a serialization. In what follows, we tried "response": response_object usingstraightforwardnamesandself-explainingconventions to definethedifferent elements. The description is optional. Serverdescriptionformat. WehaveextendedSPOREwith Request Object a consistent oriented-object approach, as well as the addi- "root_path": string, 4http://www.springsource.org/spring-social "method": string "raw_content": string 5http://s3.amazonaws.com/dist.springframework.org/ "url_parameters": [ 6https://dev.twitter.com/docs/history-rest-search-api [ string, # key, e.g., "id" string, # type, e.g., "integer" Features. API Blender implements several precious fea- boolean # is it an optional parameter? ture. It supports the two main authentication types: using object # the default value, it can be null ] a single URL with parameters and OAuth2 [7] thanks to ] Python OAuth28. API Blender also ensures respect of the server policy; ThemethodhastobeGET,PUT,POSTorDELETE.Pro- when the hourly limit is reached or when a too-many-calls vidingraw content is optional and usefulonly for PUTand responseisidentified,thepolicymanagerwillstopforsome POST methods. If a default value is set on a URL param- timeandperiodically testif thecounterhasbeenreset. Er- eter, it will be automatically passed with the default value ror handling is taken into consideration too, whether it re- unless it is explicitly set as null. This feature can be use- gards a non-conforming configuration file or an unexpected ful in many case such as requesting a default value of 100 response. Finally,API Blendergivesthepossibility toex- responses per pages for a full-text query on Twitter search tract and normalize elements from responses. This feature API. supports simple field extraction and standardization at the momentbutthesameprocesswillbepossiblewitharbitrary Response Object subtreetransformations in thenearfuture. "expected_status_code": integer, "serialization_format": serialization_format, Requestchaining. Theopennatureof APIBlendercom- "expected_schema": json_schema_object, bined to the flexibility of Python can fill many needs. Re- "integration": extractor_object quest chaining becomes very simple with Python and com- plex interactions can become easy-to-maintain Python li- The expected code is optional and defaults to 200. The se- braries. We illustrate this with the following example on rialization format has to be JSON or XML at the moment. two Twitter APIs. The program below retrieves the last Theexpectedschemaoftheresponseisoptionalandcanbe three pages of tweets containing the keywords“good spirit” defined as a JSON schema [9]. At the moment, we define thenfetchesthelocalsocialnetwork(followersandfollowees) a simple extractor that allows a mappingbetween a unified of the authorsof thetweets. model and response fields. We use ‘.’ as a path separator. Forinstance,wecouldhave"post.content": "post_data.text" if our integrated model was {"post": {"content": string}} Example of request chaining with Python with a response model of {"post_data": {"text": string}}. import apiblender With a careful normalization model (for instanceusingcon- cepts of an ontology), this allows to integrate data coming blender = apiblender.Blender() from different platforms. As an extension, this semantic model could also be used to describe the inputs of services, # Retrieving 3 pages of result a first step towards semantic service orchestration. blender.load_server("twitter-search") blender.load_interaction("search") users = set() 3. THEPYTHONIMPLEMENTATION for p in range(1,3): blender.set_parameters({"q": "good spirit", Python is becoming increasingly popular among develop- "page": p}) ers. On the social coding platform GitHub, it is ranked response = blender.blend() third.7 We find Python to be simple, flexible, and to have ts=response["prepared_content"]["results"]: many useful libraries. We have chosen to implementation API Blenderin this language. API Blender is available for twitt in ts users.add(twitt["from_user"]) online at https://github.com/netiru/apiblender. # Retrieving followers / followees for each user Structure. The module structureoffered by Python allows blender.load_server("twitter-generic") us toadopt thefollowing light structure. for user in users: blender.load_interaction("followers") blender.set_parameters({"screen_name":user}) API Blender package followers = blender.blend() main.py Controller blender.load_interaction("followees") server.py Server and interactions blender.set_parameters({"screen_name":user}) policy.py Policy management followees = blender.blend() auth.py Authentification management config/ JSON configuration files # Printing everything --general.json General config print("User Name: %s" % user) --apis/ API config files print("\tFollowers: %s" % \ followers["prepared_content"]) WefounditconvenienttohaveonefileperAPIserverwhere print("\tFollowees: %s" % \ we gather the descriptions for the server and its interac- followees["prepared_content"]) tions. Currently,theAPIBlendersupportsthetwoTwitter APIs (generic and search), Facebook, Google+, FlickR and Youtube. 7After JavaScript and Ruby, 8Created and maintained by SimpleGeo Inc. https://github.com/languages https://github.com/simplegeo/python-oauth2 4. CONCLUSIONS [3] D.Crockford. The application/json media typefor API Blender has been designed in the context of the JavaScript object notation (JSON). ARCOMEM project on social Web archiving, and is put to http://www.ietf.org/rfc/rfc4627.txt?number=4627, use in thisproject to crawl and integrate data from various July 2006. IETF, Network Information Group. social Web platforms. We have found its flexibility useful [4] F. Cuny.SPORE– Specificationsto a POrtable Rest in the light of the dynamicity of social Web platforms and Environment. managed to conveniently integrate the five platforms cur- https://github.com/SPORE/specifications, rentlysupported: Twitter,Facebook,FlickR,Google+,and November2011. Youtube. Itisofpotentialuseinanyapplicationthatneeds [5] T. Erl. Service-Oriented Architecture: A Field Guide toaccesssimilarREST-inspiredWebAPIsandtoexportre- to Integrating XML and Web Services. PrenticeHall sponses in a common schema. The source code being avail- PTR, UpperSaddleRiver, NJ,USA,2004. able on GitHub, we hope to solicit contributions, either in [6] M. Hadley.Web Application Description Language. theform of extensionsof thebasefunctionalities, orinthat http://www.w3.org/Submission/wadl/, August2009. ofAPIdescriptions. Forfuturework,weseemanypromising W3C Member Submission. opportunities such as: [7] E. Hammer, D.Recordon, and D.Hardt.The OAuth 1. smarter processing of responses, making use of the se- 2.0 authorization protocol. mantics of the services described, in the spirit of the http://tools.ietf.org/html/draft-ietf-oauth-v2-23, semantic description ofWeb services`a laOWL-S[11]; January 2012. 2. developingmore standard request chaininglibraries; [8] D.Jordan and J. Evdemon. Web 3. a possible integration of thedifferent inputschemas; servicesbusinessprocessexecutionlanguageversion2.0. 4. more research for asmarter snooze management; http://docs.oasis-open.org/wsbpel/2.0/OS/wsbpel-v2.0-OS.html 5. distributingrequestsacross different servers. April2007. Theyallrequiretobeveryconsciousoftheexistingtrade-off [9] E. K.Zyp.A JSON media typefor de- between completeness and flexibility. scribingthestructureandmeaningofJSONdocuments. http://tools.ietf.org/html/draft-zyp-json-schema-03, 5. ACKNOWLEDGMENTS November2010. The described work was funded by the European Union [10] L. Mandel. DescribeREST Web services with WSDL SeventhFrameworkProgramme(FP7/2007–2013)undergrant 2.0. agreement 270239 (ARCOMEM). http://www.ibm.com/developerworks/webservices/library/ws-restws May 2008. IBM Technical Library. 6. REFERENCES [11] D.Martin, M. Burstein,J. Hobbs,O.Lassila, D.McDermott, S. McIlraith, S.Narayanan, [1] G. Alonso. Mythsaround web services. IEEE Data M. Paolucci, B. Parsia, T. Payne,E. Sirin, Eng. Bull.,25(4):3–9, 2002. N.Srinivasan,and K.Sycara. OWL-S:Semantic [2] R.Chinnici, J.-J. Moreau, A. Ryman,and Markup for Web Services. S.Weerawarana. Web Services Description Language http://www.w3.org/Submission/OWL-S/, November (WSDL)2.0. http://www.w3.org/TR/wsdl20/, June 2004. W3C Member Submission. 2007. W3C Recommendation.