THE ACL ANTHOLOGY NETWORK CORPUS Dragomir R. Radev Department of Electrical Engineering and Computer Science School of Information University of Michigan, Ann Arbor Pradeep Muthukrishnan Department of Electrical Engineering and Computer Science University of Michigan, Ann Arbor Vahed Qazvinian Department of Electrical Engineering and Computer Science University of Michigan, Ann Arbor Amjad Abu-Jbara Department of Electrical Engineering and Computer Science University of Michigan, Ann Arbor ABSTRACT We introduce the ACL Anthology Network (AAN), a comprehensive manually curated networked database of citations, collaborations, and summaries in the field of Computational Linguistics. We also present a number of statistics about the network including the most cited authors, the most central collaborators, as well as network statistics about the paper citation, author citation, and author collaboration networks. 1 Introduction The ACL Anthology1 is one of the most successful initiatives of the Association for Computational Linguistics (ACL). The ACL is a society for people working on problems involving natural language and computation. It was initiated by Steven Bird (Bird et al., 2008) and is now maintained by Min Yen Kan. It includes all papers published by ACL and related organizations as well as the Computational Linguistics journal over a period of four decades. ACL Anthology has a major limitation in that it is just a collection of papers. It does not include any citation information or any statistics about the productivity of the various researchers who contributed papers to it. We embarked on an ambitious initiative to manually annotate the entire Anthology and curate the ACL Anthology Network (AAN)2. AAN was started in 2007 by our group at the University of Michigan (Radev et al., 2009a, b). AAN provides citation and collaboration networks of the articles included in the ACL Anthology (excluding book reviews). AAN also includes rankings of papers and authors based on their centrality statistics in the citation and collaboration networks, as well as the citing sentences associated with each citation link. These sentences were extracted automatically using pattern matching and then cleaned manually. Table 1 shows some statistics of the current release of AAN. 1 http://www.aclweb.org/anthology-new/ 2 http://clair.si.umich.edu/anthology/ 1 Number of papers 18,290 Number of authors 14,799 Number of venues 341 Number of paper citations 84,237 Citation network diameter 22 Collaboration network diameter 15 Number of citing sentences 77,753 Table 1: Statistics of AAN 2011 release In addition to the aforementioned annotations, we also annotated each paper by its institution in the goal of creating multiple gold standard data sets for training automated systems for performing tasks like summarization, classification, topic modeling, etc. Citation annotations in AAN provide a useful resource for evaluations multiple tasks in Natural Language Processing. The text surrounding citations in scientific publications has been studied and used in previous work. Nanba and Okumura (1999) used the term citing area to refer to citing sentences. They define the citing area as the succession of sentences that appear around the location of a given reference in a scientific paper and have connection to it. They proposed a rule-based algorithm to identify the citing area of a given reference. In (Nanba et al., 2000) they use their citing area identification algorithm to identify the purpose of citation (i.e. the author’s reason for citing a given paper). In a similar work, Nakov et al. (2004) use the term citances to refer to citing sentences. They explored several different uses of citances including the creation of training and testing data for semantic analysis, synonym set creation, database curation, summarization, and information retrieval. Other previous studies have used citing sentences in various applications such as: scientific paper summarization (Elkiss et al., 2008; Qazvinian and Radev, 2008; Mei and Zhai, 2008; Qazvinian et al., 2010; Qazvinian and Radev, 2010; Abu-Jbara and Radev, 2011a), automatic survey generation (Nanba et al., 2000; Mohammad et al., 2009), and citation function classification (Nanba et al., 2000; Teufel et al., 2006; Siddharthan and Teufel, 2007; Teufel, 2007). Other services that are built more recently on top of the ACL Anthology include the ACL Anthology Searchbench and Saffron. The ACL Anthology Searchbench (AAS) (Schäfer et al., 2011) is a Web-based application for structured search in ACL Anthology. AAS provides semantic, full text, and bibliographic search in the papers included in the ACL Anthology corpus. The goal of the Searchbench is both to serve as a showcase for using NLP for text search, and to provide a useful tool for researchers in Computational Linguistics. However, unlike AAN, AAS does not provide different statistics based on citation networks, author citation and collaboration networks, and content-based lexical networks. Saffron3 provides insights to a research community or organization by automatically analyzing the content of its publications. The analysis is aimed at identifying the main topics of investigation and the experts associated with these topics within the community. The current version of Saffron provides analysis for ACL and LREC publications as well as other IR and Semantic Web publication libraries. 2 Curation The ACL Anthology includes 18,290 papers (excluding book reviews and posters). We converted each of the papers from PDF to text using a PDF-to-text conversion tool (www.pdfbox.org). After this conversion, we extracted the references semi-automatically using string matching. The conversion process outputs all the references as a single block of continuous running text without any delimiters between references. Therefore, we manually inserted line breaks between references. These references were then manually matched to other papers in the ACL Anthology using a “k-best” (with k = 5) string matching algorithm built into a CGI interface. A snapshot of this interface is shown in Figure 1. The matched references were stored together to produce the citation network. If 3 http://saffron.deri.ie/ 2 the cited paper is not found in AAN, we have 5 different options the user can choose from. The first option is “Possibly in the anthology but not found,” which is used if the string similarity measure failed to match the citation to the paper in AAN. The second option, “Likely in another anthology,” is used if the citation is for a paper in a related conference. We considered the following conferences as related conferences AAAI, AMIA, ECAI, IWCS, TREC, ECML, ICML, NIPS, IJCAI, ICASSP, ECIR, SIGCHI, ICWSM, EUROSPEECH, MT, TMI, CIKM and WWW. The third option is used if the cited paper is a journal paper, a technical report, PhD thesis or a book. The last two options are used if the reference is not readable because of an error in the PDF to text conversion or if it is not a reference. We only use references to papers within AAN while computing various statistics. In order to fix the issue of wrong author names and multiple author identities we had to perform some manual post-processing. The first names and the last names were swapped for a lot of authors. For example, the author name "Caroline Brun" was present as "Brun Caroline" in some of her papers. Another big source of error was the exclusion of middle names or initials in a number of papers. For example, Julia Hirschberg had two identities as "Julia Hirschberg" and "Julia B. Hirschberg." Other numerous spelling mistakes existed. For instance, “Madeleine Bates" was misspelled as "Medeleine Bates." There were about 1000 such errors that we had to correct manually. In some cases, the wrong author name was included in the metadata and we had to manually prune such author names. For example, “Sofia Bulgaria” and “Thomas J. Watson” were incorrectly included as author names. Also, there were cases of duplicate papers being included in the anthology. For example, C90-3090 and C90-3091 are duplicate papers and we had to remove such papers. Finally, many papers included incorrect titles in their citation sections. Some used the wrong years and/or venues as well. For example, the following is a reference to a paper with the wrong venue. “Hiroshi Kanayama Tetsuya Nasukawa. 2006. Fully Automatic Lexicon Expansion for Domain- oriented Sentiment Analysis. In ACL.” Figure 1: CGI interface used for matching new references to existing papers 3 The cited paper itself was published in EMNLP 2006 and not ACL 2006 as shown in the reference. In some cases, the wrong conference name was included in the metadata itself. For example, W07-2202 had “IJCNLP” as the conference name in the metadata while the right conference name is “ACL”. Also, we had to normalize conference names. For example, joint conferences like “COLING-ACL” had “ACL-COLING” as the conference name in some papers. Our curation of ACL Anthology Networks allows us to maintain various statistics about individual authors and papers within the Computational Linguistics community. Figures 2 and 3 illustrate snapshots of the different statistics computed for an author and a paper respectively. For each author, AAN includes number of papers, collaborators, author and paper citations, and known affiliations as well as h- index, citations over time, and collaboration graph. Moreover, AAN includes paper metadata such as title, venue, session, year, authors, incoming and outgoing citations, citing sentences, keywords, bibtex item and so forth. In addition to citation annotations, we have manually annotated the gender of most authors in AAN using the name of the author. If the gender cannot be identified without any ambiguity using the name of the author, we resorted to finding the homepage of the author. We have been able to annotate 8,578 authors this way: 6,396 male and 2,182 female. The annotations in AAN enable us to extract a subset of ACL-related papers to create a self- contained dataset. For instance, one could use the venue annotation of AAN papers and generate a new self-contained anthology of articles published in BioNLP workshops. Figure 2: Snapshot of the different statistics computed for an author 3 Networks Using the metadata and the citations extracted after curation, we have built three different networks. The paper citation network is a directed network in which each node represents a paper labeled with an ACL ID number and edges represent citations between papers. The paper citation network consists of 18,290 papers (nodes) and 84,237 citations (edges). The author citation network and the author collaboration network are additional networks derived from the paper citation network. In both of these networks a node is created for each unique author. In the author citation network an edge is an occurrence of an author citing another author. For example, if a paper written by Franz Josef Och cites a paper written by Joshua Goodman, then an edge is created between Franz Josef Och and Joshua Goodman. Self-citations cause self-loops in the author citation network. The author citation network consists of 14,799 unique authors and 573,551 edges. Since the same author may cite another author in several papers, the network may consist of duplicate edges. The author citation network consists of 325,195 edges if duplicates are removed. In the author collaboration network, an edge is created for each collaborator pair. For example, if a paper is written by Franz Josef Och and Hermann Ney, then an edge is created between the two authors. Table 2 shows some brief statistics about the different releases of the data set (2008 – 2011). 4 Table 3 shows statistics about the number of papers in some of the renowned conferences in Natural Language Processing. Figure 3: Snapshot of the different statistics computed for a paper Year Network Paper Author Author citation citation collaboration network network network 2008 n 13,706 11,337 11,337 m 54,538 196,505 39,963 2009 n 14,912 12,499 12,499 m 61,527 230,658 45,429 2010 n 16,857 14,733 14,733 m 72,463 477,124 52,036 | 14733 | 2011 n 18,290 14,799 14,799 m 84,237 573,551 56,966 Table 2: Growth of citation volume (n: number of nodes; m: number of edges) 5 Venue Number of Papers Number of Citations COLING 3,644 12,856 ACL 3,363 25,499 Computational Linguistics 699 12,080 EACL 704 2,657 EMNLP 1,084 7,903 CoNLL 533 3,602 ANLP 334 2,773 Table 3: Statistics for popular venues Various statistics have been computed based on the data set released in 2007 by (Radev et al., 2009a, b). These statistics include modified PageRank scores, which eliminate PageRank's inherent bias towards older papers by normalizing the score by age (Radev et al., 2009a, b), Impact factor, correlations between different measures of impact like h-index, total number of incoming citations, and PageRank. We also report results from a regression analysis using h-index scores from different sources (AAN, Google Scholar) in an attempt to identify multi-disciplinary authors. 4 Ranking This section shows some of the rankings that were computed using AAN. Table 4 lists the 10 most cited papers in AAN along with their number of citations in Google Scholar as of June 2012. The difference in size of the two sites explains the difference in absolute numbers of citations. The relative order is roughly the same except for the more interdisciplinary papers (such as the paper on the structure of discourse), which are disproportionately getting fewer citations in AAN. The highest cited paper is (Marcus et al., 1993) with 775 citations within AAN. The next papers are about Machine Translation, Maximum Entropy approaches, and Dependency Parsing. Table 5 shows the same ranking (number of incoming citations) for authors. In this table, the values in parentheses exclude self-citations. Other ranking statistics in AAN include author h-index and authors with the least Average Shortest Path (ASP) length in the author collaboration network. Tables 6, 7 show top 10 authors according these two statistics respectively. Citations Rank Title AAN Google Scholar 1 775 3,936 Building A Large Annotated Corpus Of English: The Penn Treebank 2 615 2,995 The Mathematics Of Statistical Machine Translation: Parameter Estimation 3 591 3,145 Bleu: A Method For Automatic Evaluation Of Machine Translation 4 475 1,408 Minimum Error Rate Training In Statistical Machine Translation 5 473 1,877 A Systematic Comparison Of Various Statistical Alignment Models 6 436 1,711 Statistical Phrase-Based Translation 7 344 1,346 A Maximum Entropy Approach To Natural Language Processing 8 343 2,929 Attention Intentions And The Structure Of Discourse 9 339 1,488 A Maximum-Entropy-Inspired Parser 10 325 1,399 Moses: Open Source Toolkit for Statistical Machine Translation Table 4: Papers with the most incoming citations in AAN and their number of citations in Google Scholar as of June 2012. 6 Rank citations Author Name 1 (1) 7553 (7463) Och, Franz Josef 2 (2) 5712 ( 5469) Ney, Hermann 3 (3) 4792 ( 4668) Koehn, Philipp 4 (5) 3991 ( 3932) Marcu, Daniel 5 (4) 3978 ( 3960) Della Pietra, Vincent J. 6 (7) 3915 ( 3803) Manning, Christopher D. 7 (6) 3909 (3842) Collins, Michael John 8 (8) 3821 ( 3682) K lein, Dan 9 (9) 3799 (3666) Knight, Kevin 10 (10) 3549 (3532) Della Pietra, Stephen A. Table 5: Authors with most incoming citations (the values in parentheses are using non- self- citations) 4.1 PageRank scores AAN also includes PageRank scores for papers. It must be noted that the PageRank scores should be interpreted carefully because of the lack of citations outside AAN. Specifically, out of the 155,858 total number of citations, only 84,237 are within AAN. Table 8 shows AAN papers with the highest PageRank per year scores (PR) Rank h-index Author Name 1 21 Knight, Kevin 2 19 Klein, Dan 2 19 Manning, Christopher D. 4 18 Marcu, Daniel 4 18 Och, Franz Josef 6 17 Church, Kenneth Ward 6 17 Collins, Michael John 6 17 Ney, Hermann Table 6: Authors with the highest h-index in AAN Rank ASP Author Name 1 2.977 Hovy, Eduard H. 2 2.989 Palmer, Martha Stone 3 3.011 Rambow, Owen 4 3.033 Marcus, Mitchell P. 151 11 3.041 Levin, Lori S. 2213 6 3.052 Isahara, Hitoshi 123 7 3.055 Flickinger, Daniel P. 8 3.071 Klavans, Judith L. 9 3.073 Radev, Dragomir R. 10 3.077 Grishman, Ralph Table 7: Authors with the smallest Average Shortest Path (ASP) length in the author collaboration network 7 Rank PR Title 1 955.73 A Stochastic Parts Program And Noun Phrase Parser For Unrestricted Text 2 820.69 Finding Clauses In Unrestricted Text By Finitary And Stochastic Methods 3 500.56 A Stochastic Approach To Parsing 4 465.52 A Statistical Approach To Machine Translation 5 345.11 Building A Large Annotated Corpus Of English: The Penn Treebank 7 318.76 The Contribution Of Parsing To Prosodic Phrasing In An Experimental 6 304.11 TTehxe tM-Taot-hSepmeeactihc sS Oysft eSmta tistical Machine Translation: Parameter 8 265.44 EAsttteimntaiotino nI ntentions And The Structure Of Discourse 9 194.06 A Maximum Entropy Approach To Natural Language Processing 10 171.25 Word-Sense Disambiguation Using Statistical Methods Table 8: Pa pers with the highest PageRank per year scores (PR) 5 Related Phrases We have also computed the related phrases for every author using the text from the papers they have authored, using the simple TF-IDF scoring scheme. Table 9 shows an example where top related words for the author Franz Josef Och are listed. Word TF-IDF 1 alignment 3060.29 2 translation 1609.64 3 bleu 1270.66 4 rouge 1131.61 5 och 1070.26 6 ney 1032.93 7 alignments 938.65 8 translations 779.36 9 prime 606.57 10 training 562.10 Table 9: Snapshot of the related words for Franz Josef Och 6 Citation Summaries The citation summary of a paper, P, is the set of sentences that appear in the literature and cite P. These sentences usually mention at least one of the cited paper’s contributions. We use AAN to extract the citation summaries of all articles, and thus the citation summary of P is a self-contained set and only includes the citing sentences that appear in AAN papers. Extraction is performed automatically using string-based heuristics by matching the citation pattern, author names and publication year within the sentences. The example in Figure 4 shows part of the citation summary extracted for Eisner's famous parsing paper4. In each of the 4 citing sentences in Figure 4 the mentioned contribution of (Eisner, 1996) is underlined. These contributions are “cubic parsing algorithm” and “bottom-up-span algorithm” and “edge factorization of trees.” This example suggests that different authors who cite a particular paper may discuss different contributions (factoids) of that paper. Figure 5 shows a snapshot of the citation summary for a paper in AAN. The first field in AAN citation summaries is the ACL id of the citing 4Eisner, J. (1996). Three new probabilistic models for dependency parsing: An exploration. In Proceedings of the 34th Annual Conference of the Association for Computational Linguistics (ACL-96), pp. 340-345. 8 paper. The second field is the number of the citation sentence. The third field represents the line number of the reference in the citing paper. In the context of DPs, this edge based factorization method was proposed by (Eisner, 1996). Eisner (1996) gave a generative model with a cubic parsing algorithm based on an edge factorization of trees. Eisner (1996) proposed an O(n3) parsing algorithm for PDG. If the parse has to be projective, Eisner's bottom-up-span algorithm (Eisner, 1996) can be used for the search. Figure 4: Sample citation summary of (Collins, 1996) 5 The citation text that we have extracted for each paper is a good resource to generate summaries of the contributions of that paper. In previous work, (Qazvinian and Radev 2008), we used citation sentences and employed a network-based clustering algorithm to summaries of individual papers and more general scientific topics, such as Dependency Parsing, and Machine Translation (Radev et al. 2009 a, b). Figure 5: Snapshot of the citation summary of (Resnik, 1999)6 7 Experiments This corpus has already been used in a variety of experiments (Qazvinian and Radev 2008; Hall et al., 2008; Councill et al., 2008; Qazvinian et al., 2010). In this section, we describe some NLP tasks that can benefit from this data set. 7.1 Reference Extraction 6 Philip Resnik, 1999. “Mining The Web For Bilingual Text,” ACL’99. 9 After converting a publication's text from PDF to text format, we need to extract the references to build the citation graph. Up till the 2008 release of AAN, we did this process manually. Figure 6 shows a reference string in the text format consisting of 5 references spanning multiple lines. The task is to split the reference string into individual references. Till now, this process has been done manually and we have processed 155,858 citations of which 61,527 citations are within AAN. This data set has already been used for the development of a reference extraction tool, ParsCit (Councill et al., 2008). They have trained a Conditional Random Field (CRF) to classify each token as “Author” or “Venue” or “Paper Title”, etc in a reference string using manually annotated reference strings as training data. 7.2 Paraphrase Acquisition Previously, we showed in (Qazvinian and Radev, 2008) that different citations to the same paper they discuss various contributions of the cited paper. Moreover we discussed in (Qazvinian and Radev, 2011) that the number of factoids (contributions) show asymptotic behavior when the number of citations grow (i.e., the number of contributions of a paper is limited). Therefore, intuitively multiple citations to the same paper may refer to the same contributions of that paper. Since these sentences are written by different authors, they often use different wording to describe the cited factoid. This enables us to use the set of citing sentence pairs that cover the same factoids to create data sets for paraphrase extraction. For example, the sentences below both cite (Turney, 2002) and highlight the same aspect of Turney’s work using slightly different wordings. Therefore, this sentence pair can be considered paraphrases of each other. In (Turney, 2002), an unsupervised learning algorithm was proposed to classify reviews as recommended or not recommended by averaging sentiment annotation of phrases in reviews that contain adjectives or adverbs. For example, Turney (2002) proposes a method to classify reviews as recommended/not recommended, based on the average semantic orientation of the review. References David Chiang and Tatjana Scheffler. 2008. Flexible com- position and delayed tree-locality. In The Ninth Inter- national Workshop on Tree Adjoining Grammars and Related Formalisms (TAG+9). Aravind K. Joshi and Yves Schabes. 1997. Tree- adjoining grammars. In G. Rozenberg and A. Salo- maa, editors, Handbook of Formal Languages, pages 69â124. Springer. 99 Laura Kallmeyer and Maribel Romero. 2004. LTAG semantics with semantic unification. In Proceedings of the 7th International Workshop on Tree-Adjoining Grammars and Related Formalisms (TAG+7), pages 155â162, Vancouver, May. Laura Kallmeyer. 2007. A declarative characterization of different types of multicomponent tree adjoining grammars. In Andreas Witt Georg Rehm and Lothar Lemnitzer, editors, Datenstrukturen f¨ur linguistische Ressourcen und ihre Anwendungen, pages 111â120. T. Kasami. 1965. An efficient recognition and syntax algorithm for context-free languages. Technical Re- port AF-CRL-65-758, Air Force Cambridge Research Laboratory, Bedford, MA Figure 6: Sample reference string showing multiple references split over multiple lines. 10