Beyond Archimedes: the history and future of the Arboreal software Mark J. Schiefsky March 9, 2015 This paper describes the history and rationale of the development of Arboreal, a software application originally designed in the course of the Archimedes Project to enable the reading and analysis of structured XML documents. It also provides an update on current development of the software and sketches some directions for further work. Peter Damerow inspired and contributed to the development of this software at every stage from its inception until his death, and its existence would be unthinkable without him. I hope that this history will illuminate an important dimension of his work and convey the essence of his compelling vision for the place of information technology in humanistic scholarship. Above all, I hope that my description of the present state of this software and its future development will demonstrate that this vision is still very much alive today. 1. History and Rationale The Archimedes Project was initiated by Peter Damerow and Jürgen Renn at the Max Planck Institute for the History of Science (MPIWG) in the late 1990s. It was conceived as the digital component of a major research project of Department I of the Institute on the long-term development of mechanical knowledge from the ancient world up to the early modern period. The goal of the project was to exploit the potential of emerging digital methods to study the content of mechanical knowledge and to disseminate the results of this research. Earlier work at the Perseus Project of Tufts University had demonstrated the great power of new tools for linguistic analysis, such as morphological analyzers and digitized dictionaries, for the creation of a new kind of online environment for the reading of ancient Greek and Latin texts. The possibility of extending this approach to the history of science was an exciting one, and very much called for by the large volume of textual sources involved in the study of a discipline like mechanics. More specifically, Archimedes was motivated by the need to represent the conceptual structure of mechanical thinking – relations between concepts such as “force”, “weight”, and “motion”, such as the notion that “motion implies force” that is ubiquitous in early physical thought. To achieve a digital representation of the conceptual content of mechanical texts at this level of detail was a challenging task that went well beyond anything then existing in the internet. In 1999 I came to the MPIWG as a posdtoc to work on the Archimedes Project, with a mandate to introduce the technology developed by the Perseus Project to the research environment of the Institute. This was a year of intellectual joys for me, not least because I had the remarkable privilege of working closely with Peter Damerow on a daily basis. We discussed everything from the origin of writing and mathematics in the 1 third millenium BCE to markup practices for electronic texts and the history and culture of contemporary Berlin. My own scholarly perspective as a classical philologist and historian of science was immensely enriched, in ways that have continued to shape my intellectual outlook to this day. One topic that was of constant concern in our daily work during this period was the design of electronic working environments that could meet the demands of the Archimedes Project for the detailed representation of the content of mechanical knowledge. I quickly became familiar with the wide range of ingenious prototypes that Peter had created for the Macintosh using FileMaker. These were based on a simple idea: a text was split up into sentences which were then loaded into a FileMaker database, in which each sentence was an individual record. This strategy made it possible to browse through the text sentence-by-sentence, to track the terminology in which ideas were expressed in individual sentences, and to add translations or comments on sentences in a systematic manner. FileMaker’s powerful indexing and viewing functions made it possible to view clusters of sentences together and to see connections between the data that were not otherwise apparent. And all of this could be achieved without any knowledge of formal programming languages. Even though the technological basis was quirky and idiosyncratic, it was clear that a powerful vision lay behind it. The use of technology was motivated by scholarly questions and tailored to scholarly aims. Peter consistently emphasized the importance of dynamic interactivity of the computing environment and human scholarly input, and the need to keep any technological solution as simple as possible. And of course the overall goal was to create resources that would be freely available online without any restrictions on access due to copyright considerations. During the year that I spent at the MPIWG we succeeded in incorporating the results of the Perseus morphological analysis software and online dictionaries into these working environments. But despite the ingenuity of Peter’s FileMaker prototypes, it was clear to all of us involved in the project that Archimedes demanded more powerful and robust tools. One of Peter’s early decisions had been that the project texts would be tagged using the XML markup language, which was still relatively new at that time. This was exactly the right call, given that XML is the now the de facto standard for text markup; at the time, however, there was very little user-friendly software available for working on XML texts, so the decision to use XML was something of a leap into the unknown. By the end of 1999, when I returned to Harvard as an assistant professor of Classics, we had in hand the first set of digitized texts making up the Archimedes corpus: a set of early modern writings on mechanics in Latin and Italian by authors such as Guidobaldo del Monte, Tartaglia, and Galileo. Correcting and tagging these in a simple XML format took up the lion’s share of the project’s efforts in the subsequent year. In 2000 we also secured three years of funding for Archimedes from the Deutsche Forschungsgemeinschaft and the National Science Foundation; in 2001 Malcolm Hyman, a brilliant young linguist and classicist, joined the project as a postdoc. At this point Malcom convinced me that it was time to bite the bullet and create new software from scratch that would address the basic technical challenges of making it possible to work with XML texts in the ways required by the Archimedes Project. Thus Arboreal came to be. 2 I will now describe the basic features of this software in the form that it reached during the Archimedes Project; for convenience we may think of this as “Arboreal 1.0”.1 Arboreal is conceived on the analogy of a traditional web browser, albeit one that works on XML rather than HTML texts. But one immediate difference is that when the user opens an XML document in Arboreal, instead of a single window we see two panes, one depicting the XML tree structure on the left and another content pane on the right. The user can navigate through the document by selecting nodes in the tree pane. As elements in the tree are selected in the tree pane they are rendered in the content pane. Arboreal allows for very complex XML structures (e.g. a text with many deletions and supplements indicated in the markup) to be rendered in whatever way is suitable for the application in question; the display of XML tags can be toggled on or off at will. Thus the XML markup is brought to life in a way that makes it accessible to the user, while the underlying format of the document is unchanged. Clicking on any word in the content pane reveals a pull-down menu that provides access to morphological and dictionary information. This information is generated on the Harvard Archimedes server (http:// archimedes.fas.harvard.edu), on which we have implemented a unified frontend to various backend morphological analyzers (http://archimedes.fas.harvard.edu/donatus). Texts can contain as many languages as are supported by the software on the server, as long as these are tagged in the XML markup, and only a single server request is needed to generate the complete set of morphological analyses for all languages in a document. Searching can be carried out using lexical forms and regular expressions; in addition, search results can be used to navigate through the document tree. Arboreal allows for the annotation of individual words or groups of words as instances of terms, which can be manipulated and visualized in a special term editor. Terminology annotations can be saved as XML documents and subjected to further analysis. Arboreal also allows for the study of different texts in parallel to one another, making it possible to carry out systematic comparisons between texts and translations, and to create and edit translations. Finally, Arboreal enables the creation of XML content of various types. Arbitrary XSLT scripts can be applied to the text currently loaded in the main window; moreover, the terminology, morphology, and matching files that underlie the program’s other functions are also in XML format.2 The development of this software resulted from years of intensive effort and collaboration between many individuals with different intellectual approaches and technical competences. It would have been impossible without the programming genius and intellectual vision of Malcolm Hyman, who had the original insight that such software was possible, the ability to bring it into being, and the determination to do so. The initial stage of development took place at Harvard University, with close collaboration and consultation with Peter Damerow, Jürgen Renn, and other members 1 In terms of the actual numbering, the last version on which Malcolm Hyman worked was designated “Arboreal 5.16”. 2 For a longer description of the functionality of “Arboreal 1.0” see Schiefsky 2007. This description omits a number of functions provided by Arboreal that are not directly relevant to my argument here. In particular, I should note that the program is also designed to interact smoothly with image repositories and tools for image annotation, to enable the analysis of visual as well as textual content. 3 of Department I of the MPIWG. There were many trips between Boston and Berlin in those years. When Malcolm took up a position at the Institute in 2005, he was able to work even more closely with the colleagues there. During the period between 2005 and Malcolm’s tragic death in 2009, Arboreal was used intensively in creating a translation of a Chinese text on mechanics at the MPIWG and for studying the terminology and deductive structures of Euclid’s Elements (Schiefsky 2007). While Arboreal was designed with the needs of the Archimedes Project in mind, it turned out to be a highly general tool. In fact, Arboreal embodies several key features that challenged – and continue to challenge – the basic way in which the Internet functions as a medium for the creation and representation of knowledge. First, Arboreal moves beyond browsing; it enables the creation of richly structured digital content in providing facilities for term annotation and the generation of new XML documents from existing texts. Second, it also moves beyond the search-dominated paradigm of the current Internet. Although Arboreal has powerful capabilities in this regard, searching is not conceived as an end in itself; rather, the program is designed in order to make search results the starting point for further analysis. Third, Arboreal provides a robust platform for multilingual computing, and a model for providing the user with integrated access to diverse linguistic resources. Finally, Arboreal provides for a distinctive kind of interactivity, since the files it generates are themselves XML files that can be subjected to further analysis by the software itself. In the most general terms, Arboreal is a tool that contributes to the long-term project of making the Internet into what has been called an “epistemic web” – a domain in which unstructured, unanalyzed data is transformed into structured information and knowledge. In this vision, which descends from Peter Damerow’s insights but represents the fruit of the entire course of development described in this history, the Internet becomes a vehicle for transmitting knowledge, understanding, and, we may hope, also wisdom.3 Since the formal conclusion of the Archimedes Project in 2004, a great deal has changed in the universe of the Internet. Social networks have exploded, enabling an exponential increase in user-generated content. Google Translate now provides a crude translation for all the world’s main languages and offers the eventual possibility of a linguistically transparent Web. Natural language processing applications have greatly advanced beyond the stage of context-free morphological analysis that was still the state of the art at the time that Archimedes began. And there are vastly more digital corpora now available for research. The “big data” approach – using statistical methods to analyze and extract meaning from huge corpora – has enabled entirely new questions to be asked and new approaches to be taken to traditional questions. Yet in many ways the ideal of the epistemic web seems more remote than ever. In the area of digital scholarship, researchers seem trapped in the “browsing” and “search” paradigms. The massive success of Google has contributed to an tendency to reduce analysis to searchability; the problem of giving the user informed assistance in what to search for has been neglected. Social networks offer the user the possibility to create 3 For the concept of the epistemic web, see Hyman and Renn. 4 content, but this tends to be limited to unstructured text or images. There is still an urgent need, then, for software that embodies the distinctive features of Arboreal as outlined above. In some ways the current environment renders the need for such software all the more acute. Recent challenges posed by the “big data” movement suggest that statistical approaches to large corpora render the need for traditional scholarly (and other kinds) of analysis irrelevant. Indeed it has recently been argued that big data and the associated modes of analysis are on the verge of eliminating the need for models and explanatory hypotheses in scholarship and in science (Anderson 2009). But while there may be many ways of creating knowledge out of unstructured information, there is still a crucial place for human input – both in determining the questions that should be posed to automatic systems and in interpreting the results they produce. “Big data” approaches have their place in the sciences and – increasingly – in the humanities, but it is a mistake to think that they can answer all interesting questions. Informed input is needed not just to interpret results, but also to pose the questions. What is particularly lacking in the arguments in support of the “big data” approach is a sense of the power of interactivity – of the way in which automatically generated results only gain meaning from informed questions, and are shaped by them. Reflecting on tools like Arboreal can help us to see how technology can be used to foster these humanistic ends. Indeed I would argue that the potential of the technology itself will not be fully realized unless such interactivity is kept front and center in the development process. To return to history. The tragic death of Malcolm Hyman in 2009 dealt a severe blow to Arboreal’s development. With Peter Damerow’s passing in 2010, we lost another of the program’s original sources of inspiration. But I am delighted to announce that development of the code has recently resumed with the assistance of two French colleagues, Professor Said-Esteban Belmehdi of the Université de Lille and his graduate student Julien Razanajao. Working in close collaboration with me, Belmehdi and Razanajao have updated the code to work with contemporary versions of Java and succeeded in integrating some powerful new features.4 In the remainder of this paper I shall describe these features, give some examples of their use, and outline some of the principal goals that remain for further work. 2. “Arboreal 2.0”: networks and visualization The principal innovation in this new and improved version of Arboreal is the ability to generate networks and to visualize them using the Gephi software library (http:// 4 It is a pleasure to acknowledge the programming skills and dedication of my two French collaborators, who worked tirelessly, successfully, and without any special institutional or financial support to decipher and extend Malcolm Hyman’s brilliant but very complex Java code. Without their work none of the analysis I describe in the rest of this paper would have been possible, and the future of Arboreal would be in serious doubt. 5 gephi.github.io). This provides a standard format (.gexf) for encoding graphs in XML, as well as a powerful set of algorithms and rendering tools for viewing and analyzing graph data. The current version of Arboreal makes it possible to generate and render two kinds of graphs: graphs of the distribution of morphological variants of a given word across sections of a text, and semantic networks expressing the relations between different terms. I shall illustrate these with examples drawn from the history of mechanics and the texts of the Archimedes project corpus. a) Morphological graphs, as defined above, have their principal use in the exploration of the connections between the language and formal structure of texts. Consider the example of the very first ancient Greek text dedicated to theoretical mechanics, the Mechanica Problemata or Mechanical Problems attributed to Aristotle. This text contains a long introduction on the wondrous properties of the circle and circular motion, which in the author’s view underlie the explanation of all mechanical movements. The introduction is followed by a set of 35 “problems” which are posed using a standard formulation, then answered. Thus problem 1 asks why it is that larger balances are (allegedly) more accurate than smaller, and the author goes on to give an explanation of this fact in terms of circular motion. The text states that the balance is explained in terms of the circle, the lever in terms of the balance, and all other mechancial movements in terms of the lever. But in fact the author often appeals to circular motion directly rather than the lever. We can see this by considering the following graph of the occurrences of μοχλός (“lever”) throughout the text. The large nodes represent the text itself (on the right) and the lemmatized form of the term (on the right); the lemmatized form is joined to the different morphological variants, which are themselves linked to sections (i.e. “problems”) in the text. From this graph we can see at once that the term μοχλός has a fairly wide distribution across the text although the problems in which it does not occur also stand out clearly (these are the nodes arranged concentrically around the “Problemata Mechanica” root node). 6 A more complex example is provided by Guidobaldo del Monte’s Mechanicorum Liber, a key text in early modern mechanics that has a clear formal structure articulated on the Euclidean model. After a preface on the nature and importance of mechanics, Guidobaldo sets out a number of basic assumptions or postulates and goes on in six main sections to treat of the balance as well as the five “mechanical powers” familiar from Greek antiquity: the lever, the wheel and axle, the pulley, the wedge, and the screw. Each of the six sections is clearly divided into propositions, lemmas, and corollaries, which are tagged in the XML markup (just as the different “problems” are in the Aristotelian text mentioned above). If we consider the graph of the distribution of the term gravitas or “heaviness”, we find a strikingly high frequency in Proposition IV of Book I, “On the balance” (De libra; the thickness of the edge indicates frequency). Upon inspection of the text, we see that this is because of a large number of instances of the phrase “center of gravity” (centrum gravitatis) in this proposition. The frequency of this term is due to the fact that in Proposition IV, Guidobaldo is arguing against thinkers 7 who claimed, on the basis of the medieval theory of “positional heaviness” (gravitas secundum situm) that a balance displaced from equilibrium will not return to equilibrium, because the weight that is elevated has less “positional heaviness” than the one that is depressed below the original level. To this medieval theory Guidobaldo opposes the Archimedean theory of center of gravity, arguing that an object suspended from its center of gravity will not move no matter what its orientation is to the horizontal. This “equilibrium controversy” - as Peter Damerow and Jürgen Renn have called it (Damerow and Renn 2012) – was generated by the tension between two different traditions of mechanical knowledge: one centered on the medieval concept of “positional heaviness” and another rooted in the Archimedean study of centers of gravity. The relevance of the medieval concept to Proposition IV of book I is clearly shown in the morphological graph of the distribution of situs (below). Thus, these graphs make it possible to correlate language and formal structure in a precise and visually informative way. 8 b) As Malcolm Hyman argued, semantic networks provide a powerful tool for studying conceptual development in the history of science (Hyman 2006). In such networks, individual terms in the texts are represented as nodes in a network, and the edges linking them express the strength of their association with other terms. Very crudely speaking, terms which frequently occur in close proximity to one another have a higher association than other terms. Thus, to take an example from Hyman’s paper, “force” in an article on theoretical physics will have different associations in an article on theoretical physics (e.g. “potential”, “field”, “electromagnetic”) than in an article on the behavior of police towards African Americans in the American South (e.g. “violence”, “abuse”). The meaning of a term within a particular text is constituted at least in part by its relations to other terms; “force” in the sense of physical strength exerted by humans is a different concept from “force” in the context of theoretical physics, although the term used is the same. A semantic network is a kind of model of the conceptual structure of a text, in which terms serve as proxies for concepts. Such an approach is particularly useful in studying the semantic field of a concept such as force in texts which span a range of centuries and languages. In the early history of mechanics force is typically seen as a cause of motion; but the language of force can also be used of effects rather than causes (e.g. “a forceful blow”). There is also the distinction between physical force and a more generalized power or capacity to affect, as well as that between a force that acts instantaneously and one that persists. The Greek term δύναμις, for example, tends to refer to a general capacity or faculty to affect, rather than something that causes motion in particular; the best translation is 9 often “power”, which also expresses the tendency of a δύναμις to persist through the duration of its activity. The term ἰσχύς is glossed by Bonitz in his Index Aristotelicus as both “motive force” (vis motrix) and “bodily strength” (robur corporis); yet he also notes that it is typically used as a synonym for δύναμις.5 A third Greek term, βία, has connotations of violence as well as physical strength, and is particularly important for its use to express the distinction between “natural” and “violent” motion in the Aristotelian tradition. Thus, Greek presents us with a set of terms that overlap in their meanings, though each has distinctive nuances. Similar remarks might be made for the Latin (potentia, virtus, and vis) and Italian (forza, potenza/possanza, and virtù) terminology on the basis of appropriate lexica. While these remarks on general usage are important, understanding the terminology of force in a particular text requires taking account of the way in which the term is actually used. It is this that the semantic network method enables us to investigate in a precise, rigorous and repeatable manner. In this perspective, the meaning of a term in a text is constituted by its place in the semantic space – a web of connections that model the text’s conceptual structure. For the generation of semantic networks the current version of Arboreal implements the Semantic Vectors package released in Google Code (https://code.google.com/p/ semanticvectors/). I will simply sketch the general idea here, referring to the online documentation for the details of this particular implementation. The basic idea is to represent the distribution of terms in a document via a term-document matrix. Thus, for example, the rows of the matrix may correspond to segments of text while columns correspond to particular terms; in this simple model, the value of any element of the matrix (r, c) is the number of occurrences of term c in segment r. Once such a matrix is constructed we can apply statistical methods and linear algebra techniques to derive measures for the similarity between different rows (comparing segments to segments) or columns (comparing terms to terms). A key step is dimensionality reduction, in which transformations such as singular value decomposition are used to reduce the size of the matrix; the reduced matrix is then interpreted as a representation of the document in semantic space. The net effect of this is to eliminate the “noise” caused by phenomena such as synonyms: if terms A and B are both found regularly in conjunction with the same cluster of terms C, A and B will end up very close to one another in the semantic space. Another way of looking at this is that A and B express the same concept within the semantic space, which is thus a model of the text’s conceptual structure. Once the semantic space has been constructed, we determine the association of terms to one another using standard metrics such as cosine similarity. These associations become the labels of the edges linking different nodes in the graph. In analyzing a particular text, we begin by reducing morphological variants to lexical forms (thus English is and was ==> be, Latin vires and vi ==> vis) using the web services of the Harvard Archimedes server (http://archimedes.fas.harvard.edu/donatus) 5 See Bonitz s.v. ἰσχύς, available at http://archimedes.fas.harvard.edu/pollux/. 10