QACTIS Enhancements in TREC QA-2006 P. Schone, G. Ciany*, R. Cuttsb , P. McNamee=, J. Mayfield=, Tom Smith= U.S. Department of Defense Ft. George G. Meade, MD 20755-6000 ABSTRACT of past-year systems, but we believe it provides an appro- priate reflection of system performance over time. The QACTIS system has been tested in previous years at Afterdiscussionofthesystemenhancements,wecon- the TREC Question Answering Evaluations. This paper duct a post-evaluation analysis of the results from the describes new enhancements to the system specific to TREC-2006evaluation. Oursystemimprovedslightlythis TREC-2006, including basic improvements and threshold- year in terms of factoid and list answering. However, we ingexperiments,filteredandInternet-supportedpseudo-rel- experienced10%and20%relativelossesinsystemperfor- evance feedback for information retrieval, and emerging manceduerespectivelytounsupportednessandinexactness statistics-driven question-answering. For contrast, we also -- numbers which are too large to go without notice. The compareourTREC-2006systemperformancetothatofour inexactness losses seem high but hopefully such degrada- top systems from TREC-2004 and TREC-2005 applied to tion has been uniformly observed across systems. On the this year’s data. Lastly, we analyze evaluator-declared otherhand,theunsupportednessismoresuspicious. Unlike unsupportedness of factoids and nugget decisions of manyothersystems,whichusetheInternetasapre-mecha- “other”questionstounderstandmajornegativechangesin nism for finding answers and thereby have a chance of performance for these categories over last year. recalling the wrong file, our system solely draws its answersfromTRECdocumentsanddoesnotminetheWeb 1. INTRODUCTION for answers. We conducted a number of post-evaluation QACTIS (pronounced "cactus"), which breaks out to experiments. Oneoftheseattemptedtomakeadetermina- "Question-Answering for Cross-Lingual Text, Image, and tion as to whether the unsupported labels were justified. Speech,"isaresearchprotoype systembeingdevelopedby We found through this analysis no systematic biases. the U.S. Department of Defense. The goals and descrip- Along a similar vein, at TREC-2005, QACTIS’s “Other” tionsofthissystemarespecificallydescribedinpastTREC answerer received the highest score, whereas this year, it descriptions (see Schone, et al., 2004, 2005 in [1], [2]). In suffereda40%degradationinoverallscore. Weconducted this paper, though, we provide a self-contained description astudytounderstandthisdegradation. Thisevaluationalso ofmodificationsthathavebeenmadetothesystemin2006. eliminated concerns about potential assessment problems. Therewerethreemajorpointsofstudyuponwhichwecon- ducted research this year: (1) basic improvements to the 2. CY2006 SYSTEM ENHANCEMENTS general processing strategy, (2) information retrieval In 2006, there were a number of new avenues of research enhancementsasaprefilter,and(3)amovetowardintegra- on QACTIS. As mentioned earlier, these fall into three tion of more purely-statistical question answering. We maindirections. Specifically,theseinvolvedimprovements describeeachoftheseresearchavenuesinsomedetail. For to the base system, information retrieval enhancements for thesakeofdemonstratingtheseimprovements,weevaluate preselection of appropriate documents, and, lastly, a pro- ourbestsystemsfromTREC-2004andTREC-2005onthis cesstomoveawayfromsymbolicprocessingtoamoresta- year’s evaluation data as a means of comparison. This tistical system. We discuss each of these in turn. comparisondoesnotcompletelyre-createallofthenuances 2.1 General System Improvements 2.1.1. Overcoming Competition-level Holes * Dragon Development Corporation, Columbia, MD b One major modification was designed to overcome prob- Henggeler Computer Consultants, Columbia, MD = Johns Hopkins Applies Physics Laboratory, Laurel MD lems that only arise with the appearance of fresh data -- Report Documentation Page Form Approved OMB No. 0704-0188 Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. 1. REPORT DATE 3. DATES COVERED 2006 2. REPORT TYPE 00-00-2006 to 00-00-2006 4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER QACTIS Enhancements in TREC QA-2006 5b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S) 5d. PROJECT NUMBER 5e. TASK NUMBER 5f. WORK UNIT NUMBER 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION U.S. Department of Defense,Fort George G. Meade,MD,20755-6000 REPORT NUMBER 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S) 11. SPONSOR/MONITOR’S REPORT NUMBER(S) 12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release; distribution unlimited 13. SUPPLEMENTARY NOTES 14. ABSTRACT 15. SUBJECT TERMS 16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF 18. NUMBER 19a. NAME OF ABSTRACT OF PAGES RESPONSIBLE PERSON a. REPORT b. ABSTRACT c. THIS PAGE 9 unclassified unclassified unclassified Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39-18 problems which unfortunately only really occur during were unsuccessful. We reasoned that use of a more competitions. In TREC2005, we had noticed that there recently-built content extractor with such resolution were a number of questions which our parser failed to embedded could be especially beneficial. BBN was able properlyhandle;therewereotherquestionsforwhichthe togenerateoutputforusfromtheirSERIF[3]engine,and system did not know what kind of information it was we began work to incorporate this information into an seeking; there were questions that were so long that our exactnessfilter. Unfortunately,wewerenotabletomake NIL-tagger generated inadvertent false positives; and useofthisinformationpriortotheevaluation. Ultimately, lastly, there were many QACTIS-provided answers (as the only additional resolution that we could incorporate much as 20% relative) marked inexact. intothesystembythetimeofthecompetitionwastoget The first two of these were readily solved. We the base system to augment city names with their corre- ensured that the system always parsed any previously- spondingstatenameswhensuchinformationwaspresent unprocesseddocumentsandthatithandledtheseproperly in the data. atruntime. Wealsoattemptedtorequirethatanyfactoid question whose target answer form was unknown would 2.1.2. What is the Actual Information Need? at least return an entity type. Also, we sought to prevent Basedonevaluationsoverpastyears,wenotedthatQAC- non-responses for other-style questions by requiring the TISproducederroneousanswesforquestionsaboutcourt system to revisit the question in a new way if an earlier decisions, court cases, ranks, ball teams, scores, cam- stage failed to provide a response. These changes were puses, manufacturers, and, in some cases, titles of works important for yielding a more robust question answerer, ofart. Theseproblemswereduelargelytoissuesofeither and it is possible that as much as 0.5-1% of the factoid underspecificityortoprovidingahyponymforaconcept scoreimprovementsisattributabletothesefixes(particu- rather than a required instance of that concept. larly the parsing fixes). With regard to underspecificity, “teams” provide a Perhaps the most dramatic change to handling the greatexample. Ifaquestionwereoftheform“Whatteam problem of the unforeseen was to change the system’s won Super Bowl ...,” it is clear to a human that the team scoring metric to take length of question into consider- thatisbeingreferencedisanNFLfootballteam. Instead, ation. Our system attempts to provide a probability-like if “World Cup” replaced “Super Bowl,” the team should estimate of the likelihood that some answer is legitimate be a soccer team. Formerly, the system would seek out giventhequestion. Allthenon-stopwordsofthequestion anykindofteamfortheappropriateresponse--aproblem are important in the question-answering process, so of underspecificity. To avoid this problem, we encoded longerquestionswillnaturallyhavelowerestimatedprob- knowledge into the system to help it better be able to abilities than shorter questions. Nevertheless, it had pre- reach the correct level of specificity particularly with viously been our policy to use a threshold as a means of teams. Likewise, in TREC-2005, the system was pre- estimating whether an answer should be reported as a paredtoidentifytitlesofworksofart,butwhetherornot NIL. Thismeantthatlongquestionsweremorelikelyto it could do it was subject to the way the question was erroneously report NIL as their answers than short ques- posed. We tried to incorporate more generality into its tions. Through least squares fit, we determined that the ability to recognize when such a work of art was being probability scores were approximately proportional to requested. We have not removed all problems with this 0.01length_in_wordsforquestionswithatleastthreewords. underspecificity,sothiswillbecontinuedworkaswepre- Therefore, we multiplied the scores of such questions by pare toward TREC-2007. 100(length_in_words-3). Weconductedanexhaustivesearch The hyponym/instance-of issue is likewise a promi- nent problem in the system. If the system were to see a anddeterminedanoptimalnewNILthreshold(of10-12). question“Whatwasthecourtdecisionin...”or“whatwas In our developments, this augmentation to the weight thescore...”thesystemwouldthinkthatitwaslookingfor seemedtohaveonlylimitedeffectonscorebutprevented some hyponym of “court decision” and “score” rather accidental discarding of legitimate answers. than a particular verdict (guilty, innocent) or a numeric Thelastchallengewastoovercomethelargenumber value,respectively. Weimplementedanumberofspecial- ofanswersthatQACTISproducedthatwerebeingidenti- ized functions to tackle rarer-occuring questions such as fied as inexact in past years. Our system might identify these and ensure that the appropriate type of information the appropriate last name of a person that should be was provided to the user. reportedasananswer,oritmightidentifythecitywithout thestate. Wehadpreviouslybuiltnameandanaphorares- 2.1.3. Missing Information Needs: Auto-hypernymy olutionintooursystemwhichwehadnotbeenusing,and There are related problems to looking for either hyp- we experimented with various settings of these compo- onyms or instances of classes which are due to lack of nentstoseeifwemightgetsomeadditionalgains,butwe worldknowledgeinsomeareas. Forsystemsthatusethe Internet to provide them potential answers, they get results for factoid answering and Variant1 actually around the problem of missing world information. Our yieldedsignificantlossesinlistperformance. Needlessto basesystemis,forthemostpart,self-containedandwedo say,wechosetonotselecttheseupdateddatasourcesfor not currently make direct use of the Web. Therefore, we use in our actual evaluation systems. This will be a sub- need to ingest resources to support our question-answer- ject of study for the future. ing. Inpastyears,wehavemadeuseofWordNet[4]anda knowledge source we had previously developed called Table 1: Wikipedia Ingestion SemanticForests[5],pluswehadtargetedspecificcatego- Baseline Variant1 Variant2 riesofquestionsandderivedlargeinventoriesofpotential QA Set concepts under those categories through the use of Wiki- #1 List #1 List #1 List pedia[6]. Thisyear,however,wetriedtogrowourworld 201-700 165 164 knowledgetomuchmorethanhundredsofcategoriesand 894-1393 116 113 instead try to (a) ingest much or all of Wikipedia’s taxo- 1394-1893 133 133 133 nomic structure, and (b) automatically induce taxonomic 1894-2393 154 .165 152 .137 152 .169 structure on the fly. 1.1-65.5 72 .197 71 .130 71 .188 Forthefirsteffort,wedownloadedtheentireEnglish 66.1-140.5 121 .163 124 .109 121 .157 component of Wikipedia and distilled out all of its lists. Thenwedevelopedcodethatcouldturnthoselistsintoa 2.1.4. Longer Term Attempts structure akin to the taxonomic structure required for ThereweretwootherareasofresearchonQACTISwhich ingestion by our system. Time constraints limited our weundertookwithintentionsofincorporatingbythetime ability to do this in a flawless fashion (and revisiting this of the evaluation but which required more effort than issue is certainly in order for the future). expected to make ready on time. These are mentioned In addition to the use of this Wiki-generated taxo- only briefly for the sake of completeness. One of these nomic structure, we also experimented with hypernym areas dealt with morphological information and the other inductionasameansoffindingstillmoreinformation. In with multitiered processing. 2004,Snow,etal[7] describedaninterestingmethodfor With morphology, our system attempts to crudely hypernym induction that was based on supervised generatedirectedconflationsetsforallofthewordsofthe machinelearningusingfeaturesderivedfromdependency question and for the documents which hopefully contain parses. Oncetrained,thelearnercanbeappliedtounan- theanswers. Anumberofquestionshavebeenanswered notated corpora to identify new hypenym pairs. Snow incorrectly due to incorrect morphological information provideduswithhiscode,andwebeganinvestigatingthis related often to word sense. We therefore began what technologyandextendingitsscalabilityforapplicationto turnedouttobeasignificantefforttoconvertthewaythe our question-answering problem. system did morphological processing to one that would We used these two new datasets to augment the data also make use of the part of speech in its stemming pro- that we previously had (and which we had been growing cess. We hope that this information will strengthen the by hand throughout the year) to see if these approaches system at a later point even though it is currently not would yield system improvements. We created two vari- embedded in the processing. ant taxonomic dictionaries of different sizes and plugged Another effort which we were not able to finish themintotheexistingsystem. InTable1,weillustratethe attemptedtoconvertQACTIS’scurrentprocessfromone results of these variants as compared with our baseline that fuses all forms of annotation (named entity, parsing, system that makes use of largely hand-assembled data. etc.)fromasinglestreamintoonewhereeachstreamcan (Notethatthescoreslistedarethenumberofnumberone be accessed independently. This would allow the system answers (#1) and the F-score for lists on past TREC sets to derive an answer from one stream even if another withquestionidentifiersinthespecifiedranges. Inthese stream would have yielded some other interpretation. evaluations, also, the system judgments are automatic, Thisnotionhaspotentiallyverypositivegains,butweare and the judgments do not count off for unsupportedness currently at some distance away from knowing its long- nor for temporal incorrectness. Moreover, the evaluation term benefits. counts as correct any outputs which have exact answers Althoughthefull-blownmorphologyrevisionwasnot embeddedtherein,butwhichmaynottrulybeexact. For incorporated by the time of evaluation, we were able to example, “Coach Harold Solomon” would be scored as incorporate a weaker effort regarding morphology with correct even if the exact form should only be “Harold regard to list-processing. The QACTIS system has been Solomon.”) The table illustrates a disappointing result: developed primarily with a focus on factoid answering, that despite this interesting effort, the taxonomy-growing but morphological structure of questions was not well approach as it currently stands yielded slightly negative addressed for tackling lists. We therefore did work on beefingthesystemupintermsofitslist-handlingcapabil- IRsystemforthe2005datasetwasmuchbetter--thetop ityand,inthelongrun,thiseffortprovedtobequiteuse- 10 documents contained the correct answer 70% of the ful in that our list-answering capability on the whole time while the top 30 documents contained the correct improved substantially. answer 80% of the time. ApseudoIRsetwasbuiltforthe1894seriesusingthe 2.2 Retrieving Documents answersetprovidedbyTREC-wereferredtothissetasa ’perfectIR’set. Thenumberofcorrectanswersprovided Like many other question-answering systems, QACTIS bythebasesystemwhenthisdatasetwasusedwas~60% beginsitsquestion-answeringphasebyfirstattemptingto -- an absolute gain of 20% just by removing irrelevant find documents on the same subject We had used the documents. Lemur system [8], version 2.2, since TREC13, and have An additional step was added to the overall system found its results to be satisfactory. In fact, at TREC14, that attempted to filter the IR using the named entities usingthissystemoutoftheboxyieldedoneofthetopIR present in the question. This list also includes dates, systems. We experimented with new versions of Lemur, titles,andanythinginquotes. Thisprocessdidprovidean but were not able to get any better results for QA. increase in scores as long as it was not overly aggressive Even still, when we look at the results of our ques- in filtering out too many documents. tion-answerer,weseethatithasaless-than-perfectupper Further attempts were made to include multi-word bound due to limitations in information retrieval. If we terms and low-frequency words (words in the question could enhance our ability to identify appropriate docu- whichhadalowerfrequencyofoccurrenceintheoverall ments, we would likely have a higher performance and a corpus) as filter terms, but there was not enough time to higher upper bound on performance. We set out to adequately analyze the effect. Additional parameters improve our ability to preselect documents which would such as how many of the top 1000 documents should we hopefullycontainthedesiredanswers. Weexperimented examine,howmanydocumentsshouldweretainandhow withtwoapproaches. The firstof thesewasanapproach manydocumentsfromtheoriginalIRshouldwekeepby which identified key phrases from the question and tried default also had to be factored in to the result. to ensure that the returned documents actually contained By the time of the TREC-2006 evaluation, it was thosephrases. Wewillcallthisprocessphrase-basedfil- determined that no more than the top 50 documents tering of information retrieval. The second process used shouldbeexamined. Therewasnodifferenceinoursys- the Web as a mechanism for pseudo-relevance feedback. tem in examining 50 or 75 documents. 100 documents We discuss each of these techniques. degradedtheoverallsystemperformance. Therewasalso asignificantboostinlookingat50documentsasopposed 2.2.1 Phrase-based IR Filtering to just 30. Also, because of list questions, it was deter- By the time we had competed our system in the TREC- mined arbitrarily that at least 10 documents should be 2005competition,thebasesystemasappliedtooneofthe retained. Since our IR system showed the top 5 docs for older TREC collections (the 1894 question series) was each question to be relevant about 60% of the time, we gettingmeanreciprocalranksofabout40%. Uponexam- decidedtokeepthetop5documentsasamatterofcourse. inationofthedocumentsreturnedbytheIRcomponent,it Weranourphrase-basedfilteringonallofthecollec- was discovered that a large number of irrelevant docu- tions at our disposal on the day before the TREC-2006 ments were being returned. One reason for this was that evaluation. Table 2 illustrates these results (whose scor- peoples’ names, such as ’Virginia Woolf’, were broken ing follows the paradigm mentioned in Table 1). As can into two separate query terms. The resultant documents be seen, this approach affords small (2.6% relative) but returned some that contained ’Virginia Woolf’ as well as positive improvements in the overall system performance. some that related to a ’Bob Woolf’ who lived in ’Vir- ginia’. A question pertaining to ’Ozzy Osborne’ also returned a document containing a reference to a woman Table 2: Filtered IR DevSet Improvements who owned a dog named Ozzy which bit a ’Mrs. Osborne’ on the wrist. Baseline w/ Filtered IR Diff in QA Set Furtheranalysis ofthe IRset showed thatthe top10 #1 Mrr List #1s documentsforeachquestioncontainedthecorrectanswer 201-700 165 .432 172 .446 +7 55%ofthetime;thetop30documentscontainedthecor- 894-1393 116 .351 118 .348 +2 rect answer 67% of the time. (The numbers were deter- 1394-1893 133 .393 138 .402 +5 minedbycomparingthedocumentlistreturnedbytheIR 1894-2393 154 .431 .165 158 .444 .176 +4 system to the list of correct documents for each question 1.1-65.5 72 .395 .197 71 .402 .197 -1 as provided by the TREC competition committee.) The 66.1-140.5 121 .445 .163 127 .456 .162 +6 2.2.2. Google-Enhanced Information Retrieval recentyearsofTREC,ascomparedtothebaseline,theS2 Thesecondapproachtoprefilteringwasamultisteptech- system in preliminary tests yielded a 6.4% relative niquethattookadvantageoftheInternetwithoutactually improvement in factoid performance and a 4.1% relative trying to mine answers from it. This is a process which improvement in list performance. apparentlyhasbeenusedbyotherTREC-QAparticipants in past years. 2.2.3. A Word about Coupled Retrieval To improve our document retrieval phase, we used One last experiment we tried was the coupling of these Google to select augmentation terms for each question. two prefilters. The hope was that if each could give an Each question in the test set was converted to a Google improvement, then perhaps in combination the improve- query and submitted to Google using the Google API. ment would increase. Unfortunately, this was not the The top 80 unique snippets returned for each question case. It turns out that the approaches are somewhat at wereusedastheaugmentationcollection. Givenaques- odds with each other. The phrase-filtering approach tion, we counted the number of times each snippet word attemptstoensurethatonlydocumentsthatcontainsome co-occurredwithaquestionwordinthesnippetsentence. or all of the important question phrases should be Thesesumsweremultipliedbytheinversedocumentfre- retained, while the Google-assisted approach attempts to quencyoftheterm;documentfrequencieswerecalculated lookfordocumentsthatmighthaveterminologythatwas fromtheAQUAINTcollection. Theresultingscoreswere notintheoriginalquestion. Ifoneappliesaphrase-based usedtoranktheco-occurringterms. Thetopeightterms filtering system to documents that have been obtained by were selected as augmentation terms, and were added to the Google-assisted process, the likelihood is that even theoriginalquerywithweight1/3. Theresultingqueries fewerofthedocumentsthanbeforewillactuallyhavethe were then used for document retrieval. appropriateterminology. Wetriedseveralothervariations In selecting these parameters, we were faced with on this theme after the actual TREC submission, but no many parameter choices for definition of co-occurrence, combination yielded an improvement in overall perfor- term weighting, etc. With over a thousand combinations mance. to choose from, it was not practical to run full question- answeringtestsoneachonetoselectthebest.Instead,we 2.3. Beginning to Incorporate Statistical QA used a proxy for question answering performance: the In the past few TREC evaluations, there has been an numberofwordsthatoccurinquestionanswersthatwere emergenceofstatisticalQAsystemswhichhavetheprop- selectedasexpansiontermsbythemethod. Weminedthe erty that they learn relations between the posed question, answer patterns from past TREC QA tracks for words. theanswerpassage,andtheactualanswer. Then,whena Each time a method selected one of these words as an userposesafuturequestion,variousanswerpassagesare expansion term for the training collection, it was given a evaluated using statistical or machine-learning processes point. WeusedthehighestscoringmethodinourTREC- to determine how likely they are to contain a needed 2006 run. answer. As a final step, the system must distill out the answerfromtheexistingpassage. Statisticallearnersare Table 3: Google-Enhanced Improvements particularly appealing in that they hold potential capabil- Google-Enhanced ity of developing language-independent QA. For such QA Set BL capability, one need only provide question-answer pairs TF LS LT S S2 for training. #1 154 156 130 156 152 157 Webeganin2005todevelopastatisticalQAsystem. 1894-2393 Mrr .431 .455 .377 .455 .442 .462 Theinfrastructureforthissystemwasinplacebythetime List .165 .191 .148 .192 .179 .191 of the TREC-2006 evaluation and the system had begun #1 72 76 80 81 76 79 to be taught how to automatically answer a limited num- 1.1-65.5 Mrr .395 .434 .405 .443 .434 ber of questions, so we thought we would couple it with List .191 .176 .179 .182 .189 .195 theexistingsystemandallowittoanswerthosefewques- #1 121 126 116 123 122 130 tionstructuresthatitwasequippedtoaddress. Sincethis 66.1-140.5 Mrr .445 .464 .473 .458 .463 .473 systemisnewandemerging,weprovideabitmoreinfor- List .163 .158 .179 .159 .163 .168 mation about the process and ways that we attempted to exploit the process during 2006. Inourdevelopmentswiththisprocess,wewerequite excitedaboutthegainswewereseeingwithit. Weexper- 2.3.1. From Document Selection to Passage Selection imented with five different configurations of this process Thefirststepoftheprocessofdevelopingastatisticalsys- and one process (S2) yielded particularly successful tem was to move from mere document selection to some results. In Table 3, as can be seen from the three most form of passage selection. The first 45 documents potentially language-independent process for answering reported from the Lemur IR system were screened to the questions. Using a copy of Wikipedia, we first used identifysentences(andsometimessurroundingsentences) named-entity matching and part of speech tagging to see which reflected the information from the important noun if we could draw out an answer directly from the Wiki wordsofthequestion. Thefirst80sentencesthatsatisfied pages. Barring this, we looked for redundant but nor- the criteria were retained and the sentence selection pro- mallyrareinformationfromtheSVMsentencesand,ifit cess was terminated. The 45-document and 80-sentence existed, this information was returned as the answer. limits were determined to be empirically optimal thresh- old. 3. SYSTEM EVALUATIONS The next goal was to order these sentences by their 3.1. Description of Results potential for answering the question. The reordering In TREC-2006, we submitted three runs from among the component was based on support vector machines various configurations at our disposal. All of our runs (SVMs). Thereorderingeffortwastreatedasatwo-way used the same “Other” processing as in TREC-2005 classification problem -- a customary domain for SVMs. (exceptthatthissystemwasslightlymorerobusttofailure The classifier was based on 26-dimensional feature vec- than last year). Also, in each situation, the system tors that were drawn from the data. Examples of these reported the top 20 answers from our factoid system as features were: (a) reporting 1.0 if the direct object from the“list”response. Intermsoffactoids,thefirstofthese thequestionwasintheputativeanswersentenceand0.25 runs made use of our base engine but its information otherwise; (b) reporting 1.0 if the direct object from the retrievalphasewasprefilteredusingphrase-basedfiltering question was missing but a WordNet synonym is in the asmentionedbefore. Thesecondsystemreplacedphrase- putative answer sentence, and 0.25 otherwise; (c) report- based filtering with our Google-enhanced information ing the ratio of question words to putative answer sen- retrieval efforts. The third system was the same as the tence words; and so forth. The classifier was then second but whenever the statistical system was deemed presentedwithpositiveexamplesoffeaturevectorsdrawn itself able to answer the question, it would supplant the from actual answer sentences of past-year TRECs and it original answer with its own. Since the statistical system wasalsopresentedwithacomparablenumberofnegative isinitsinfancy,therewereveryfewanswersthatitactu- examples drawn from bogus sentences. ally supplanted. Fromthesetrainingexamples,thesystemwastaught The results of these runs are detailed in Table 4. with quite good accuracy to learn the difference between Under “Factoid,” the number of correct answers is listed good question-answering sentences and poor ones. In andisfollowedbythetriple(unsupported,inexact,tempo- fact, for questions where the initial IR actually captured rally-incorrect)andbythefractionoffirstplaceanswers. relevant documents, the percentage of true answer sen- Under the “List” and “Other” scores are the NIST- tencesidentifiedbytheSVMonaheldoutTRECQAcol- reported F-scores. The “All” category is the average of lection was: 30% in the top-1 sentence, 43% in the top-2 thethreeprecedingcolumns,whichrepresentstheofficial sentences,59%inthetop-5sentences,74%inthetop-10 NISTscore. Tooursurprise,noneofthesevariationspro- sentences, and 80% in the top-15 sentences. It seemed vided significantly different results in the “All” category. highly likely that this process could afford a dramatic However, it seems clear that factoids were negatively improvement in overall performance. impactedbyGoogle-enhancedIRascomparedtophrase- based filtering, and the opposite is true for Lists. 2.3.2. Pulling out the Answers from the Sentences Thenextissuewastoextracttheanswerfromtheanswer Table 4: TREC 2006 Performance sentences. Onestrategywastoinsertthesesentencesinto the existing question-answerer and hope it could distill Strategy Factoid List Other All outtheanswer. Thisprocessyieldeda20%relativedeg- Phrase-filtered IR 107 .147 .148 .185 radation in performance due largely to the fact that the + improved QA (10/20/5) current system requires itself to find all relevant compo- [#1] .266 nents of questions, whereas the best sentence may have Google-enhanced 95 .156 .151 .181 the answer but not all relevant question components IR, improved QA (14/22/4) (neededforsupportedness). Althoughwewillattemptto [#2] .236 modifythebasesystemduringtheremainderof2006and 2007totackletheproblem,therewasalsoadesiretogeta Google-enhanced 96 .156 .154 .183 fully-statistical QA system. IR, improved QA, (14/22/4) Wehaveyettodevelopafullstatisticallearningpro- some statistical .238 cess for finding answers, but we did begin a simple and QA [#3] 3.2. Comparison to Past Years 3.3. Considering the “Other”s With these scores seeming to be only marginally better At TREC-2005, the F=0.248 that our system generated than they were last year, we wanted to determine if there was the maximum score for any of the “Other” question had actually been any true system improvements since answerers. Thisyear,ourF-scoredroppedbyanabsolute last year. We were able to identify our best competition 10%andourpositionfelltojustabovethemedian. Ifwe systemsfrom2004and2005andconductasmallexperi- had made changes to our “Other” answerer, we would ment to test for system improvements by applying these havebelievedthatwehadsimplyputforthapooreffortin past systems to this year’s data. Our experiments would making changes. On the other hand, since the only solely focus on factoids and we would not identify ques- change we made was to add a stage which would thrice tionsforwhichanassessor,hadheorsheseentheoutput ensure against empty answers, we had to seek to under- of the older system, may have judged an answer as cor- stand why the performance would have fallen as it had. rect. Likewise,whereaswehadsomeparsingproblemsin At TREC-2004, our first year of participation, our years past, we would allow the system to directly access system received a very high score and only 1/4 of the the new and updated parses of today (since the broken answers were given zero credit (with half of these due to and/or empty parses have long since been removed). It non-responsiveness of our system). In this year, though, wasourexpectationthatthesetwooversightswouldlikely halfofouranswersweregivennocrediteventhoughour balanceeachotherandprovideafairlyaccuratecompari- method for “Other”-answering is sort of a “kitchen sink” son of past year performance to that of the current year. approachwhichreportstonsofinformation. Wetherefore Additionally, since past-year systems were not con- reviewed the first ten of these zero-scored answers to see cerned with temporally inaccurate answers, and since what would have changed. “unsupported”isdifficulttotrulyjudgewithouttheinput Reviewingthe“Other”questionsisanon-trivialtask. of an assessor, we scored the three systems by allowing Thecoreissuewiththeseisnotknowinghowthedetermi- what TREC-2006 assessors had declared to be “Right,” nationismadeastowhethersomethingisvitalornot. It “Locally Correct”, and “Correct but Unsupported.” The appeared this year that since there were so many ques- following table provides the performance comparisons. tions asked from each series, the “Other” questions had littleinformationtochoosefromthatwasbothnoveland vital. Evenso,therearethreesituationsthatariseingiv- Table 5: TREC 2006 vs. Past Years ingasystemazeroscoreinsuchantheevaluation:(a)the QAsystemdidnotreturnitemsthatassessorsfoundtobe Factoid TREC Year valuable, (b) the QA system did return such items and #R+L+U/“Correct” Score receivednocredit,and(c)theQAsystemproduceditems TREC-2006 System 122 / .303 that these assessors deemed to be non-vital but other TREC-2005 System 95 / .236 assessorsmighthavebeenperceivedofasnuggets. Since thetaskissubjective,anevaluationof“c”isnotaparticu- TREC-2004 System 53/ .132 larlyhelpfuldirectiontostudy. Yetwewilltouchbriefly (TREC-2006 w/o Filter) (116 / .288) on the first two of these given our in-depth review of the ten zero-scored answers that we studied. (It should be Table5showsagratifyingresult. Fromthefirstthree notedforreference,though,thatthereissomesubjectivity rows we see that there have been reasonable improve- whichisinconsistent:in164.8,creditisgivenforthefact mentstooursystemoverthecourseofthepasttwoyears. thatJudiDenchwasawardedOrderoftheBritishEmpire, The last row of Table 5 is merely informational. but credit was not given in 153.8 for Alfred Hitchcock’s Since we were only able to submit three runs to TREC, receiving of higher honors as a Knight Commander.) wewerenotabletodeterminetheimpactofourbasicsys- By far the biggest problem for us with category “a” temimprovementsasopposedtothosethatwerecoupled above was that what were being deemed as nuggets this withIRimprovements. Withtheincorporationofthelast year were largely less-important pieces of information row, we are able to see that the basic system improve- whichsurfacedtothetopasvitalsbecausemoreinterest- mentscontributedtoabout21morerightanswersandthat ing information was posed as questions. If one were to IRcontributedto6beyondthat. Wedidnotruntheexper- ask a system: “Tell me everything interesting and unique iment of enhanced IR without the basic additions to the youcanaboutWarrenMoon”(Q141.8),onewouldexpect system,butwhilewewereoriginallydevelopingthealgo- to receive information about his life profile: birth, death rithms, this paradigm was tested and it was typically the (if appropriate), his profession, and other major suc- case that IR improvements and basic improvements had cesses. SincetheQ141seriesasksabouthispositionona 1-3 correct answers in common. football team, the college where he played ball, his birth year,histimeasaprobowler,hiscoaches,andtheteams on which he played, the remaining relevant information mustaddresshissuccessesandpossiblyhisdeath. Since hehasnotdied,onlysuccessesremain. Thus,thefactthat Table 6: Unsupported Answers: Why? hehadthethirdall-timehighestcareerpassingyards,and Our Answer a15-yearfootballcareerarenote-worthy. Evenso,ourIR QID Reason Unsupported [Document] prefilter rejected one of the documents containing one of these items, and the other item appeared in a 17th-place 145.4 february23, Needed a conviction date. Docu- document ... deeper than our Other processing typically 1999 mentdated3/2/1999refersto“last goes. [NYT19990 week” which is ambiguous. Furtherreviewinginthe“a”arena,wenotedthatvital 302.0069] nuggetsfor152.7(Mozart)appearedinour17thand60th 154.4 Margot Needed most-frequent actress in documents; a vital for 163.8 (Hermitage) was not in our Kidder Superman. Document states that top100;thesolevitalfor164.8wasin47thplace;thetwo [APW19981 Kidder starred with Reeve, but vitalsfrom175.5werein18thplaceandnon-top-100;and 213.1025] nothing about “most” so forth. The absence of such information in the higher- 172.1 Burlington Needed a city and state of com- IR documents was obviously a leading contributor to our [NYT20000 pany’s origin. The answer is inex- reducedscores. Inpastyears,therewerefewerquestions 124.0364] act, too, but this document only asked per series, and many of those questions did not says “based in,” not originated in. focussomuchonkeyeventsofpeopleandorganizations but were focused more on exercising the QA systems. 182.5 Scotland Needed country of Edinburg These facts seem to be the primary reason why our [APW19990 Fringe. Document discusses poli- “Other” results would be so drastically degraded. 506.0176] tics in Scotland and a “fringe However, there are a few instances of “b” occurring party” -- polysemy problem. as well. That is, our system reported vital information 188.1 California Needed US state with highest avo- thatwasoverlooked. In163.8,oursystemreportedwith- [APW19990 cado production. California is outcreditthattheHermitagewas“comparabletotheLou- 117.0079] only mentioned in passing and vre in Paris or the Metropolitan Museum in New York,” nothing mentions production rates. which was a specified vital nugget. Such issues are less 189.7 Edinburgh Needed city of JK Rowling in frequent, though, and they are not unexpected given that [NYT20000 2000. Document states that she our system reports tons of information as answers. Fur- 112.0203] lived in Edinburgh in 1993. thermore, if the answers we perused are indicative, the Unclear if she lived there in 2000. issue of vitals not receiving credit would probably con- 190.1 PITTS- Needed city of HJ Heinz. The tribute less that .05 absolute to the current F-score. BURGH byline gives Pittsburgh and dis- [APW19990 cusses company profits, but does 3.4 Unsupportedness 615.0036] not explicitly say its base as there. As mentioned previously, the large number of answers from our system that assessors were tagging as “unsup- 191.3 Germany Neededcountrythatwonfirstfour ported”seemedsomewhatsuspicioustousgiventhatour [XIE199906 IRFR World Cups. Document system does not draw its answers from the Web. We 20.0031] mentions “Germany” and “four” sought to review the answers being proposed by our sys- but not that Germany won 4 times. tem and determine what the unsupported issues were. 194.3 six Need number of players at 1996 First, based on the cumulative information provided [XIE199603 WorldChessSuperTournament. for all 59 competing system runs, we were able to deter- 20.0094] Documentmentions6players,but mine that the average run had 12 answers that were wrong tournament and game. declared to be inexact. We looked at our highest-scored 214.6 seven Need number of Miss America factoid run and noted that we had 10 apparently unsup- [NYT20000 pageant judges. Document is on ported answers. Although 10 was less than the average, 717.0370] Miss Texas Scholastic Pageant we still wanted to understand the issues. We reviewed which had 7 judges. each answer and found that all the answers were indeed unsupported (and possible inexact as well). The table below summarizes this information: the question number 4 FUTURE DIRECTIONS (QID), the answer our system reported, and the reason The future of QACTIS still holds a direction of multilin- why the answer was unsupported. gual and multimedia question-answering as a primary goal. YetweanticipatefutureparticipationinTRECnext yearuntilwehaveironedoutthewrinklesinoursystem. Our focus on textual QA for the next year will be to addresstheissuesthathaveyettobecompletedbutwhat were mentioned in this paper, such as improvements and exactnessfilteringusingmoremoderncontentextractors, better incorporation of hypernyms, and making improve- mentstoourstatisticalQAsystem. Wealsoplantomake our baseline system cleaner and more robust. 5 REFERENCES [1] Schone, P., Ciany, G., McNamee, P., Kulman, A., Bassi, T. , "Question Answering with QACTIS at TREC 2004” The 13th Text Retrieval Conference (TREC-2004),Gaithersburg,MD.NISTSpecialPub- lication 500-261,2004. [2] Schone, P., Ciany, G., Cutts, R.., McNamee, P., May- field, J., Smith, T. , "QACTIS-based Question Answering at TREC-2005” The 14th Text Retrieval Conference (TREC-2005), Gaithersburg, MD, ,2005. [3]BBN’s SERIFTM engine. [4]MillerG.A.,BeckwithR.,FellbaumC.,GrossD.,and Miller K. J. “WordNet: An online lexical database.” InternationalJournalofLexicography3(4):235-244, 1990. [5]P.Schone,J.Townsend,C.Olano,T.H.Crystal. “Text Retrieval via Semantic Forests.” TREC-6, Gaithers- burg, MD. NIST Special Publication 500-240, pp. 761-773, 1997 [6] www.wikipedia.org [7] Snow, R., Jurafsky, D., and Ng, A.Y., "Learning syn- tactic patterns for automatic hypernym discovery". NIPS 2004. [8] The LEMUR System. URL: http://www- 2.cs.cmu.edu/~lemur