Title: “Developing a process-oriented translation test for assessing English - Arabic basic translation skills” Author: Antar Solhy Abdellah Publication date: 2007 Source: CDELT (Centre for Developing English Language Teaching) 32nd Conference; ELT; a Futuristic vision, April (2007). Cairo: Ain Shams University 02 DEVELOPING A PROCESS-ORIENTED TEST FOR ASSESSING BASIC TRANSLATION SKILLS Antar Solhy Abdellah Qena Faculty of Education South Valley University ABSTRACT The study reviews translation validated tests and proposes a process-oriented translation test for assessing basic translation skills for freshmen English majors at the faculty of Education. The proposed test is developed based on the process approach to translating and translation teaching, and is confined to translation from English to Arabic. Translation basic skills (reading comprehension – researching- analytical thinking- and production) are the four dimensions of the test. Introduction While translation as a literary art has been the object of numerous studies that focused on the linguistic as well as the cultural aspects of the process of translation, very little research has been done in the area of statistically measuring the performance of students/translators through valid and reliable tools. Most studies discuss how to develop the competence, but an equally important issue is the question of how to find out whether the aim has been achieved and where best to look for evidence of this. Ghonsooly (1993: 55) explains that 'while translation methodology has been influenced by improvements in translation theory, its testing counterpart has remained untouched.' Translation competence can be studied from two perspectives: from the perspective of the product (i.e. the target 02 text,' its quality, its appropriateness for the specified purpose), or from the perspective of the process (i.e. the efficacy of the decision-making process, and the mental processes involved in the act of translating). Hairston states overtly that the process is more important than the product: "We cannot teach students to write by only looking at what they have written. We must also understand how that product came into being, and why it assumed the form it did. We have to try and understand what goes on during the act of writing. We have to do the hard thing, examine the intangible process, rather than the easy thing, evaluate the tangible product." (Hairston 1982: 81) Few empirical studies have been published on the subject of evaluation within translator training programmes at university level (Tirkkonen-Coildit 1991, Seguinot 1991, El-Banna 1993 are samples of these studies). This may be attributed to the difficulty of measuring a competence that is not yet fully defined, a competence that uses the most complex cognitive processes the human brain is capable of, a competence that combines, in Beeby's words, 'a number of different sub-competencies that seem to include the world, the universe and everything and are intricately interrelated' (2000:185). Furthermore, competence in translation (like Chomsky's linguistic competence) is an abstract concept that can only be measured in performance. Any attempt to delimit marking criteria may seem reductionist, or take so long to put into practice that it is not viable. Adab (2000: 215) points out that one of the questions to be considered in the context of developing translation competence is that of "how to evaluate the target text, as product of the process. 00 This is also necessary in order to determine the level of competence achieved by the translator and to identify areas in which competence is still to be developed." Another consideration relates to the question of how the assessor could perform this task reliably, in the sense of a more objective and less subjectively-oriented judgment of the product. Adab argues that the identification of a set of criteria could form the basis, both for production and evaluation of the product. "Improving translation competence should then be achievable, in terms of performance in transfer competence and production, through the awareness of the relative merits of different transfer strategies and careful selection from potential translation solutions." (2000: 215) Setting defined criteria for this purpose can assist in raising awareness of the decision-making and revision stages of the production process. El Sheik (1990:77) assures the same point when stating that translation "has often been misused in foreign language testing as a test of every thing connected with proficiency in a foreign language"; such tests were often used as a criterion for purposes "other than measuring translation itself". In order to assess the quality of translation, or the performance of the student/ translator, we need first to make clear why a target text is evaluated. Adab (2000: 215) lists the purposes that may exist for evaluating a target text; 'to assess the suitability of the text for its intended reader and use; to evaluate language competence (usually L2, L3); to determine levels of intercultural awareness; or to identify levels and types of translation competence'. He concludes that 'knowing the reason for evaluation and the criteria by which a text will be evaluated could help to improve the accuracy of this process, by giving a definition of the specific task in a given translation situation. Hence in order to measure translation ability we need to work on the basis of a framework that segments the components of this ability and that takes into consideration the different variables involved in the process; such as the readability of the text, time pressure, purpose of the translation, audience to whom the translation is delivered, and the language level required. 02 This study explores the practical experiments in the literature and proposes a model for 'examining the intangible process' of translating from English to Arabic. Based on this model a test is developed for measuring freshmen English majors' basic translation skills. Review of Literature on translation testing: SEVTE 1992 Stansfield, Scott and Kenyon (1992) worked to identify the variables that constitute translation ability. They drew on the results of a project conducted by the Center for Applied Linguistics (CAL) and reformulated the final version of the 'Spanish into English Verbatim Translation Exam' (SEVTE). SEVTE was originally designed to meet the needs of FBI officers who work with Spanish cases. One major challenge in the process of developing the test was to identify new translation skill-level descriptions (SLDs) which were needed to inform the test development process and, in particular, to inform the scoring of the test and the conversion of the scores to a 0-5 scale. These SLDs included a statement regarding the kinds of documents the translator can render into the target language, followed by a description of the accuracy of the translation, and the translator's use of grammar (morphology and syntax), vocabulary, spelling, punctuation, style and tone. SLDs were divided into two parts Accuracy and Expression, separate scores for each part were assigned. Accuracy involves content, i.e. the mistranslation, omission or addition of information in the source document. Expression involves form, i.e., the linguistic variables such as grammar, syntax, vocabulary, spelling, punctuation, style and tone. Results of the validation study supported the division of translation abilities. SEVTE consists of two sections: Multiple choice (MC) and production. The MC section includes 60 items 35 of which are in the form of translating a single word or phrase in the context of a sentence, while the other 25 are in the form of error detection where items are written in the target language only. 02 Errors may consist of incorrect grammar, word order, vocabulary, punctuation, or spelling. With the MC part, 30- 32 % of the items assess knowledge of grammar, 60% assess knowledge of vocabulary, 8% assess knowledge of mechanics and 5% of the items contain no errors. 35 minutes are allotted for this section with a maximum possible score of sixty. The production section of SEVTE consists of 28 production items: 15 Word or Phrase Translation (WPT) items which require translation of underlined words or phrases in sentences, 10 Sentence Translation (ST) items which require translation of complete sentences, and 3 Paragraph Translation (PT) items which require translation of entire paragraphs. 35 minutes are allotted to the first two sections (WPT and ST) and 48 minutes to the PT section. Each section is scored by a different rater. Scoring criteria for each section are detailed below. For the WPT section, the items are scored as either correct or incorrect with a maximum score of 15 points for accuracy. For the SW section, a rater scores the sentences on the basis of accuracy of the translation, i.e., the extent to which the original meaning has been appropriately conveyed. A scale of zero to five points1 is used for 1 SLDs (Skill-level Descriptions) for Accuracy: 0+ Has no real ability to translate connected discourse. Efforts to translate contain many mistranslations and omissions, and very little information from source document is conveyed. 1 Renders translations whose accuracy is deficient, with frequent mistranslations and omissions, and may make inappropriate additions. Much of the information from longer source document is lost. 1+ Produces translations whose accuracy is inadequate, containing many mistranslations or omissions, and possibly additions. Almost all nuances are lost. 2 Produces translations whose accuracy is mostly adequate and without severe substantive omissions, but without many nuances, and with quite a few mistranslations. May include some additions for clarification of areas the translator can not accurately convey. 2+ Produces translation whose accuracy is adequate, but contain some mistranslations or omissions, and reflect a limited ability to convey nuances. 3 Produces translations whose accuracy is good, with occasional minor mistranslations or omissions. Can handle clearly identifiable nuances. 02 assigning points for each of the ten sentences for a maximum of 50 points for accuracy. For the PT section, the three translated paragraphs are evaluated for correctness of Grammar (morphology), Expression (here refers to word order, vocabulary, tone, and style), Mechanics (spelling and punctuation), and Accuracy (appropriateness of meaning conveyed). Students are allowed to use dictionaries in this section of the test only. From zero to five points are awarded in each of the four above categories for each paragraph for a maximum of 45 points for Expression (Expression [ word order, vocabulary, tone, and style], Mechanics [spelling and punctuation], and Grammar [morphology]) and 15 points for Accuracy (appropriateness of meaning conveyed). Thus the PT section has an overall maximum of 60 points, the total mark for the production section is 125, and the total mark for the whole SEVTE is 185. Accuracy therefore will be calculated with reference to a maximum of 80 points (15 for the WPT, 50 for the ST, and 15 for the PT section), while Expression will be calculated with reference to a maximum of 105 points (60 for the MC section, and 45 for the PT section). The following diagram shows a summary of the SEVTE description and its scoring criteria. SEVTE MC Production 60 items- 60 points 28 items 3+ Produces translations whose accuracy is good ,there are occasional omissions, or sporadic minor mistranslations: nuances are not always conveyed exactly or are not conveyed at all. 4 Renders translations whose accuracy is excellent; almost all nuances are conveyed and there are no mistranslations. 4+ Can produce a target text that is totally accurate, coveys all nuances, and is devoid of mistranslations or omissions. 5 Can produce translations that are an exact reflection of the source text in all aspects, even when translating difficult or abstract structures. Can produce work that is totally accurate, with no mistranslations or omissions. 02 35 25 15 W/P 10 Sentence 3 Paragraph Word / Error translation translation translation Phrase Detection in context 35 ms. 48 ms. Ss can use 30-32 % assess dictionaries. knowledge of grammar. each section, different rater. 6 0 % assess W/PT: correct/ incorrect – 15 marks knowledge of accept appropriate translations. vocabulary. ST: on the basis of Accuracy 8% assess (appropriate meaning conveyed) on knowledge of a scale of 0-5, 50 points mechanics. PT: on the basis of correctness of 5% contain Grammar, Expression, Mechanics no errors. and Accuracy. 5 points for each category, for each paragraph, 20 points per paragraph, on a scale of 0-5, 60 points. 60 points (PT) + 15 (W/PT) + 50 (ST) = 125 Points. 35 ms. 60 points(MC) + 125 (Production) = 185 total Expression Accuracy 60 (MC) + 45 (PT) = 105 15 (WPT)+50 (ST)+15 (PT) = 80 185 TOTAL Figure (1) outline of the STEEVE 02 Results of the exam showed that SEVTE Accuracy and SEVTE Expression measure different constructs. Thus neither score can substitute for the other. This implies that although a person can translate information accurately from Spanish, he or she may not be able to express it appropriately in English. Similarly, although a person can express a translation appropriately in English, the information conveyed may not be accurate. Results also showed that SEVTE Accuracy and SEVTE Expression appear to be valid measures. Both were found to correlate highly with translation skill levels assigned by comparing direct translations to the SLDs specified. A third results was that Accuracy appears to be the more valid measure of translation ability,; as Accuracy showed moderate to moderately high correlations with all criterion variables. Expression was neither as highly nor as consistently correlated with criterion variables as Accuracy. Thus, Expression may represent a secondary, though still important, construct in translation. El-Banna 1993 El-Banna was concerned with the problem of the lack of proper and uniform evaluation of ESL /EFL learners' translation skill in most faculties of Education. He attributes this shortage to two main reasons: 1. the non-availability of foreign language translation test that have been designed to assess foreign language candidates' skill in translation with some degree of accuracy, and 2. the reluctance of departmental translation tests to sample EFL learners' skill in translating English- Arabic texts or vice versa. ( 1993:5) His concern was with the actual performance of translating and how it is represented through students' achievement in translation tests. He deals with translation as one major skill that is essential for language learning especially at the university level. The test consists of two parts; translation from English into Arabic, and translation from Arabic into English. Students are given four choices for the translation of a whole sentence which is appropriate in length and which is also informative of daily life situations. 02 The test consists, in its final form, of 25 sentences with multiple choices in each part of the test, so the total number is 50 items2, and the time allotted for the whole test is 80 minutes. The sample consisted of 92 EFL college freshmen at two faculties of Education: Kafr El-Sheikh and Tanta. Test content, intrinsic validity and reliability were calculated and results were used to determine different levels of proficiency in translation as this is the main purpose of the test. Three levels were indicated through students' raw scores in both parts: Beginner 3-8 in part I , and 3-11 in part II. Intermediate 9- 11 in part I and 12-15 in part II. Advanced 12-14 in part I and 16-23 in part II. El-Banna's work in the MC translation test seems to have good potential for measuring translation skills. Although the process of constructing an MC translation test is considerably complicated than constructing an open-ended translation test, the time of administration and scoring is much less compared with that in an open-ended test. While El-Banna's work is a promising step in the direction of validating translation tests, it is still an entry test for candidates applying for the department of English in Faculties of Education. The need is still urgent to design measuring instruments for assessing the development of translation skills through and after the teaching of different translation courses the four years of study in the department of English. Orozco 2000 2 An example from part I: Thanks very much for showing me around. I'm really much obliged. .ادج رطضم ةقيقح انأ .يداشرلإ لايزج ار كش .أ .اريثك رذتعأ ةقيقح و ، نارودلا يل كحيضوتل لايزج ارك ش .ب .اريثك كل نتمم عقاولا يف انأو .يتلوج يف يتقفارمل لايزج اركش .ج (1993:80) .اريثك كل نتمم اقح يننإ . يل كليصوتل لايزج اركش .د An example from part II: .أطخ بكترأ مل يننإف .موللا اذه لك ينملت لا a. Oh, Don't put all the blame on me. I've done nothing wrong. b. Don't blame. I hasn't done anything wrong. c. Oh! Don't put all blaming on me. I don't commit a fault. d. Don't blame me for anything. I don't commit any mistake. (1993: 89) 02