Item-based assessment of translation competence: Chimera of objectivity versus prospect of reliable measurement

Authors

  • June Eyckmans Ghent University
  • Philippe Anckaert Université libre de Bruxelles

DOI:

https://doi.org/10.52034/lanstts.v16i0.436

Keywords:

translation assessment, objectivity, reliability, CDI-method, translation competence

Abstract

In the course of the past decade, scholars in Translation Studies have repeatedly expressed the need for more empirical research on translation assessment. Notwithstanding the many pleas for “objectivity” that have been voiced in the literature, the issue of reliability remains unaddressed. Although there is no consensus on the best method for measuring the quality of human or machine translations, it is clear that in both cases measurement error will need to be accounted for. This is especially the case in high-stake situations such as assessments that lead to translation competence being certified. In this article we focus on the summative assessment of translation competence in an educational context. We explore the psychometric quality of two assessment methods: the CDI method (Eyckmans, Anckaert, & Segers, 2009) and the PIE method (Kockaert & Segers, 2014; 2017; Segers & Kockaert, 2016). In our study, the reliability of both methods is compared empirically by scoring the same set of translations (n > 100) according to each method.

Author Biography

June Eyckmans, Ghent University

June Eyckmans is Assistant Professor in the Department of Translation, Interpreting and Communication at Ghent University. She obtained her PhD at the Radboud University of Nijmegen (the Netherlands) in 2004 on the methodology of L2 vocabulary assessment. Her research interests include cognitive approaches to foreign language learning and the methodology of interpreting and translation assessment.

References

Al-Qinai, J. (2000). Translation quality assessment: Strategies, parameters and procedures. Meta, 45(3), 497–519.

Anckaert, Ph., Eyckmans, J., & Segers, W. (2008). Pour une évaluation normative de la compétence de traduction. International Journal of Applied Linguistics, 155, 53–76.

Anckaert, Ph., Eyckmans, J., Justens, D. & Segers, W. (2013). Bon sens, faux-sens, contresens et non-sens sensdessus dessous: Pour une évaluation fidèle et valide de la compétence de traduction. In J.-Y. Le Disez & W. Segers (Eds.), Le bon sens en traduction (pp. 79–94). Rennes: Presses universitaires de Rennes.

Eyckmans, J., Anckaert, Ph. & Segers, W. (2009). The perks of norm-referenced translation evaluation. In C. Angelelli & H. Jacobson (Eds.), Testing and assessment in translation and interpreting (pp. 73–93). Amsterdam: John Benjamins.

Eyckmans, J., Anckaert, Ph. & Segers, W. (2016). Translation and interpretation skills. In D. Tsagari & J. Banerjee (Eds.), Handbook of second language assessment (pp. 219–235). Berlin: De Gruyter/Mouton.

Eyckmans, J., Segers, W. & Anckaert, Ph. (2012). Translation assessment methodology and the prospects of European collaboration. In D. Tsagari & I. Csépes (Eds.), Collaboration in language testing and assessment (pp. 171–184). Frankfurt am Main: Peter Lang.

Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford: Oxford University Press.

Baker, F. (2001). The basics of item response theory. University of Maryland, College Park: ERIC Clearinghouse on Assessment and Evaluation.

Bonett, D. G. (2002). Sample size requirements for testing and estimating coefficient alpha. Journal of Educational and Behavioral Statistics, 27(4), 335–340.

Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment. Newbury Park, CA: Sage.

Colina, S. (2002). Second language acquisition, language teaching and translation studies. The Translator, 8(1), 1–24.

Colina, S. (2009). Further evidence for a functionalist approach to translation quality evaluation. Target, 21, 235–264.

Conde Ruano, T. (2005). No me parece mal: Comportamiento y resultados de estudiantes al evaluar traducciones. (Unpublished doctoral dissertation). University of Granada, Granada.

Conrad, S. H. (1948). Characteristics and uses of item-analysis data. Psychological Monographs, 62, 1–48.

Ebel, R. L. (1979). Essentials of educational measurement. Englewood Cliffs, NJ: Prentice Hall.

EULITA. (2009). http://www.eulita.eu/home

Gile, D. (2005). La traduction: La comprendre, l’apprendre. Paris: PUF.

Han, C. (2016). Reporting practices of rater reliability in interpreting research: A mixed-methods review of 14 journals (2004–2014). Journal of Research Design and Statistics in Linguistics and Communication Science, 3(1), 49–75.

Henryson, S. (1971). Gathering, analyzing, and using data on test items. In R. L. Thorndike (Ed.), Educational

Measurement (2nd ed., pp. 153-159). Washington, DC: Council on Education,

Horton, D. (1998). Translation assessment: Notes on the interlingual transfer of an advertising text. IRAL, 36(2), 95–119.

House, J. (1981). A model for translation quality assessment. Tübingen: Gunter Narr.

ISO 17100:2015. Translation Services: Requirements for Translation Services. Technical Committee ISO/TC37, 2015

Kockaert, H. J., & Segers, W. (2014). Evaluation de la traduction: La méthode PIE (Preselected Items Evaluation). Turjuman. Revue de Traduction et d’Interprétation / Journal of Translation Studies,23(2), 232–250.

Kockaert H. J., Segers W. (2017). Evaluation of legal translations: PIE method (Preselected Items Evaluation). Journal of Specialised Translation, 27, 148–163.

Lado, R. (1961). Language testing: The construction and use of foreign language tests: A teacher’s book. London: Longmans.

Larose, R. (1998). Méthodologie de l’évaluation des traductions. Meta, 43(2), 163–186.

Lee-Jahnke, H. (2001). Aspects pédagogiques de l’évaluation en traduction, Meta, 46(2), 258–271.

Martínez, R. (2014). A deeper look into metrics for translation quality assessment (TQA): A case study. Miscelánea: A Journal of English and American Studies, 49, 73–94.

Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York, NY: McGraw-Hill.

PACTE. (2000). Acquiring translation competence: Hypotheses and methodological problems in a research project. In A. Beeby, D. Ensinger, & M. Presas (Eds.), Investigating translation (pp. 99–106). Amsterdam: John Benjamins.

PACTE. (2011). Results of the validation of the PACTE translation competence model: Translation problems and translation competence. In C. Alvstad, ‎A. Hild, &‎ E. Tiselius (Eds.), Methods and strategies of process research: Integrative approaches in translation studies (pp. 317–343). Amsterdam: John Benjamins.

Pidgeon, D., & Yates, A. (1968). An introduction to educational measurement. London: Routledge.

Prégent, R. (1990). La préparation d’un cours. Montréal, QC: École polytechnique.

Pym, A. (1992). Translation error analysis and the interface with language teaching. In C. Dollerup & A. Loddegaard (Eds.), Teaching translation and interpreting. Training, talent and experience: Papers from the first Language International Conference, Elsinore, Denmark, 31 May–2 June, 1991 (pp. 279–288). Amsterdam: John Benjamins.

Sager, J. C. (1989). Quality and standards: The evaluation of translations. In C. Picken (Ed.), The translator’s handbook (pp. 91–102). London: ASLIB.

Samuels, P. (2015). Statistical methods: Scale reliability analysis with small samples, Birmingham: Birmingham City University, Centre for Academic Success.

Secară, A. (2005). Translation evaluation: A state of the art survey. Proceedings of the eCoLoRe/MeLLANGE Workshop Leeds (pp. 39–44). Manchester: St. Jerome.

Segers, W., & Kockaert, H. J. (2016). Can subjectivity be avoided in translation evaluation? In M. Thelen, G. van Egdom, D. Verbeeck, & B. Lewandowska-Tomaszczyk (Eds.), Łódź Studies in Language, vol: 41, Translation and Meaning: New Series (pp. 69–78). Frankfurt am Main: Peter Lang.

Stejkal, J. (2006). Quality assessment in translation. MultiLingual, 80(17), 41–44.

Waddington, C. (2001). Different methods of evaluating student translations: The question of validity. Meta,46(2), 331–325.

Waddington, C. (2004). Should student translations be assessed holistically or through error analysis? Lebende Sprachen, 49(1), 28–35.

Wiersma, W., & Jurs, S.G. (1990). Educational measurement and testing (2nd ed.). Boston, MA: Allyn and Bacon.

Williams, M. (2001). The application of argumentation theory to translation quality assessment. Meta, 46(2), 327–344.

Williams, M. (2009). Translation quality assessment, Mutatis Mutandis, 2(1), 3–23.

Downloads

Additional Files

Published

29-01-2018

How to Cite

Eyckmans, J., & Anckaert, P. (2018). Item-based assessment of translation competence: Chimera of objectivity versus prospect of reliable measurement. Linguistica Antverpiensia, New Series – Themes in Translation Studies, 16. https://doi.org/10.52034/lanstts.v16i0.436

Issue

Vol. 16 (2017): Translator Quality—Translation Quality: Empirical Approaches to Assessment and Evaluation

Section

Articles

License

Authors who publish with this journal agree to the following terms:

  • Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under the CC BY-NC 4.0 Deed that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal. The material cannot be used for commercial purposes.
  • Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
  • Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).