Why methods matter: Approaching multimodality in translation research
DOI:
https://doi.org/10.52034/lanstts.v17i0.522Keywords:
multimodality, multimodal translation, research method, metafunctional analysis, multimodal corpus research in TS, reception researchAbstract
The study of multimodal phenomena calls upon translation scholars to cross disciplinary boundaries and adopt a range of theoretical and methodological approaches. The diversity of the multimodal landscape brings about research challenges that must be carefully addressed to ensure that these research efforts yield useful and credible results. This special issue is dedicated to a discussion on how to engage in multimodal translation research: how traditional research methods can be adapted and what kinds of novel approaches can be adopted or developed in order to deal with a diversity of multimodal data. In this introduction, we first discuss definitions of mode and multimodality and reflect on the nature of multimodality as a topic of research within Translation Studies. We then explain our rationale for dedicating the special issue to research methods and introduce three areas of multimodal translation research that, in our view, merit particular attention from a methodological point of view. Finally, we introduce the articles contained in this special issue.
References
Abuczki, A., & Ghazaleh, E. B. (2013). An overview of multimodal corpora, annotation tools, and schemes. Argumentum, 9, 86–98.
Adami, E. (2017). Multimodality. In O. García, N. Flores, & M. Spotti (Eds.), The Oxford handbook of language and society (pp. 451–472). Oxford: Oxford University Press.
Alves Veiga, M. (2006). Subtitling reading practices. In J. Ferreira Duarte, A. Assis Rosa, & T. Seruya (Eds.), Translation Studies at the interface of disciplines (pp. 161–168). Amsterdam: John Benjamins.
Antonini, R. (2007). SAT, BLT, Spirit Biscuits, and the Third Amendment: What Italians make of cultural references in dubbed texts. In Y. Gambier, M. Shlesinger, & R. Stolze (Eds.), Doubts and directions in Translation Studies: Selected contributions from the EST Congress, Lisbon 2004 (pp. 153–167). Amsterdam: John Benjamins.
Bairstow, D. (2011). Audiovisual processing while watching subtitled films: A cognitive approach. In A. Şerban, A. Matamala, & J.-M. Lavaur (Eds.), Audiovisual translation in close-up: Practical and theoretical approaches (pp. 205–219). Bern: Peter Lang.
Baldry, A., & Thibault, P.J. (2006). Multimodal transcription and text analysis: A multimedia toolkit and coursebook. London: Equinox.
Baños, R., Bruti S., & Zanotti, S. (2013). Corpus linguistics and audiovisual translation: In search of an integrated approach. Perspectives, 21(4), 483–490.
Bordwell, D., & Thompson, K. (1986). Film art. New York, NY: Alfred A. Knopf.
Brems, E., & Ramos-Pinto, S. (2013). Reception and translation. In Y. Gambier & L. van Doorslaer (Eds.), Handbook of translation studies, Vol. 4, (pp. 142–147). Amsterdam: John Benjamins.
Cabezas-Cáceres, C. (2013). Audiodescripció i recepció. Efecte de la velocitat de narració l'entonació i l'explicitació en la comprensió fílmica. Tesis doctoral. Retrieved from http://hdl.handle.net/10803/113556.
Caniato, M. (2014). Double meaning re-negotiation: Italian films in Flanders as cultural objects. inTRAlinea Special Issue: Across Screens Across Boundaries. Retrieved from http://www.intralinea.org/specials/article/2075.
Casetti, F., & Di Chio, F. (1990). Analisi del Film. Milan: Bompiani.
Chiaro, D., & Antonini, R. (2005). The quality of dubbed television programmes in Italy: The experimental design of an empirical study. In M. B. Paganelli & N. Maxwell (Eds.), Cross-cultural encounters: Linguistic perspectives (pp. 33–44). Rome: Officina Edizioni.
Chiaro, D. (2007). The effect of translation on humour response: The case of dubbed comedy in Italy. In Y. Gambier, M. Shlesinger, & R. Stolze (Eds.), Doubts and directions in translation studies: Selected contributions from the EST Congress, Lisbon 2004 (pp. 137–152). Amsterdam: John Benjamins.
Chmiel, A., & Mazur, I. (2012). AD reception research: some methodological considerations. In E. Perego (Ed.), Emerging topics in translation: Audio description (pp. 57–80). Trieste: EUT.
Chueasuai, P. (2013). Translation shifts in multimodal text: A case of the Thai version of Cosmopolitan. JoSTrans, The Journal of Specialised Translation, 20, 107–121.
Croft, W., & Cruse, D. A. (2004). Cognitive linguistics. Oxford: Oxford University Press.
Desilla, L. (2014). Reading between the lines, seeing beyond the images: An empirical study on the comprehension of implicit film dialogue meaning across cultures. The Translator, 20(2), 194–214.
Dondis, D.A (1973). A primer of visual literacy. Cambridge, MA: MIT Press.
Espindola, E. (2012). Systemic functional linguistics and audiovisual translation studies: A conceptual basis for the study of the language of subtitles. DELTA: Documentação de Estudos em Lingüística Teórica e Aplicada, 28, 495–513.
Everett, Y.U. (2015). Reconfiguring Myth and Narrative in Contemporary Opera. Bloomington: Indiana University Press.
Evison, J. (2010). What are basics of analyzing a corpus? In A. O´Keeffe & M. McCarthy (Eds.), The Routledge handbook of corpus linguistics (pp. 122–135). London: Routledge.
Feng, D., & Espindola, E. (2013). Integrating systemic functional and cognitive approaches to multimodal discourse analysis. Ilha do Desterro, 64, 85–110.
Forceville, C. (2007). Book review: Multimodal Transcription and Text Analysis: A Multimedia Toolkit and Coursebook. Journal of Pragmatics, 39(6), 1235–1238.
Fuentes Luque, A. (2003). An empirical approach to the reception of AV translated humour: A case study of the Marx Brothers’ “Duck Soup”. The Translator, 9(2), 293–306.
Genette, G., & Maclean, M. (1991). Introduction to the paratext. New Literary History, 22(2), 261–272.
Gibbons, A. (2012). Multimodality, cognition and experimental literature. New York, NY: Routledge.
Ghia, E. (2012). The impact of translation strategies on subtitle reading. In E. Perego (Ed.), Eye tracking in audiovisual translation (pp. 157–182). Rome: Aracne Editrice.
Göpferich, S. (2010), Data documentation and data accessibility in translation process research. The Translator, 16(1), 93–124.
Halliday, M.A.K. (1978). Language as social semiotic: The social interpretation of language and meaning. London: Edward Arnold.
Halliday, M.A.K. (1985). An introduction to functional grammar. London: Edward Arnold.
Halliday, M.A.K., & C. Matthiessen. (2004) 2013. An Introduction to Functional Grammar. 3rd ed. New York: Routledge.
Halverson, S.L. (2007). Investigating gravitational pool in translation: The case of English progressing construction. In R. Jääskeläinen, T. Puurtinen, & H. Stotesbury (Eds.), Text, processes, and corpora: Research inspired by Sonja Tirkkonen-Condit (pp. 175–196). Joensuu: Joensuu Yliopistopaino.
Handford, M. (2010). What can a corpus tell us about specialist genres? In A. O’Keeffe & M. McCarthy (Eds.), The Routledge handbook of corpus linguistics (pp. 254–269). London: Routledge.
Hovy, E., & Lavid, J. (2010). Towards a ‘science’ of corpus annotation: A new methodological challenge for corpus linguistics. International Journal of Translation, 22(1), 13–36.
Jewitt, C. (2014). Multimodal approaches. In S. Norris & C. D. Maier (Eds.), Interactions, images and texts: A reader in multimodality (pp. 127–136). Boston, MA: De Gruyter.
Jiménez Crespo, M.A. (2012). Lost or lost in translation: a contrastive corpus-based study of original and localized US websites. JoSTrans, The Journal of Specialised Translation, 17, 126–173.
Kaindl, K. (2004). Multimodality in the translation of humor in comics. In E. Ventola, C. Charles, & M. Kaltenbacher (Eds.), Perspectives on multimodality (pp. 173–192). Amsterdam: John Benjamins.
Kaindl, K. (2013). Multimodality and translation. In C. Millán & F. Bartrina (Eds.), The Routledge handbook of translation studies (pp. 257–269). London: Routledge.
Ketola, A. (2016). Towards a multimodally oriented theory of translation: A cognitive framework for the translation of illustrated technical texts. Translation Studies, 9(1), 67–81.
Ketola, A. (2018). Word–image interaction in technical translation: Students translating an illustrated text. Acta Universitatis Tamperensis 2364. Tampere: Tampere University Press.
Kokkola, S., & Ketola, A. (2015). Thinking outside the “Methods Box”: New avenues for research in multimodal translation. In D. Rellstab & N. Siponkoski (Eds.), Rajojen dynamiikkaa, Gränsernas dynamik, Borders under negotiation, Grenzen und ihre Dynamik. VAKKI-symposiumi XXXV 12.–13.2.2015 (pp. 219–228). Vaasa: VAKKI Publications 4. Retrieved from http://www.vakki.net/publications/2015/VAKKI2015_Kokkola&Ketola.pdf.
Kress, G. (2010). Multimodality: A social semiotic approach to contemporary communication. New York, NY: Routledge.
Kress, G., & van Leeuwen, T. (1996) 2006. Reading images: The grammar of visual design. London: Routledge.
Kress, G., & van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. London: Arnold.
Kruger, J.-L., & Doherty, S. (2018). Triangulation of online and offline measures of processing and reception in AVT. In E. Di Giovanni & Y. Gambier (Eds.), Reception studies and audiovisual translation (pp. 91–109). Amsterdam: John Benjamins.
Lång, J., Mäkisalo, J., Gowases, T., & Pietinen S. (2013). Using eye tracking to study the effect of badly synchronized subtitles on the gaze paths of television viewers. New Voices in Translation Studies, 10, 72–86.
Langacker, R. W. (1987/2008). Cognitive grammar: A basic introduction. Oxford: Oxford University Press.
López Rodriguez, C. I. (2007). Understanding scientific communication through the extraction of the conceptual and rhetorical information codified by verbs. Terminology, 13(1), 61–84.
Mangiron, C. (2016). Reception of game subtitles: An empirical study. The Translator, 22(1), 72–93.
Mangiron, C. (2018). Reception studies in game localisation: Taking stock. In E. Di Giovanni & Y. Gambier (Eds.), Reception studies and audiovisual translation (pp. 277–296). Amsterdam: John Benjamins.
Martinec, R., & Salway, A. (2005). A system for image–text relations in new (and old) media. Visual Communication, 4(3), 339–374.
Millán-Varela, C. (2004). Exploring advertising in a global context: Food for thought. The Translator, 10(2), 245–267.
Mubenga, K. S. (2009). Towards a multimodal pragmatic analysis of film discourse in audiovisual translation. Meta, 54(3), 466–484.
Mubenga, K. S. (2010). Investigating norms in interlingual subtitling: A systemic functional perspective. Perspectives: Studies in Translatology, 18(4), 251–274.
O’Hagan, M. (2009). Towards a cross-cultural game design: An explorative study in understanding the player experience of a localised Japanese video game. JosTrans. The Journal of Specialised Translation, 11, 211–233.
O’Halloran, K. L., & Smith, B.A. (2011). Multimodal studies. In K. L. O’Halloran & B. A. Smith (Eds.), Multimodal studies: Exploring issues and domains (pp. 1–13). New York, NY: Routledge.
Oittinen, R., Ketola A., & Garavini, M. (2018). Translating picturebooks: Revoicing the verbal, the visual and the aural for a child audience. New York, NY: Routledge.
Orero, P., Doherty, S., Kruger, J.-L., Matamala, A., Pedersen, J., Perego, E., Romero-Fresco, P., Rovira-Esteva, S., Soler-Vilageliu, O., & Szarkowska, A. (2018). Conducting experimental research in audiovisual translation (AVT): A position paper. JoSTrans, The Journal of Specialised Translation, 30, 105–126.
Page, R. (2010). Introduction. In R. Page (Ed.), New perspectives on narrative and multimodality (pp. 1–13). New York, NY: Routledge.
Pauwels, L. (2012). A multimodal framework for analyzing websites as cultural expressions. Journal of Computer-Mediated Communication, 17, 247–265.
Perego, E., Del Missier, F., Porta, M., & Mosconi, M. (2010). The cognitive effectiveness of subtitle processing. Media Psychology, 13(3), 243–272.
Perego, E. (2012). Introduction. In E. Perego (Ed.), Eye tracking in audiovisual translation (pp. 7–11). Rome: Aracne Editrice.
Perego, E., Laskowska, M., Matamala, A., Remael, A., Robert, I. S., Szarkowska, A., Vilaró, A., & Bottiroli, S. (2016). Is subtitling equally effective everywhere?: A first cross-national study on the reception of interlingually subtitled messages. Across Languages and Cultures, 17(2), 205–229.
Pérez Payá, M. (2010). Recortes de cine audiodescrito: El lenguaje cinematográfico en Taggetti Imagen y su reflejo en la audiodescripción. In C. Jiménez Hurtado, A. Rodríguez, & C. Seibel (Eds.), Un corpus de cine: Teoría y práctica de la audiodescripción (pp. 111–179). Granada: Tragacanto.
Pérez-González, L. (2014a). Audiovisual translation: Theories, methods and issues. London: Routledge.
Pérez-González, L. (2014b). Multimodality in translation and interpreting studies: Theoretical and methodological perspectives. In S. Bermann & C. Porter (Eds.), A companion to translation studies (pp. 119–131). Chichester: Wiley-Blackwell.
Quinci, C. (2015). Translators in the making: An empirical longitudinal study of translation competence and its development (Unpublished doctoral dissertation). Universitá degli Studi di Trieste, Trieste.
Ramos Caro, M. (2018). Los estudios de recepción en Traducción Audiovisual: Aspectos metodológicos. In A. Rojo (Ed.), Diseños y métodos de investigación en traducción (pp. 99–124). Madrid: Anthropos.
Rodríguez-Inés, P. (2013). ¿Cómo traducen traductores y profesores de idiomas?: Estudio de corpus. Meta, 58(1), 165–190.
Rodríguez-Inés, P. (2017). Corpus-based insights into cognition. In J. W. Schwieter & A. Ferreira (Eds.), The handbook of translation and cognition (pp. 265–289). Hoboken, NJ: John Wiley & Sons.
Saldanha, G., & O’Brien, S. (2013). Research methodologies in translation studies. Manchester: St. Jerome.
Serafini, F. (2010). Reading multimodal texts: Perceptual, structural and ideological perspectives. Children’s Literature in Education, 41, 85–104.
Soler Gallego, S. (2013). La traducción accesible en el espacio multimodal museográfico (Unpublished doctoral dissertation). Universidad de Granada, Granada.
Soto Almela, J. (2013). La traducción de términos culturales en el contexto turístico español-inglés: Recepción real en usuarios anglófonos. Quaderns. Revista de Traducció, 20, 235–250.
Szarkowska, A., & Gerber-Morón, O. (2018). Viewers can keep up with fast subtitles: Evidence from eye movements. PLoS ONE, 13(6), e0199331.
Torresi, I. (2007). Translating the visual: The importance of visual elements in the translation of advertising across cultures. In D. Kenny & K. Ryou (Eds.), Across boundaries: International perspectives on translation studies (pp. 38–55). Newcastle: Cambridge Scholars.
Tuominen, T. (2012). The art of accidental reading and incidental listening: An empirical study on the viewing of subtitled films. Acta Universitatis Tamperensis 1794. Tampere: Tampere University Press.
Tuominen, T. (2018). Multi-method research: Reception in context. In E. Di Giovanni & Y. Gambier (Eds.), Reception studies and audiovisual translation (pp. 69–89). Amsterdam: John Benjamins.
Tymoczko, M. (2007). Enlarging translation, empowering translators. Manchester: St. Jerome.
Van Meerbergen, S. (2010). Nederländska bilderböcker blir svenska: En multimodal översättningsanalys [Dutch picturebooks become Swedish: A multimodal translation analysis]. Stockholm: University of Stockholm Press.
Widler, B. (2004). A survey among audiences of subtitled films in Viennese cinemas. Meta, 49(1), 98–101.
Yayoi, U.E. (2015). Reconfiguring myth and narrative in contemporary opera. Bloomington, IN: Indiana University Press.
d’Ydewalle, G., Praet, C., Verfaillie, K., & Van Rensbergen, J. (1991). Watching subtitled television: Automatic reading behavior. Communication Research, 18(5), 650–666.
Downloads
Published
How to Cite
Issue
Section
License
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under the CC BY-NC 4.0 Deed that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal. The material cannot be used for commercial purposes.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
