Methodological considerations for survey research: Validity, reliability, and quantitative analysis
DOI:
https://doi.org/10.52034/lanstts.v19i0.549Keywords:
Survey, validity, reliability, measurement invariance, Cognitive Translation StudiesAbstract
As translation and interpreting studies continue to develop cognitive theories of translator and interpreter behavior and processing, there has been increased emphasis on research methods and data collection methodologies to glean new insights into the translation process. This article presents a critical review of survey research methods in Cognitive Translation Studies and argues for their inclusion as a means of better understanding translator and interpreter attitudes, behaviors, perceptions, and values. The article begins with a reflection on measurement and the need for alignment with theoretical frameworks and constructs; then it reviews important considerations when developing theoretically-grounded, empirically-based survey instruments, namely, validity, reliability, measurement invariance, and quantitative analysis. The article concludes with a call for additional methodological reflection on developing and using survey instruments.
References
Agbo, A. A. (2010). Cronbach’s alpha: Review of limitations and associated recommendations. Journal of Psychology in Africa, 20(2), 233–239. https://doi.org/10.1080/14330237.2010.10820371
Alves, F. (Ed.). (2003). Triangulating translation: Perspectives in process-oriented research. John Benjamins. https://doi.org/10.1075/btl.45
Alves, F. (2015). Translation process research at the interface: Paradigmatic, theoretical, and methodological issues in dialogue with cognitive science, expertise studies, and psycholinguistics. In A. Ferreira & J. W. Schwieter (Eds.), Psycholinguistic and cognitive inquiries into translation and interpreting (pp. 17–40). John Benjamins. https://doi.org/10.1075/btl.115.02alv
Alves, F., & Hurtado Albir, A. (2017). Evolution, challenges, and perspectives for research on cognitive aspects of translation. In J. W. Schwieter & A. Ferreira (Eds.), The handbook of translation and cognition (pp. 537–554). Wiley. https://doi.org/10.1002/9781119241485.ch29
Alvstad, C., Hild, A., & Tiselius, E. (2011). Methods and strategies of process research: Integrative approaches in translation studies. In C. Alvstad, A. Hild, & E. Tiselius (Eds.), Methods and strategies of process research (pp. 1–9). John Benjamins. https://doi.org/10.1075/btl.94
American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (1999). Standards for educational and psychological testing. American Educational Research Association.
Angelelli, C. V. (2004). Revisiting the interpreter’s role: A study of conference, court, and medical interpreters in Canada, Mexico, and the United States. John Benjamins. https://doi.org/10.1075/btl.55
Angelelli, C. V., & Baer, B. J. (Eds.). (2016). Researching translation and interpreting. Routledge. https://doi.org/10.4324/9781315707280
Balling, L. W. (2008). A brief introduction to regression designs and mixed-effects modelling by a recent convert. Copenhagen Studies in Language, 36, 175–192.
Bandalos, D. L. (2018). Measurement theory and applications for the social sciences. Guilford Press.
Beaujean, A. A. (2014). Latent variable modeling using R: A step-by-step guide. Routledge. https://doi.org/10.4324/9781315869780
Behr, D., & Shishido, K. (2016). The translation of measurement instruments for cross-cultural surveys. In C. Wolf, D. Joye, T. W. Smith, & Y.-C. Fu (Eds.), The SAGE handbook of survey methodology (pp. 269–287). SAGE. https://doi.org/10.4135/9781473957893.n19
Bernardini, S. (2001). Think-aloud protocols in translation research: Achievements, limits, future prospects. Target, 13(2), 241–263. https://doi.org/10.1075/target.13.2.03ber
Bolaños-Medina, A. (2014). Self-efficacy in translation. Translation and Interpreting Studies, 9(2), 197–218. https://doi.org/10.1075/tis.9.2.03bol
Bollen, K. A. (2002). Latent variables in psychology and the social sciences. Annual Review of Psychology, 53(1), 605–634. https://doi.org/10.1146/annurev.psych.53.100901.135239
Bontempo, K., & Napier, J. (2011). Evaluating emotional stability as a predictor of interpreter competence and aptitude for interpreting. Interpreting, 13(1), 85–105. https://doi.org/10.1075/intp.13.1.06bon
Borsboom, D. (2005). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge University Press. https://doi.org/10.1017/CBO9780511490026
Boyle, G. J., Saklofske, D. H., & Matthews, G. (Eds.). (2015). Measures of personality and social psychological constructs. Academic Press.
Brown, J. D. (1996). Testing in language programs. Prentice Hall.
Buchanan, E. A., & Hvizdak, E. E. (2009). Online survey tools: Ethical and methodological concerns of human research ethics committees. Journal of Empirical Research on Human Research Ethics, 4(2), 37–48. https://doi.org/10.1525/jer.2009.4.2.37
Carifio, J., & Perla, R. J. (2007). Ten common misunderstandings, misconceptions, persistent myths and urban legends about Likert scales and Likert response formats and their antidotes. Journal of Social Sciences, 3(3), 106–116. https://doi.org/ 10.3844/jssp.2007.106.116
Carifio, J., & Perla, R. J. (2008). Resolving the 50-year debate around using and misusing Likert scales. Medical Education, 42(12), 1150–1152. https://doi.org/10.1111/j.1365-2923.2008.03172.x
Chan, E. K. H. (2014). Standards and guidelines for validation practices: Development and evaluation of measurement instruments. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 9–24). Springer. https://doi.org/10.1007/978-3-319-07794-9_2
Chesterman, A. (2009). The name and nature of translator studies. Hermes, 22(42), 13–22. https://doi.org/10.7146/hjlcb.v22i42.96844
Choi, B. C. K., & Pak, A. W. P. (2005). A catalog of biases in questionnaires. Preventing Chronic Disease, 2(1), A13.
Clark, A. (1996). Being there: Putting brain, body, and world together again. MIT Press. https://doi.org/10.7551/mitpress/1552.001.0001
Cole, D. A., & Preacher, K. J. (2014). Manifest variable path analysis: Potentially serious and misleading consequences due to uncorrected measurement error. Psychological Methods, 19(2), 300–315. https://doi.org/10.1037/a0033805
Cortina, J. M. (1993). What is coefficient alpha?: An examination of theory and applications. Journal of Applied Psychology, 78(1), 98–104. https://doi.org/10.1037/0021-9010.78.1.98
Coulacoglou, C., & Saklofske, D. H. (2017). Psychometrics and psychological assessment: Principles and applications. Academic Press.
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957
Csizér, K., & Dörnyei, Z. (2005). The internal structure of language learning motivation and its relationship with language choice and learning effort. Modern Language Journal, 89(1), 19–36. https://doi.org/10.1111/j.0026-7902.2005.00263.x
Dancette, J. (1997). Mapping meaning and comprehension in translation: Theoretical and experimental issues. In J. H. Danks, G. M. Shreve, S. B. Fountain, & M. K. McBeath (Eds.), Cognitive processes in translation and interpreting (pp. 77–103). SAGE.
DeCastellarnau, A. (2018). A classification of response scale characteristics that affect data quality: A literature review. Quality & Quantity, 52(4), 1523–1559. https://doi.org/10.1007/s11135-017-0533-4
DeVellis, R. F. (2017). Scale development: Theory and applications (4th ed.). SAGE.
Diamantopoulos, A. (Ed.). (2008). Formative indicators [Special issue]. Journal of Business Research, 61(12).
Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105(3), 399–412. https://doi.org/10.1111/bjop.12046
Edwards, J. R., & Bagozzi, R. P. (2000). On the nature and direction of relationships between constructs and measures. Psychological Methods, 5(2), 155–174. https://doi.org/10.1037/1082-989X.5.2.155
Ericsson, K. A., & Simon, H. A. (1984). Protocol analysis: Verbal reports as data. MIT Press.
Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272–299. https://doi.org/10.1037/1082-989X.4.3.272
Flaskerud, J. H. (2012). Cultural bias and Likert-type scales revisited. Mental Health Nursing, 33(2), 130–132. https://doi.org/10.3109/01612840.2011.600510
Goodwin, L. D., & Leech, N. L. (2003). The meaning of validity in the new Standards for Educational and Psychological Testing: Implications for measurement courses. Measurement and Evaluation in Counseling and Development, 36, 181–191. https://doi.org/10.1080/07481756.2003.11909741
Gorard, S. (2010). Measuring is more than assigning numbers. In G. Walford, E. Tucker, & M. Viswanathan (Eds.), The SAGE handbook of measurement. SAGE. https://doi.org/10.4135/9781446268230.n20
Graybill, P., Aggas, J., Dean, R. K., Demers, S., Finigan, E. G., & Pollard Jr., R. Q. (2010). A community-participatory approach to adapting survey items for deaf individuals and American Sign Language. Field Methods, 22(4), 429–448. https://doi.org/10.1177/1525822X10379201
Groth-Marnat, G., & Wright, A. J. (2016). Handbook of psychological assessment. Wiley.
Halverson, S. L. (2010). Cognitive translation studies: Developments in theory and method. In G. M. Shreve & E. Angelone (Eds.), Translation and cognition (pp. 349–369). John Benjamins. https://doi.org/10.1075/ata.xv.18hal
Halverson, S. L. (2019). ‘Default’ translation: A construct for cognitive translation and interpreting studies. Translation, Cognition & Behavior, 2(2), 187–210. https://doi.org/10.1075/tcb.00023.hal
Hambleton, R. K., & Patsula, L. (1998). Adapting tests for use in multiple languages and cultures. Social Indicators Research, 45(1–3), 153–171. https://doi.org/10.1023/A:1006941729637
Han, C. (2018). Latent trait modeling of rater accuracy in formative peer assessment of English–Chinese consecutive interpreting. Assessment & Evaluation in Higher Education, 43(6), 979–994. https://doi.org/10.1080/02602938.2018.1424799
Harkness, J. A., Pennell, B.-E., & Shoua-Glusberg, A. (2004). Survey questionnaire translation and assessment. In S. Presser, J. M. Rothgeb, M. P. Couper, J. T. Lessler, E. Martin, J. Martin, & E. Singer (Eds.), Methods for testing and evaluating survey questionnaires (pp. 453–473). Wiley. https://doi.org/10.1002/0471654728.ch22
Harkness, J. A., Villar, A., & Edwards, B. (2010). Translation, adaptation, and design. In J. A. Harkness, M. Braun, B. Edwards, T. P. Johnson, L. Lyberg, P. P. Mohler, B.-E. Pennell, & T. W. Smith (Eds.), Survey methods in multinational, multiregional, and multicultural contexts (pp. 115–140). Wiley. https://doi.org/10.1002/9780470609927.ch7
Harpe, S. E. (2015). How to analyze Likert and other rating scale data. Currents in Pharmacy Teaching and Learning, 7(6), 836–850. https://doi.org/10.1016/j.cptl.2015.08.001
House, J. (2013). Towards a new linguistic-cognitive orientation in translation studies. Target, 25(1), 46–60. https://doi.org/10.1075/target.25.1.05hou
Hubscher-Davidson, S. (2009). Personal diversity and diverse personalities in translation: A study of individual differences. Perspectives, 17(3), 175–192. https://doi.org/10.1080/09076760903249380
Hurtado Albir, A. (Ed.). (2017). Researching translation competence by PACTE Group. John Benjamins. https://doi.org/10.1075/btl.127
Hvelplund, K. T. (2014). Eye tracking and the translation process: Reflections on the analysis and interpretation of eye-tracking data. MonTI Special Issue—Minding Translation, Special Issue 1, 201–223. https://doi.org/10.6035/monti.v0i0.292854
Hvelplund, K. T. (2017). Eye tracking in translation process research. In J. W. Schwieter & A. Ferreira (Eds.), The handbook of translation and cognition (pp. 248–264). Wiley. https://doi.org/10.1002/9781119241485.ch14
Jääskeläinen, R. (2010). Think-aloud protocol. In Y. Gambier & L. van Doorslaer (Eds.), Handbook of translation studies (Vol 1, pp. 371–373). John Benjamins. https://doi.org/10.1075/hts.1.thi1
Jääskeläinen, R. (2011). Studying the translation process. In K. Malmkjær & K. Windle (Eds.), The Oxford handbook of translation studies (pp. 123–135). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199239306.013.0010
Jakobsen, A. L. (2003). Effects of think aloud on translation speed, revision, and segmentation. In F. Alves (Ed.), Triangulating translation: Perspectives in process oriented research (pp. 69–95). John Benjamins. https://doi.org/10.1075/btl.45.08jak
Jamieson, S. (2004). Likert scales: How to (ab)use them. Medical Education, 38, 1212–1218. https://doi.org/10.1111/j.1365-2929.2004.02012.x
Jiménez Ivars, A., Pinazo Catalavud, D., & Ruiz i Forés, M. (2014). Self-efficacy and language proficiency in interpreter trainees. The Interpreter and Translator Trainer, 8(2), 167–182. https://doi.org/10.1080/1750399X.2014.908552
Just, M. A., & Carpenter, P. A. (1980). A theory of reading: From eye fixation to comprehension. Psychological Review, 87(4), 329–354. https://doi.org/10.1037/0033-295X.87.4.329
Kane, M. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64(3), 425–461. https://doi.org/10.3102/00346543064003425
Kankaraš, M., & Moors, G. (2010). Researching measurement equivalence in cross-cultural studies. Psihologija, 43(2), 121–136. https://doi.org/10.2298/PSI1002121K
King, G., Murray, C. J. L., Salomon, J. A., & Tandon, A. (2004). Enhancing the validity and cross-cultural comparability of measurement in survey research. American Political Science Review, 98(1), 191–207. https://doi.org/10.1017/S000305540400108X
Kruger, J.-L., Soto Sanfiel, M. T., Doherty, S., & Ibrahim, R. (2016). Towards a cognitive audiovisual translatology: Subtitles and embodied cognition. In R. Muñoz Martín (Ed.), Reembedding translation process research (pp. 171–194). John Benjamins. https://doi.org/10.1075/btl.128.09kru
Lee, J. W., Jones, P. S., Mineyama, Y., & Zhang, X. (2002). Cultural differences in response to a Likert scale. Research in Nursing & Health, 25(4), 295–306. https://doi.org/10.1002/nur.10041
Lee, S.-B. (2014). An interpreting self-efficacy (ISE) scale for undergraduate students majoring in consecutive interpreting: Construction and preliminary validation. The Interpreter and Translator Trainer, 8(2), 183–203. https://doi.org/10.1080/1750399X.2014.929372
Lee, S.-B. (2018). Exploring a relationship between students’ interpreting self-efficacy and performance: Triangulating data on interpreter performance assessment. The Interpreter and Translator Trainer, 12(2), 166–187. https://doi.org/10.1080/1750399X.2017.1359763
Li, D. (2004). Trustworthiness of think-aloud protocols in the study of translation processes. International Journal of Applied Linguistics, 14(3), 301–313. https://doi.org/10.1111/j.1473-4192.2004.00067.x
Likert, R. A. (1932). A technique for the measurement of attitudes. Archives of Psychology, 22(140), 1–55.
Litwin, M. (1995). How to measure survey reliability and validity. SAGE. https://doi.org/10.4135/9781483348957
Lubke, G. H., Dolan, C. V., Kelderman, H., & Mellenbergh, G. J. (2003). On the relationship between sources of within- and between-group differences and measurement invariance in the common factor model. Intelligence, 31, 543–566. https://doi.org/10.1016/S0160-2896(03)00051-5
McGorry, S. Y. (2000). Measurement in a cross-cultural environment: Survey translation issues. Qualitative Market Research, 3(2), 74–81. https://doi.org/10.1108/13522750010322070
Mellinger, C. D. (2015). On the applicability of Internet-mediated research methods to investigate translators’ cognitive behavior. Translation & Interpreting, 7(1), 59–71.
Mellinger, C. D., & Hanson, T. A. (2017). Quantitative research methods in translation and interpreting studies. Routledge. https://doi.org/10.4324/9781315647845
Mellinger, C. D., & Hanson, T. A. (2018). Interpreter traits and the relationship with technology and visibility. Translation and Interpreting Studies, 13(3), 366–392. https://doi.org/10.1075/tis.00021.mel
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741
Milfont, T. L., & Fischer, R. (2010). Testing measurement invariance across groups: Applications in cross-cultural research. International Journal of Psychological Research, 3(1), 111–121. https://doi.org/10.21500/20112084.857
Muñoz Martín, R. (2016). Of minds and men–computers and translators. Poznań Studies in Contemporary Linguistics, 52(2), 351–381. https://doi.org/10.1515/psicl-2016-0013
Muñoz Martín, R. (2017). Looking toward the future of cognitive translation studies. In J. W. Schwieter & A. Ferreira (Eds.), The handbook of translation and cognition (pp. 554–572). Wiley. https://doi.org/10.1002/9781119241485.ch30
Norman, G. (2010). Likert scales, levels of measurement and the ‘laws’ of statistics. Advances in Health Sciences Education, 15(5), 625–632. https://doi.org/10.1007/s10459-010-9222-y
Nunnally, J. C. (1978). Psychometric theory. McGraw-Hill.
Oakes, M. P., & Ji, M. (Eds.). (2012). Quantitative methods in corpus-based translation studies. John Benjamins. https://doi.org/10.1075/scl.51
O’Brien, S. (2009). Eye tracking in translation process research: Methodological challenges and solutions. In I. M. Mees, F. Alves, & S. Göpferich (Eds.), Methodology, technology, and innovation in translation process research: A tribute to Arnt Lykke Jakobsen (pp. 251–266). Samfundslitteratur.
O’Brien, S. (2013). The borrowers: Researching the cognitive aspects of translation. Target, 25(1), 5–17. https://doi.org/10.1075/target.25.1.02obr
O’Brien, S., & Saldanha, G. (2014). Research methodologies in translation studies. Routledge. https://doi.org/10.4324/9781315760100
Peters, G.-J. Y. (2014). The alpha and the omega of scale reliability and validity: Why and how to abandon Cronbach’s alpha and the route towards more comprehensive assessment of scale quality. European Health Psychologist, 16(2), 56–69.
Pöchhacker, F. (2005). From operation to action: Process-orientation in interpreting studies. Meta, 50(2), 682–695. https://doi.org/10.7202/011011ar
Reid, J. (1990). The dirty laundry of ESL survey research. TESOL Quarterly, 24(2), 323–338. https://doi.org/10.2307/3586913
Risku, H. (2012). Cognitive approaches to translation. In C. A. Chapelle (Ed.), The encyclopedia of applied linguistics (pp. 1–10). Wiley-Blackwell. https://doi.org/10.1002/9781405198431.wbeal0145
Risku, H. (2014). Translation process research as interaction research: From mental to socio-cognitive processes. MonTI Special Issue—Minding Translation, Special Issue 1, 331–353. https://doi.org/10.6035/MonTI.2014.ne1.11
Rönkkö, M., & Evermann, J. (2013). A critical examination of common beliefs about partial least squares path modeling. Organizational Research Methods, 16(3), 155–174. https://doi.org/10.1177/1094428112474693
Rubin, R. B., Rubin, A. M., Graham, E. E., Perse, E. M., & Seibold, D. R. (2011). Communication research measures II: A sourcebook. Routledge. https://doi.org/10.4324/9780203871539
Schaeffer, M., Huepe, D., Hansen-Schirra, S., Hofmann, S., Muñoz, E., Kogan, B., Herrera, E., Ibáñez, A., & García, A. (2020). The Translation and Interpreting Competence Questionnaire: An online tool for research on translators and interpreters. Perspectives, 28(1), 90–108. https://doi.org/10.1080/0907676X.2019.1629468
Schwarz, N. (2007). Cognitive aspects of survey methodology. Applied Cognitive Psychology, 21(2), 277–287. https://doi.org/10.1002/acp.1340
Shreve, G. M., & Angelone, E. (2010a). Translation and cognition. John Benjamins. https://doi.org/10.1075/ata.xv
Shreve, G. M., & Angelone, E. (2010b). Translation and cognition: Recent developments. In G. M. Shreve & E. Angelone (Eds.), Translation and cognition (pp. 1–13). John Benjamins. https://doi.org/10.1075/ata.xv.01shr
Shreve, G. M., Angelone, E., & Lacruz, I. (2018). Are expertise and translation competence really the same?: Psychological reality and the theoretical status of competence. In I. Lacruz & R. Jääskeläinen (Eds.), Innovation and expansion in translation process research (pp. 37–54). John Benjamins. https://doi.org/10.1075/ata.18.03shr
Smith, T. W. (2010). Survey across nations and cultures. In P. V. Marsden & J. D. Wright (Eds.), Handbook of survey research (pp. 733–763). Emerald.
Streiner, D. L. (2003). Being inconsistent about consistency: When coefficient alpha does and doesn’t matter. Journal of Personality Assessment, 80(3), 217–222. https://doi.org/10.1207/S15327752JPA8003_01
Van Bork, R., Rhemtulla, M., Waldorp, L. J., Kruis, J., Rezvanifar, S., & Borsboom, D. (2019). Latent variable models and networks: Statistical equivalence and testability. Multivariate Behavioral Research, 1–24. https://doi.org/10.1080/00273171.2019.1672515
Vogt, W. P., & Johnson, R. B. (2016). The SAGE dictionary of statistics and methodology (5th ed.). SAGE.
Warrens, M. J. (2015). On Cronbach’s alpha as the mean of all split-half reliabilities. In R. Millsap, D. Bolt, L. van der Ark, & W. C. Wang (Eds.), Quantitative psychology research (pp. 293–300). Springer. https://doi.org/10.1007/978-3-319-07503-7_18
Wicherts, J. M., & Dolan, C. V. (2010). Measurement invariance in confirmatory factor analysis: An illustration using IQ test performance of minorities. Educational Measurement: Issues and Practice, 29(3), 39–47. https://doi.org/10.1111/j.1745-3992.2010.00182.x
Wilcox, J. B., Howell, R. D., & Breivik, E. (2008). Questions about formative measurement. Journal of Business Research, 61(12), 1219–1228. https://doi.org/10.1016/j.jbusres.2008.01.010
Wirth, R. J., & Edwards, M. C. (2007). Item factor analysis: Current approaches and future directions. Psychological Methods, 12(1), 58–79. https://doi.org/10.1037/1082-989X.12.1.58
Zlomke, K. R. (2009). Psychometric properties of internet administered versions of Penn State Worry Questionnaire (PSWQ) and Depression, Anxiety, and Stress Scale (DASS). Computers in Human Behavior, 25, 841–843. https://doi.org/ 10.1016/j.chb.2008.06.003
Published
How to Cite
Issue
Section
License
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under the CC BY-NC 4.0 Deed that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal. The material cannot be used for commercial purposes.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
