Processing of a Multi-representational Science Passage by ESL Learners

Nurjanah Mohd Jaafar, Siew Ming Thang


Verbal input is integral and is the core component in any language learning materials. As such, despite the incorporation of non-verbal input to facilitate language learning, language learners, more often than not, give attention more to the verbal components as opposed to the non-verbal ones. This paper reports on a study undertaken to examine 28 ESL undergraduates’ eye fixations when reading a science passage in English with an accompanying diagram. The data collection involves two stages. The first stage concerned the reading of the science passage and the second stage comprised short retrospective interviews conducted to explore the reasons behind the learners’ processing. Results of the eye tracking analysis indicate that similar to the processing of language learning materials, the ESL learners’ processing of the science passage was also mainly focused on the text (henceforth, “heavily text-based”). Findings from the retrospective interviews indicate that although a heavily text-based processing strategy was evident, most of the learners found the graphic input helpful. Analysis of the interviews also revealed that some learners were not aware of the strategies that they employed during the reading process. Although the eye tracking data imply that the ESL learners lacked the awareness and strategy to read and process multi-representational science texts in English effectively, the interviews suggest that they were aware of the potential and benefits of processing the accompanying diagram for overall comprehension. Effective design principles for multi-representational materials are suggested to promote more strategic processing among learners.


multiple representations; ESL learners; processing science texts; eye tracking; interviews

Full Text:



Bartholomé, T., & Bromme, R. (2009). Coherence formation when learning from text and pictures: What kind of support for whom? Journal of Educational Psychology. 101(2), 282–293.

Bisson, M.-J., Van Heuven, W. J. B., Conklin, K., & Tunney, R. J. (2014). The role of verbal and pictorial information in multi-modal incidental acquisition of foreign

language vocabulary. The Quarterly Journal of Experimental Psychology. 68(7), 1306-1326.

Carpenter, S. K., & Olson, K. M. (2012). Are pictures good for learning new vocabulary in a foreign language? only if you think they are not. Journal of Experimental

Psychology: Learning, Memory, and Cognition. 38(1), 92–101.

Hannus, M., & Hyönä, J. (1999). Utilization of illustrations during learning of science textbook passages among low- and high-ability children. Contemporary

Educational Psychology. 24(2), 95–123.

Hegarty, M., & Just, M. A. (1993). Constructing mental models of machines from text and diagrams. Journal of Memory and Language. 32(6), 717–742.

Hochpöchler, U., Schnotz, W., Rasch, T., Ullrich, M., Horz, H., McElvany, N., & Baumert, J. (2013). Dynamics of mental model construction from text and graphics.

European Journal of Psychology of Education. 28(4), 1105–1126.

LeDoux, J. E. (1994). Emotion, memory and the brain. Scientific American. 270(6), 50–57.

Levin, J. R., Anglin, G. J., & Carney, R. N. (1987). On empirically validating functions of pictures in prose. The Psychology of Illustration. 1, 51–86.

Lohse, G. L., Biolsi, K., Walker, N., & Rueter, H. H. (1994). A classification of visual representations. Communications of the ACM. 37(12), 36–49.

Mason, L., Tornatora, M. C., & Pluchino, P. (2013). Do fourth graders integrate text and picture in processing and learning from an illustrated science text? Evidence

from eye-movement patterns. Computers & Education. 60(1), 95–109.

Mason, L., Tornatora, M., & Pluchino, P. (2015). Integrative processing of verbal and graphical information during re-reading predicts learning from illustrated text: an

eye-movement study. Reading and Writing. 28(6), 851–872.

Mohd Yusof, S., Mohd Lazim, Z., Salehuddin, K., & Mohamad Shahimin, M. (2020). Graphic novels: Understanding how fifth graders read literary text through eye

movement analysis. Kritika Kultura. 33/34, 388–427.

Pellicer-Sanchez, A., Tragant, E., Conklin, K., Rodgers, M., Llanes, A., & Serrano, R. (2018). L2 reading and reading-while-listening in multimodal learning conditions:

An eye-tracking study. ELT Research Papers 18.01. British Council.

Perez, M. M., Van Den Noortgate, W., & Desmet, P. (2013). Captioned video for L2 listening and vocabulary learning: A meta-analysis. System. 41(3), 720–739.

Plass, J. L., & Jones, L. C. (2005). Multimedia learning in second language acquisition. In R. E. Mayer, (Ed.). The Cambridge Handbook of Multimedia Learning (pp.

–488). New York: Cambridge University Press.

Rau, M. A. (2017). Conditions for the effectiveness of multiple visual representations in enhancing STEM learning. Educational Psychology Review. 29(4), 717–761.

Rayner, K. (2009). Eye movements and attention in reading, scene perception, and visual search. The Quarterly Journal of Experimental Psychology. 62(8), 1457–

doi: 10.1080/17470210902816461

Renkl, A., & Scheiter, K. (2015). Studying visual displays: How to instructionally support learning. Educational Psychology Review. 29(3), 599-621.

Scheiter, K., & Eitel, A. (2017). The use of eye tracking as a research and instructional tool in multimedia learning. In C. A. Was, F. J. Sansosti, & B. J. Morris, (Eds.).

Eye-tracking Technology Applications in Educational Research (pp. 143–164). Hershey, PA: IGI Global.

Scheiter, K., Schüler, A., & Eitel, A. (2017). Learning from multimedia: Cognitive processes and instructional support. In S. Schwan & U. Cress (Eds.). The Psychology

of Digital Learning Constructing, Exchanging, and Acquiring Knowledge with Digital Media (pp. 1–19). Cham: Springer.

Schmidt-Weigand, F., Kohnert, A., & Glowalla, U. (2010). A closer look at split visual attention in system- and self-paced instruction in multimedia learning. Learning

and Instruction. 20(2), 100–110.

Schroeder, S., Richter, T., McElvany, N., Hachfeld, A., Baumert, J., Schnotz, W., Horz, H., & Ullrich, M. (2011). Teachers’ beliefs, instructional behaviors, and students’

engagement in learning from texts with instructional pictures. Learning and Instruction. 21(3), 403–415.

Soh, O. K. (2017). Processing academic science reading texts through context effects: Evidence from eye movements. EURASIA Journal of Mathematics, Science and

Technology Education. 13(3), 771–790.

Stieff, M., Hegarty, M., & Deslongchamps, G. (2011). Identifying representational competence with multi-representational displays. Cognition and Instruction, 29(1),


Sulaiman, N. A., Salehuddin, K., & Khairudin, K. (2020). Reading English academic texts: Evidence from ESL undergraduates’ eye movement data. 3L: Language,

Linguistics, Literature®. 26(1), 60–78

Tengku, Fariqul Haq & Salehuddin, K. (2020). Pembacaan ungkapan literal dan metafora Melayu: Satu kajian penjejakan mata (Reading Malay literal and metaphorical

expressions: An eye tracking study). GEMA Online® Journal of Language Studies. 20(2), 18–35.

Tragant Mestres, E., & Pellicer-Sánchez, A. (2019). Young EFL learners’ processing of multimodal input: Examining learners’ eye movements. System. 80, 212–223.

Warid Mihat, Hazita Azman, & Soh, O. K. (2018). Bringing reading research in multilingual Nusantara into a new direction through eye-tracking. Journal of Nusantara

Studies (JONUS). 3(2), 107–123.

Warren, P., Boers, F., Grimshaw, G., & Siyanova-Chanturia, A. (2018). The effect of gloss type on learners’intake of new words during reading: Evidence from eye-

tracking. Studies in Second Language Acquisition. 40(4), 883–906.

Yoo, D. G., & Lee, J. (2013). Using contextual information in learners’ spoken language communication: An eye-tracking study. GEMA Online® Journal of Language

Studies.13(2), 5–20.



  • There are currently no refbacks.




eISSN : 2550-2131

ISSN : 1675-8021