Research Article


DOI :10.26650/sdsl2020-0009   IUP :10.26650/sdsl2020-0009    Full Text (PDF)

Multimodality in Socio-cultural Contexts: To Talk with Pictures, to Understand Pictures

Valeria ChernyavskayaLudmila Kulikova

The present paper is inspired by the challenges posed by multimodal/multi-semiotic resources in the investigation of communication. We focus on the visual resource as a socially and culturally shaped semiotic resource of making meaning and discuss how images work in different socio-cultural contexts. The interpretative complexity of the notions of ‘culture’ and ‘socio-cultural context’ is explained by looking at images as they represent information from a socio-cultural environment. The theoretical framework of the paper was rooted in Halliday, Kress and Leeuwen’s theory of social semiotics, in the concepts of multimodal communication and text linguistic view of the language-image-link according to Stöckl, Schmiz, Sandig, and in Kuße’s discourse sensitive cultural linguistics. The questions under discussion are how we communicate through visual images, and how communicative actors produce and interpret meanings through pictures. We use the multimodal texts from modern German and Russian discourse and discuss the role of the visual image in shaping our socio-cultural experience and knowledge. The paper argues for more sensitivity in respect of the expansion of visual images with regard to multimodal competence. The latter plays a crucial role in exploring the ideological and culturally specific implicatures behind the visual images.

DOI :10.26650/sdsl2020-0009   IUP :10.26650/sdsl2020-0009    Full Text (PDF)

Multimodalität im sozio-kulturellen Kontext: mit Bildern reden, Bilder verstehen

Valeria ChernyavskayaLudmila Kulikova

Der vorliegende Beitrag will zur Diskussion über die Multimodalität beitragen mit dem Ziel, auf die visuelle Ressource als notwendige Voraussetzung für das Verstehen hinzuweisen. Er fokussiert die Fragen, wie ein Bild im Diskurs wirksam wird, wie Kommunikationsteilnehmer in einem bestimmten Kulturraum Bedeutungen mithilfe einer visuellen Ressource ausdrücken und interpretieren, welche gesellschaftlich sinnvollen Bedeutungen mit dem im Text enthaltenen Bild verbunden sind. Die Auseinandersetzung mit dem Visuellen bietet einen ‚Access Point‘ zur Kontextualisierung und zugleich eine Evidenzbasis in der theoretischen Begründung dessen, wie das gemeinsame, in der kulturellen Praxis geteilte Wissen entsteht und weiter funktioniert. Die methodologischen Grundlagen werden durch die soziale Semiotik von M. A. K. Halliday, G. Kress, van Leeuwen festgelegt, beruhen auf den theoretischen Vorstellungen von Multimodalität von H. Stöckl, H. Schmiz, B. Sandig, folgen der diskurssensitiven kulturwissenschaftlichen Linguistik nach H. Kuße. Fallbeispiele aus der deutschen und russischen kommunikativen Praxis werden analysiert. Die Expansion des Bildes in sozialen Praktiken erfordert von den Teilnehmern der Kommunikation reflektierte visuelle Erfahrung und eine multimodale Kompetenz. Eine solche Kompetenz beinhaltet die Fähigkeit, ideologische, politische, kulturspezifische Implikationen von Bildern abzuleiten und zu interpretieren, die Werteinstellungen zu verstehen, die hinter der visuellen Gestaltung der Bedeutungen stehen.


EXTENDED ABSTRACT


In the present paper, we will engage with the important challenges posed by multimodal/multi-semiotic resources in communication. Multimodality is considered as a cultural technique and a communicative competence that guarantees mutual transparence and explicitness for interactors. We focus on the visual resource as a socially and culturally shaped semiotic resource of making meaning. In this framework the paper addresses the way knowledge and values are constructed through images as well as language usages. The paper attempts to point out a socio-cultural perspective on the understanding of multimodal texts. The paper discusses how images, as part of language-image-links, work in different socio-cultural contexts and shape our sociocultural experience and knowledge. Thus, the goal is to look at images as they represent information from a sociocultural environment. The interpretative complexity of the notions of ‘culture’ and ‘sociocultural context’ is explained. The analysis follows the position of discourse oriented cultural linguistics that presumes the situational and socio-cultural framework both for the production and the reception of utterances/texts. Methodology. Several approaches to multimodal communication have been developed. The theoretical framework of this paper was found in the theory of social semiotics according to Halliday, Kress, van Leeuwen, and in the concepts of multimodal communication and text linguistic view of the language-image-links according to Stöckl, Schmiz, Sandig and others. The present analysis is also in line with Kuße’s discourse sensitive cultural linguistics (diskurssensitive kulturwissenschaftliche Linguistik). From this theoretical perspective we show that there is a shift from the “linguist’s pride in language” (in Stöckl’s words) towards multimodality and visual methodology in meaning construction. We follow social semiotics in looking at how verbal and nonverbal signs (semiotics resources) are used in certain social practices. It has been emphasized in social semiotics how semiotic practices are driven by socio-cultural environments and how they are embedded in them and emerge from them. The paper looks at multimodal texts and images as they are context sensitive. Findings and Results. The theoretical reflections and text analytical observations show the explanatory charge of multimodal studies. In the provided analysis we tackle the multimodal texts and cases in point from modern German and Russian socio-cultural practice. Different types of image-language-links are considered, following the basic observations in text linguistics, namely complement and discrepancy or contrast. Multimodal metaphors and metonymy are also discussed. Visual images are iconic and also indexical (i.e. metonymic) in terms of their representation of abstract concepts and objects/events in their socio-cultural embeddedness. The authors presume that pictures should be understood in context and in combination with language. Linguistic and visual resources cooperate as they interrelate concepts. From this point of view, images are seen as an additional access point to explain meaning making process within its cognitive dimension. Next, multimodal texts, and images as a part of these texts, allow us to observe how socio-cultural work is performed. This, in turn, leads to the conclusion, that images are context embedded. In other words, they present information from the social environment. A visual image is not just an information vehicle. It is a constituent element of dynamic sense-making process. The cases under discussion show that multimodal texts could act as wrong indexicals and may cause wrong contextualization, if they are misplaced in contexts, i.e. if the context of text reception does not fit the context of text production. We can make sense of imagelanguage-texts if they fit the context. The present paper argues for more sensitivity in respect of the expansion of visual images and with regard to multimodal competence. The multimodal competence plays a crucial role in exploring the ideological and culturally specific implicatures behind the visual images. A potentially wide perspective integrating verbal text and picture within different social practices should also be considered as helpful in this framework.


PDF View

References

  • Chernyavskaya, V. (2014). Kommunikation in der Wissenschaft: Ein Plädoyer für Mehrsprachigkeit. In Alman Dili ve Edebiyatı Dergisi – Studien zur deutschen Sprache und Literatur, 2, 5-15. google scholar
  • Chernyavskaya, V. (2016). Multimodality versus “Logocentrism” in Persuasion. In Filologicheskie Nauki – Nauchnye Doklady Vysshei Shkoly – Philological Sciences – Scientific essays of Higher Education, 2, 3-10. google scholar
  • Ehlich, K. (1998). Medium Sprache. In Strohner H., Sichelschmidt L., Hielscher M. (Hrg.). Medium Sprache (S. 9-21). Frankfurt/M.: Lang. google scholar
  • Gumperz, J. (2002). Sharing Common Ground. In Keim I., Schütte W., (Hrsg.). Soziale Welten und kommunikative Stile (S. 47-55). Tübingen: Narr. google scholar
  • Feng, W. D. (2017). Metonymy and visual representation: towards a social semiotic framework of visual metonymy. In Visual Communication. 16 (4). 441-466. google scholar
  • Kress, G. (2010). Multimodality: A social semiotic approach to contemporary communication. Abingdon, UK: Routledge. google scholar
  • Kress, G. & van Leeuwen, T. (2001). Multimodal discourse. The modes and media of contemporary communication. London: Arnold. google scholar
  • Kress, G. & van Leeuwen, T. (1996 1st ed.; 2006 2nd ed.). Reading images. The grammar of visual design. London: Routledge. google scholar
  • Kulikova, L. V. (2019). Intercultural mediation: a discourse-analytical perspective. Voprosy Kognitivnoy Lingvistiki. № 3. 26-35. DOI: 10.20916/1812-3228-2019-3-26-35 google scholar
  • Kulikova, L. V. & Detinko, J. I. (2020). Discursive Construction of “Others” in the Semiotic Space of Political Communication. In E. Malyuga (Hrsg.), Functional Approach to Professional Discourse Exploration in Linguistics (S. 199-230). Singapur: Springer. google scholar
  • Kuße, H. & Chernyavskaya, V. E. (2019). Culture: Towards its explanatory charge in discourse linguistics. In Vestnik Sankt-Peterburgskogo Universiteta. Yazyk i Literatura. № 3 (16). 444-462. google scholar
  • Kuße, H. (2012). Kulturwissenschaftliche Linguistik. Eine Einführung. Göttingen: Vandenhoeck & Ruprecht. google scholar
  • Molodychenko, Е. N. (2015). Axiological Dimension in the Discourse of Consumerism: The Role of Evaluative Language in the lifestyle genre. Vestnik Tomskogo gosudarstvennogo universiteta. Filologiya – Tomsk State University Journal of Philology. № 6 (38). 55-66. DOI: 10.17223/19986645/38/5. google scholar
  • Molodychenko, E. N. (2017). Identity and Discourse: From Social Theory to Practice of Discourse Analysis. In St. Petersburg State Polytechnic University Journal. Humanities and Social sciences. Vol. 8. №3. 122–133. DOI: 10.18721/Jhss.8312. google scholar
  • Nöth, W. (2000). Der Zusammenhang von Text und Bild. In Brinker, K., Antos, G., Heinemann, W. & Sager S. (Hrsg.) Text- und Gesprächslinguistik. Ein internationales Handbuch zeitgenössischer Forschung, 1. Halbbd. (S. 489496). N.Y.: de Gruyter. google scholar
  • Sandig, B. (2006). Textstilistik des Deutschen. Berlin, N.Y.: de Gruyter. google scholar
  • Schmitz, U. (2004). Sprache in modernen Medien. Einführung in Tatsachen und Theorien, Themen und Thesen. Berlin: Erich Schmidt. google scholar
  • Spitzmüller, J. (2017). ‚Kultur‘ und das ‚Kulturelle‘: Zur Reflexivität eines begehrten Begriffs. In Zeitschrift für Angewandte Linguistik. 67, 3-23. google scholar
  • Stöckl, H. (2009). The language-image-text – Theoretical and analytical inroads into semiotic complexity. In Arbeiten aus Anglistik und Amerikanistik. №34 (2). 203-226. google scholar
  • Stöckl, H. (2010). Sprache-Bild-Texte lesen. Bausteine zur Methodik einer Grundkompetenz. In Diekmannshenke, H., Klemm, M., Stöckl, H. (Hrg.), Bildlinguistik (S. 43-70). Berlin: Erich-Schmidt. google scholar
  • Şenöz-Ayata, C. & Atasoy, İ. (2019). Ein interkultureller Blick auf Multimodalität in ausgewählten deutschen, angloamerikanischen, italienischen und türkischen Kosmetikwerbeanzeigen. Alman Dili ve Edebiyatı Dergisi - Studien zur deutschen Sprache und Literatur, 41, 1-23. https://doi.org/10.26650/sdsl2019-0004 google scholar
  • Tchernyavskaia, V. & Zakaryan, S. (2014). Cultural marked Texts: Intercultural Dimension of Text Linguistics. In Filologicheskie Nauki – Nauchnye Doklady Vysshei Shkoly – Philological Sciences – Scientific essays of Higher Education. 2. 10-16. google scholar
  • Widdowson, H. G. (2004). Text, context, pretext: Critical issues in discourse analysis. (language in society). Wiley Online Library. google scholar

Citations

Copy and paste a formatted citation or use one of the options to export in your chosen format


EXPORT



APA

Chernyavskaya, V., & Kulikova, L. (2020). Multimodality in Socio-cultural Contexts: To Talk with Pictures, to Understand Pictures. Studien zur deutschen Sprache und Literatur, 0(43), 19-38. https://doi.org/10.26650/sdsl2020-0009


AMA

Chernyavskaya V, Kulikova L. Multimodality in Socio-cultural Contexts: To Talk with Pictures, to Understand Pictures. Studien zur deutschen Sprache und Literatur. 2020;0(43):19-38. https://doi.org/10.26650/sdsl2020-0009


ABNT

Chernyavskaya, V.; Kulikova, L. Multimodality in Socio-cultural Contexts: To Talk with Pictures, to Understand Pictures. Studien zur deutschen Sprache und Literatur, [Publisher Location], v. 0, n. 43, p. 19-38, 2020.


Chicago: Author-Date Style

Chernyavskaya, Valeria, and Ludmila Kulikova. 2020. “Multimodality in Socio-cultural Contexts: To Talk with Pictures, to Understand Pictures.” Studien zur deutschen Sprache und Literatur 0, no. 43: 19-38. https://doi.org/10.26650/sdsl2020-0009


Chicago: Humanities Style

Chernyavskaya, Valeria, and Ludmila Kulikova. Multimodality in Socio-cultural Contexts: To Talk with Pictures, to Understand Pictures.” Studien zur deutschen Sprache und Literatur 0, no. 43 (Oct. 2024): 19-38. https://doi.org/10.26650/sdsl2020-0009


Harvard: Australian Style

Chernyavskaya, V & Kulikova, L 2020, 'Multimodality in Socio-cultural Contexts: To Talk with Pictures, to Understand Pictures', Studien zur deutschen Sprache und Literatur, vol. 0, no. 43, pp. 19-38, viewed 1 Oct. 2024, https://doi.org/10.26650/sdsl2020-0009


Harvard: Author-Date Style

Chernyavskaya, V. and Kulikova, L. (2020) ‘Multimodality in Socio-cultural Contexts: To Talk with Pictures, to Understand Pictures’, Studien zur deutschen Sprache und Literatur, 0(43), pp. 19-38. https://doi.org/10.26650/sdsl2020-0009 (1 Oct. 2024).


MLA

Chernyavskaya, Valeria, and Ludmila Kulikova. Multimodality in Socio-cultural Contexts: To Talk with Pictures, to Understand Pictures.” Studien zur deutschen Sprache und Literatur, vol. 0, no. 43, 2020, pp. 19-38. [Database Container], https://doi.org/10.26650/sdsl2020-0009


Vancouver

Chernyavskaya V, Kulikova L. Multimodality in Socio-cultural Contexts: To Talk with Pictures, to Understand Pictures. Studien zur deutschen Sprache und Literatur [Internet]. 1 Oct. 2024 [cited 1 Oct. 2024];0(43):19-38. Available from: https://doi.org/10.26650/sdsl2020-0009 doi: 10.26650/sdsl2020-0009


ISNAD

Chernyavskaya, Valeria - Kulikova, Ludmila. Multimodality in Socio-cultural Contexts: To Talk with Pictures, to Understand Pictures”. Studien zur deutschen Sprache und Literatur 0/43 (Oct. 2024): 19-38. https://doi.org/10.26650/sdsl2020-0009



TIMELINE


Submitted08.04.2020
Accepted13.05.2020
Published Online22.06.2020

LICENCE


Attribution-NonCommercial (CC BY-NC)

This license lets others remix, tweak, and build upon your work non-commercially, and although their new works must also acknowledge you and be non-commercial, they don’t have to license their derivative works on the same terms.


SHARE




Istanbul University Press aims to contribute to the dissemination of ever growing scientific knowledge through publication of high quality scientific journals and books in accordance with the international publishing standards and ethics. Istanbul University Press follows an open access, non-commercial, scholarly publishing.