Search

This paper adopts a construction-grammar approach to multimodal meaning. We provide a detailed analysis of the Before-After-construction used frequently in advertisements, cartoons and Internet memes. We demonstrate that parts of its generic ‘caused-change’ meaning is compositional, and rendered independently from what is overtly expressed by concrete instances of the pattern. The latter hence build on an abstract multimodal construction whose form elements are paired idiosyncratically with meaning, just like linguistic constructions proper. We show that non-standard instances of the Before-After-construction represent deviations based on a systematized standard Before-After-construction. Finally, we argue that the Before-After-construction belongs to a broader inheritance hierarchy of two-image multimodal construction types, while also providing one amongst several options to convey caused-change. Altogether, we demonstrate that multimodal expressions instantiate similar properties as unimodal expressions both across form and meaning.

In a usage-based framework, variation is part and parcel of our linguistic experiences, and therefore also of our mental representations of language. In this article, we bring attention to variation as a source of information. Instead of discarding variation as mere noise, we examine what it can reveal about the representation and use of linguistic knowledge. By means of metalinguistic judgment data, we demonstrate how to quantify and interpret four types of variation: variation across items, participants, time, and methods. The data concern familiarity ratings assigned by 91 native speakers of Dutch to 79 Dutch prepositional phrases such asin de tuin‘in the garden’ andrond de ingang‘around the entrance’. Participants performed the judgment task twice within a period of one to two weeks, using either a 7-point Likert scale or a Magnitude Estimation scale. We explicate the principles according to which the different types of variation can be considered information about mental representation, and we show how they can be used to test hypotheses regarding linguistic representations.

This article addresses visual optimal innovations such as memes, advertising images and editorial cartoons which take Michelangelo’s fresco The Creation of Adam as input, and rework it so that novel meaning is created that takes the meaning of the input as base. The authors focus on the cognitive and structural aspects of these kinds of visual stimulus. They argue that visual optimal innovations are aesthetically rewarding owing to how they invite meaning construction, and they further demonstrate that these aspects of visual communication are encoded in human memory to make up entries in a ‘visual lexicon’. The main questions addressed are: What graphic procedures are employed to create such stimuli, and how are they structured? How are existing and novel meanings evoked by visual optimal innovations, and how are they aligned? How are visual optimal innovations processed and interpreted? And, finally, how is all this knowledge – structural and conceptual – cognitively represented?

Argument diagrams are schematic representations of claim-reason complexes, displaying boxes with standpoints and arguments, and lines or arrows to relate these boxes. In four experiments, we investigate the metaphor framing effect by investigating how the perceptual cues size and number of arguments influence evaluations of arguments. In the first experiment, we looked at the effect of size and number of boxes independent of content, using an empty argument diagram. We asked participants to imagine there were arguments in the boxes and decide whether they would choose in favor or against. Participants chose according to hypotheses: the side that had larger boxes, the side that had more boxes, or the side that had both larger and more boxes. In the other experiments, the diagrams contained real standpoints and arguments. Although content was very important in judging arguments, the effect of perceptual cues was still present. We found that mainly the number of arguments influenced decisions, while size painted a more complicated picture. We discuss these results in the light of argumentation and persuasion for learning.

Emoji have become a prominent part of interactive digital communication. Here, we ask the questions: does a grammatical system govern the way people use emoji; and how do emoji interact with the grammar of written text? We conducted two experiments that asked participants to have a digital conversation with each other using only emoji (Experiment 1) or to substitute at least one emoji for a word in the sentences (Experiment 2). First, we found that the emoji-only utterances of participants remained at simplistic levels of patterning, primarily appearing as one-unit utterances (as formulaic expressions or responsive emotions) or as linear sequencing (for example, repeating the same emoji or providing an unordered list of semantically related emoji). Emoji playing grammatical roles (i.e., ‘parts-of-speech’) were minimal, and showed little consistency in ‘word order’. Second, emoji were substituted more for nouns and adjectives than verbs, while also typically conveying nonredundant information to the sentences. These findings suggest that, while emoji may follow tendencies in their interactions with grammatical structure in multimodal text-emoji productions, they lack grammatical structure on their own.

While theories on predictive processing posit that predictions are based on one’s prior experiences, experimental work has effectively ignored the fact that people differ from each other in their linguistic experiences and, consequently, in the predictions they generate. We examine usage-based variation by means of three groups of participants (recruiters, job-seekers, and people not (yet) looking for a job), two stimuli sets (word sequences characteristic of either job ads or news reports), and two experiments (a Completion task and a Voice Onset Time task). We show that differences in experiences with a particular register result in different expectations regarding word sequences characteristic of that register, thus pointing to differences in mental representations of language. Subsequently, we investigate to what extent different operationalizations of word predictability are accurate predictors of voice onset times. A measure of a participant’s own expectations proves to be a significant predictor of processing speed over and above word predictability measures based on amalgamated data. These findings point to actual individual differences and highlight the merits of going beyond amalgamated data. We thus demonstrate that is it feasible to empirically assess the variation implied in usage-based theories, and we advocate exploiting this opportunity.

Visual information contributes fundamentally to the process of object categorization. The present study investigated whether the degree of activation of visual information in this process is dependent on the contextual relevance of this information. We used the Proactive Interference (PI-release) paradigm. In four experiments, we manipulated the information by which objects could be categorized and subsequently be retrieved from memory. The pattern of PI-release showed that if objects could be stored and retrieved both by (non-perceptual) semantic and (perceptual) shape information, then shape information was overruled by semantic information. If, however, semantic information could not be (satisfactorily) used to store and retrieve objects, then objects were stored in memory in terms of their shape. The latter effect was found to be strongest for objects from identical semantic categories.

In principle, verbal and image languages have different ways of coding conceptual content. Moreover, there is no reason to believethat both modes, the linguistic and the visual, can convey identical contents, and indeed, linguists have claimed that images are not suitedfor expressing the meaning of certain linguistic categories, like negation. As the linguistic literature argues convincingly, in naturallanguage a distinction between negation and denial is justified. Employing insights in visual communication and cognition science, thispaper explores the possibilities for visually expressing negation and/or denial. At the hand of both the analysis and an empirical pilot studyof a set of advertisements, we come up with a positive answer to the title question: yes, pictures can say ‘no’.

This article examines the structure and processing of editorial cartoons appropriating Joe Rosenthal's famous Iwo Jima photograph in view of the theory of conceptual blending (Fauconnier & Turner, 2002). The main claim argued is that Blending theory accounts for complex processes of meaning constructing provoked by editorial cartoons that invoke several conceptual domains to define and evaluate their topic, Iwo Jima cartoons being cases in point. The second section introduces the relevant aspects of Blending theory: mental spaces and types of blending networks. The third section addresses the Iwo Jima image as one particular input space; a moral criterion to which the selected sociopolitical topic of the cartoon is compared and judged. The fourth section isolates four types of graphic operations blending the Iwo Jima image with topical entities. The fifth section demonstrates how various kinds of blends give rise to either positive or negative evaluations. The final section argues that the study's scope o...

In order to interpret novel metaphoric relations, we have to construct ad hoc categories under which the metaphorically related concepts can be subsumed. Shape is considered to be one of the primary vehicles of object categorization. Accordingly, shape might play a prominent role in interpreting visual metaphors (i.e., two metaphorically related objects depicted in one visual array). This study explores the role of object shape in visual metaphor interpretation of 10- to 12-year-olds. The experiment shows that participants can produce more correspondences between similarly shaped objects as compared to dissimilarly shaped objects and that they need less thinking time to do so. These findings suggest that similarity in shape facilitates the process of interpreting visual metaphors.

Comparing objects is a process necessary to cognitive tasks involving categorization. Shape is considered one of the primary vehicles for object categorization. We hypothesize that similarity in shape facilitates finding conceptual correspondences between objects, both for objects stemming from the same and from different conceptual categories. In the latter case, the comparison process requires the construction of an ad hoc category, which is also required when interpreting visual metaphors. We used three experimental tasks to investigate the role of shape in comparing objects: a similarity judgment task, a similarity rating task, and a production task. The results of our experiments support the hypothesis that an essential component of visual metaphor processing—comparing objects stemming from disparate conceptual domains—is positively affected by similarity in the objects' shape.

Verbal and Visual Rhetoric in a Media World. H van Belle, P. Gillaerts, B. van Gorp, D. van de Mieroop & K. Rutten (2013). Leiden: Leiden University Press

We investigated the relationship between familiarity, perceived ease of use, and attractiveness of graph designs in two target groups: experts and laymen in design. In the first study, we presented them with a variety of more or less common graph designs and asked them without any additional task to evaluate their familiarity, attractiveness, and perceived ease of use. They judged the familiarity and ease of use of the graphs similarly, but they differed in their attractiveness judgments. Familiarity and perceived ease of use appeared to predict attractiveness, but stronger for laymen than for designers. Laymen are attracted to designs they perceive as familiar and easy to use. Designers are attracted to designs between familiar and novel. In the second study, we asked designers and laymen to first perform an information retrieval task with the same graphs and then rate their attractiveness. Laymen’s appreciations remained the same, but the designers’ judgments of attractiveness were different from those in study 1. Correlational analyses suggest that their attractiveness judgments after use were affected not by actual usability but by perceived ease of use of the graphs.

In this paper it is argued that, contrary to computational models of language production, in the production system grammatical knowledge takes the form of conventionalized declarative schemes. Such schemes can be identified as a particular function word and an obliged element, for instance, a noun and a determiner. The argument is based on a particular pause pattern observed written language production. A cognitive linguistic account of the notion 'grammatical scheme' is given through a dicussion of Langacker's Usage based model of linguistic knowledge and the 'mental grammar'.

In our article we describe the functions of so-called 'text plans1 in two different processes of text production: (1) dictating business letters and (2) writing governmental policy issue papers. Text plans are defined as: abstract, schematic representations of the text a writer/dictator wants to convey. We look upon dictating business letters and writing policy issue papers as two writing tasks. In our article we explore the differences between the two tasks in terms of the constraints the dictator and the policy writer should meet. Our research makes clear that dictators strive for first-time-final versions and that planning the text is a necessary requirement for effective dictating. While dictating, the dictator follows the exact line of his plan. Writing policy issue papers, however, is mainly a collaborative writing task. Policy issue papers are written in groups which strive for internal consensus about the issues (and the text) at stake. Text plans are uses by policy writers to divide tasks, to guarantee cohesion between the different contributions and as the first negotiation object.

In the Netherlands, as in most other European countries, closed captions for the deaf summarize texts rather than render them verbatim. Caption editors argue that in this way television viewers have enough time to both read the text and watch the program. They also claim that the meaning of the original message is properly conveyed. However, many deaf people demand verbatim subtitles so that they have full access to all original information. They claim that vital information is withheld from them as a result of the summarizing process. Linguistic research was conducted in order (a) to identify the type of information that is left out of captioned texts and (b) to determine the effects of nonverbatim captioning on the meaning of the text. The differences between spoken and captioned texts were analyzed on the basis of on a model of coherence relations in discourse. One prominent finding is that summarizing affects coherence relations, making them less explicit and altering the implied meaning.

This chapter discusses some aspects concerning the cognitive status of pauses in research on discourse production. It starts with a concise review of some of the ‘canonical’ studies on language production in which the pause analytical methodology is adopted. Section 2 discusses methods of collecting pause data and constructing pause databases. Section 3 addresses one key issue: The empirical status of pauses. First, it is shown that the relation between pauses and cognitive processes in discourse production can be maintained by looking at what happens in the immediate neighbourhood of pauses. Based on an analysis of actual transcripts, four kinds of pauses are distinguished: Pauses signalling retrieving, pauses signalling monitoring and pauses signalling repairing processes. Secondly, this section discusses how pause time variances can be interpreted in terms of underognitive processes, and the section concludes with a discussion of how pauses are related to text structural characteristics. The fourth section discusses various statistical methods for analysing pause data, showing the kind of research questions that can be addressed by each method.

This article identifies the structural and conceptual aspects of a visual construction often used in advertisements to establish a metaphoric or associative relation, that is, symmetric object alignment (SOA). It offers an account of the formal ingredients of SOA, which fall into two groups: object-constitutive factors (like size, shape, and color) and object-depictment factors (like perspective, orientation, and distance from viewing point). Both factors allow us to treat SOA as a visual rhetorical scheme. We also discuss how SOA relates to conceptual structure, metaphoric or otherwise. Four types of relations are distinguished, based on the way in which perceptual and conceptual characteristics interact. Finally, we discuss the way in which these characteristics relate to the processing of such visual messages. One depictment factor will be focused on in particular: one-to-one versus one-to-many alignments. An attempt is made to connect these 2 patterns to the much-discussed topic in verbal metaphor the...