Applying semantic concepts to the media assigment problem in multi-media communication.

Our long term goal is an understanding of human communication in terms which would provide the basis for rational design. The kernel would be a theory of the cognitive consequences of allocating the same information to different media and modalities, based on the user's information processing characterised in computational terms. Our theory of the cognitive consequences of media/modality allocation starts from an analysis of differences in logical expressiveness of graphical and linguistic representations (Stenning \& Oberlander (1994, 1995)). This semantic approach requires conceptualisations of {\it medium} and {\it modality} that can be related to representation systems. We propose that media are the physical/perceptual aspects of representations; modalities are classes of interpretation function which map media onto meanings. These interpretations of the terms contrast with existing HCI usage. The curtailments of abstraction observed in graphics arise from interactions between medium and modality. Graphical modalities are distinguished from sentential languages by the nature of their interpretation functions. A hierarchy of expressiveness of interpretations of graphics is defined, and compared with interpretations of sentential languages. Using the expressiveness of representations to predict their cognitive properties also requires reference to the availability of constraints of their interpretation to users. Further contrasts between graphics and language emerge in the availability of their constraints. Three example domains of graphical representations are analysed from this perspective---matrix graphics; logic diagrams; and semantic networks. Some empirical evidence of the usability of these notations is reviewed as evidence that the proposed conceptualisation offers powerful generalisations.