Language and Vision

Language and vision, it seems, are intimately connected, an idea promoted by Swiss linguist Ferdinand de Saussure, who argued that the process of differentiation at work in the act of looking occurs with the emergence of language itself. Language is not a mould into which pre-existing ideas or concepts fit but a system of signs that actually constructs these ideas, dividing them up and attributing a sound to them. There are, then, no values outside of language but only those constituted by the system itself, in a signs negative difference from every other sign. Thus, we recognize a dog as a dog because it is not a cat, a bird, a jellyfish, and so on. Without language, Saussure asserts, ideas as well as sounds would be an indistinguishable continuum. And so with vision. Biologically, one might have the capacity to look, just has one has the ability to make noises, but this is different from the ability to make things intelligible, to make meanings. We have to learn to see because otherwise we could not make sense of the visual; the specular field would be the undifferentiated continuum described by Saussure.

There is a sense, therefore, in which looking is always a type of reading because it involves interpreting what is seen. (Julia Thomas, “Introduction,” Reading Images, 2001)

I find little in Saussure to support Julia Thomas’s reading. It is not so much that I disagree with her assertions, it’s just that Saussure seldom talks about vision at all. I’ve been going through the Course in General Linguistics in great detail hoping to see where such support might be marshaled, but it doesn’t appear to be there. I wish she would have referenced specific passages where she sees this connection. Instead, I am left with other distinctions in Saussure that seem more pertinent to my perception that applying his structuralist approach uncritically misses important distinctions between language and vision.

The parallel which Thomas seeks to expose is a concept of the visual as a form of “code”—the same distinction wrestled with by Roland Barthes, which he ultimately answered in the negative. Barthes claimed that images are “without code,” all the while proceeding to decode them in terms of their socially constructed denotations. The difference, I think, according to both Barthes and Thomas, is more a matter of degree. It is this aspect that Saussure deals with extensively in his lectures on the arbitrary nature of sign systems, as well as his distinction between motivated and unmotivated signs.

Saussure disputes the view taken by an American linguist, Whitney, who “regards languages as social institutions on exactly the same footing as other social institutions . . . Man, in his view, might well have chosen to use gestures, thus substituting visual images for sound patterns” (26). The definition of “language” offered by Saussure circumscribes and severs language from considerations of sensual apparatus. Language is “a system of distinct signs corresponding to distinct ideas” (26, emphasis mine). Saussure rapidly expands this into considerations of articulation and therefore, the sequential nature of language processing. The inevitable connection between language and institutions is elaborated on in successive lectures, but the central concern of Saussure is the separation of the “code” of language from the surrounding contexts of sensual processing and social currency as the most proper area of inquiry for linguistic study. The relationship between linguistics and sensual apparatus is such that linguistics cannot inform physiology (as Thomas suggests by saying “we have to learn to see because otherwise we could not make sense of the visual”), but rather physiology informs linguistics. The social relation, however, is reciprocal. This, according to Saussure, is what makes any inquiry into language difficult from conventional perspectives:

In the first place, there is the superficial view taken by the general public, which sees language merely as a nomenclature. This view stifles any inquiry into linguistic structure.

Then there is the viewpoint of the psychologist, who studies the mechanism of the sign in the individual. This is the most straightforward approach, but it takes us no further than the individual execution. It does not even take us as far as the linguistic sign itself, which is social by nature.

Even when due recognition is given to the fact that the sign must be studied as a social phenomenon, attention is restricted to those features of language which they share with institutions mainly established by voluntary decision. In this way, the investigation is diverted from its goal. It neglects those characteristics which belong only to semiological systems in general, and to languages in particular. For the sign always to some extent eludes control by the will, whether of the individual or of society: that is its essential nature even though it may not be obvious at first sight. (34)

The nature of inquiry into language is, in Saussure’s view, necessarily separate from considerations of sensual apparatus, separate from a purely psychological inquiry into reception, and separate from an inquiry into social phenomenon. Distinctions regarding the arbitrary nature of signs made by Saussure seem to be quite important to me, particularly the difference between motivated and unmotivated signs. But that will have to be continued in another post.

The main thing I struggle with is a means of separating the inquiry into visual sign systems from psychology and sociology in some meaningful way. In the case of photographs, they do not neatly map out as an “articulated” sign system. Images preserve components of social articulation, but are not by necessity articulated in and of themselves. This is the same problem that Barthes wrestled with, and I do not think it has been resolved so that it might be neatly explained, or productively reduced merely by saying that “language and vision” are intimately connected. The elusive nature of the sign is not easily reduced.