The term connectionism has been used in a variety of senses in the history of psychology. Carl Wernicke’s (1874) model of language functioning, which emphasized connections between different areas of the brain, was dubbed connectionist, as was Donald Hebb’s (1949) account of memory as the strengthening of neural connections. The modern use of the term is related to Hebb’s, and applies to computational models of human performance that are built from neuron-like units. These models can be traced back to the work of McCulloch and Pitts (1943) on the simulation of neural functioning. Their ideas were simplified and made computationally viable by Rosenblatt (1962), in his work on perceptions in the 1950s and 1960s. However, interest in perceptrons waned when Minsky and Papert (1969) showed that the best understood perceptrons had severe limitations. In particular they could not solve so-called exclusive-OR problems, in which a stimulus was to be classified in a particular way if it had one property or another property, but not both. More complex perceptrons were difficult to analyse mathematically, but interest in them revived in the late 1970s, with the availability of cheap computer power. Such computer power enabled a practical, rather than a mathematical, demonstration of what problems could be solved by complex neural networks.
Since the mid-1980s neural network computing has seen a wide variety of applications, most of which lie outside of psychology. The term connectionism, though not strictly defined, is usually applied to the use of neural networks in psychological models of a kind made popular by Rumelhart, McClelland, Hinton and others (McClelland et al. 1986; Rumelhart et al. 1986). A connectionist model contains three groups of neuron-like units called input units, hidden units and output units. The basic property of a unit is to have a level of activation, corresponding roughly to a rate of neural firing. Activation is passed along the connections that link the units into a network, so that the activation of a particular unit is affected (either positively or negatively) by the activation of the units it is linked to. Passing of activation takes place in a series of cycles, one at each tick of a (computational) clock.
The stimulus that a connectionist net is currently responding to is encoded as a pattern of activation in the input units. Cycles of activation passing lead to a pattern of activation in the output units, which encodes the net’s response. The hidden units come between the input units and the output units, and allow considerable complexity both in the responses that the net can make and in the relations between the ways it responds to different stimuli.
It is possible to specify the function of units in a connectionist network and the strengths of the connections between them. For example, a unit in a word identification system might correspond to a letter A at the beginning of the word, and this unit will be highly activated by a word that actually begins with A. In addition, this unit will have strong positive connections to units that represent words beginning with A. However, one of the most important properties of connectionist networks is that they can learn to perform tasks. But, if they are to learn, all that can be specified is the methods of encoding used by the input and the output units. The interpretation of activation of the hidden units, and of the strengths of the connections between the units cannot be predetermined. In fact, learning in such systems is defined as a change in strength of the connections.
The best known method of learning in connectionist nets is back propagation. Initially the strengths of the connections are set at random, and an input is presented to the system. It produces a random output, but is told by its teacher (usually another part of the program) what the correct output should be. The difference between the actual output and the correct output is then used to adjust the strengths of the connections, working back from the output units to the input units (hence back propagation). This procedure is repeated many times for different inputs, selected from a so-called training set. The changes to the strengths of the connections are always (very) small, because the net must not get its response to the last input right by messing up its responses to other inputs. Eventually, a set of connection strengths should emerge that allows accurate response to all the stimuli in the training set and to other stimuli from the same class, on which it might be tested. Unsupervised methods of learning are also available for connectionist nets. Competitive learning is the best known in psychology, but a technique called adaptive resonance may be more useful.
Connectionist networks have found a variety of applications, both in psychological modelling and in the real world. In psychological modelling, the area that has seen the most successful application of pure connectionist techniques has been the identification of spoken and written words. In other domains, particularly those thought of as ‘higher level’, hybrid models are more popular. These models combine connectionist techniques with more traditional symbolic ones. Interestingly, in commercial applications (see, e.g. Lisboa 1992), such as industrial process control (including robotics and machine vision) and pattern classification, hybrid system are also often used.
Although strong claims have been made for them, connectionist nets suffer certain limitations. First, despite their success in models of, in particular, spoken word identification, connectionist nets in their basic form are not able to represent sequential information in a natural way. However, the recurrent (or sequential) nets developed by Jordan (1986) do have this capability, since they feed information from the output units back to the input units (with a time delay), so that temporal sequences may be learned. A more important limitation is that current connectionist networks appear to be incapable of explaining some of the inevitable relations between bits of information stored in the human mind. As Fodor and Pylyshyn (1988:48) point out
You don’t…get minds that are prepared to infer John went to the store from John and Mary and Susan and Sally went to the store and from John and Mary went to the store but not from John and Mary and Susan went to the store.
Unfortunately it is all too easy for connectionist nets to have this property.
Alan Garnham
University of Sussex
References
Fodor, J.A. and Pylyshyn, Z.W. (1988) ‘Connectionism and cognitive architecture: a critical analysis’, Cognition 28.
Hebb, D.O. (1949) The Organization of Behavior: A Neuropsychological Theory, New York.
Jordan, M.I. (1986) ‘Attractor dynamics and parallelism in a connectionist sequential machine’, in Proceedings of the Eighth Annual Conference of the Cognitive Science Society, Hillsdale, NJ.
Lisboa, P.J.G. (ed.) (1992) Neural Networks: Current Applications, London.
McClelland, J.L., Rumelhart, D.E. and the PDP Research Group (1986) Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 2, Psychological and Biological Models, Cambridge, MA.
McCulloch, W.S. and Pitts, W.H. (1943) ‘A logical calculus of ideas immanent in nervous activity’, Bulletin of Mathematical Biophysics 5.
Minsky, M. and Papert, S. (1969) Perceptrons, Cambridge, MA.
Rosenblatt, F. (1962) Principles of Neurodynamics, New York.
Rumelhart, D.E., McClelland, J.L. and the PDP Research Group (1986) Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 1, Foundations, Cambridge, MA.
Wernicke, C. (1874) Der aphasische Symptomenkomplex, Breslau.
Further reading
Levine, D.S. (1991) Introduction to Neural and Cognitive Modeling, Hillsdale, NJ.
