MODELING MEMORY: ATTRACTOR NEURAL NETWORKS

MODELING MEMORY:
WHAT DO WE LEARN FROM ATTRACTOR NEURAL NETWORKS

Symposium "Memory: from neuron to cognition"
Académie des Sciences, Paris, April 17-18, 1997

Nicolas BRUNEL and Jean-Pierre NADAL
Laboratoire de Physique Statistique de l'Ecole Normale Supérieure
Laboratoire associé au C.N.R.S. (URA1306), à l'ENS, et aux universités Paris VI et Paris VII
Ecole Normale Supérieure
24, rue Lhomond - 75231 Paris Cedex 05

Introduction

The variety of theoretical approaches to the modeling of memory is as wide as the variety of experimental approaches. Modelers explore many different levels, from molecular and single cell models to very high level models. In this short contribution we will comment on one intermediate family of models, namely single networks with working memory properties. These networks can be built with more or less complex `formal neurons', from the binary neuron of the Hopfield model (Hopfield 1982) to spiking neurons like the integrate-and-fire neuron (as in the models by Amit et al, to be presented below). These networks can in turn be used as building blocks for models with more complex functions, as the models presented in this symposium by S. Dehaene.

Attractor neural networks

The common property of these models is the formation during learning of `memory states'. A stimulus, when shown to the neural network (assembly), elicits a configuration of activity specific to that stimulus. This configuration of activity is then learned via Hebbian synaptic modifications. These synaptic modifications in turn enable the neural assembly to sustain an active representation of the stimulus (i.e. the ensemble of neural activities specific to that stimulus), in the absence of it: a `memory state'. It was Hopfield who introduced the general concept of attractor neural network, in which this behaviour is generically observed. More specifically, in his paper of 1982 he defines an associative memory model based on formal neurons which represents the first full mathematical formalisation of Hebb ideas and proposals on the neural assembly, the learning rule, the role of the connectivity in the assembly and the neural dynamics.

Synaptic plasticity: Hebbian rules

Most associative memory models are based on Hebbian learning rules. In fact, after the basic proposals of Stent and Hebb, remarkable progresses have been made in both the experimental studies of synaptic plasticity (see talks given at this symposium) and the theoretical analysis of Hebbian type learning rules. In many cases the so called covariance rule (Sejnowski 1977) and other closely related rules (such as the BCM rule) are compatible with experimental data, and on the theoretical side these rules can be shown to be efficient in models of formal neurons (in particular the Hopfield model uses a covariance rule).

Optimal and generic properties

In the late 80's, a new theoretical approach (Gardner 1988) has led to the study of the optimal and typical (generic) properties of formal neural networks. A practical consequence was that it was possible to compare the performances of a given model with the best possible ones. An unexpected and interesting result was that, in the limit of sparse coding (that is in the limit when a given memory state activates a very low fraction of neurons), a simple Hebbian learning rule reaches the theoretical maximal storage capacity (see Meunier and Nadal 1995 and ref. therein). In addition, near to optimality performance is reached with the Hebbian rule proposed in the sixties by Willshaw et al, where synaptic efficacies can take only two values. This is an encouraging result since neurophysiological experiments indicate that the fraction of neurons participating to a given `memory state' in the observed area is indeed very low (of order 0.01, see e.g. Miyashita).

Models and psychology

Short term memory models (Nadal et al 1986), obtained as simple variants of the original Hopfield model, allow for a comparison with psychological data. These models reproduce basic properties of human working memory as studied by psychologists - forgetting of old memories which are erased by new ones, but also more elaborate phenomena, such as primacy effects and proactive interference (Nadal et al 1988). It is not clear, however, whether such simple models can account for more complex phenomena as described by A. Baddeley.
Another succesful domain is the modeling of the effect of lesions in the cortex. In particular, a phenomenon similar to prosopagnosia is generically observed in neural networks (Virasoro 1988): information relative to individual memories is lost before the information which characterizes the class.

Models and neurophysiology

From the very definition of the first Hopfield type models, it was obvious that no direct comparison with data at the neurophysiological level could be possible. Quite recently, it has been understood how to build a new generation of models (Amit et al 1994), as a compromise between the need for preserving simple systems, exhibiting the same collective properties as in the Hopfield model, and the need for incorporating more realistic details which would allow for a direct comparison with the phenomenology of recordings during memory tasks. Very encouraging results have been obtained: the self-sustaining selective neural activity exhibited in these models is in nice correspondence with the phenomenology of single unit recordings in monkeys during delayed response tasks, for example in inferior temporal (IT) cortex (Miyashita 1988) or in prefrontal (PF) cortex (Fuster 1995).
In addition to the ability to form memory states, new properties appear that cannot be present in the simpler models. In particular, due to a strong recurrent inhibition, in each memory state only a small subset of neurons fires at more elevated frequencies, and, in the absence of external stimulation, the network stabilizes in a state of low spontaneous activity (Amit and Brunel 1997).

Towards a test of the attractor neural network paradigm

An interesting development of these models concerns learning of temporal context: the hypothesis is that when two stimuli are often shown one after the other, synaptic modifications will occur in such a way as, when one of the stimuli is shown, neurons selective to the other also tend to be activated. Thus memory states corresponding to two stimuli which often appear one after the other become correlated. Models implementing such type of learning (Amit et al 1994) have been shown to reproduce quantitatively the results of the experiment of Miyashita, in which precisely these correlations were measured in IT cortex. The availability of a detailed learning dynamics (Brunel 1996) enables to predict these correlations as a function of the temporal correlations existing in the sequence of stimuli. The ability of these models to analyse experimental data and to make new predictions has made possible a collaboration between physicists (namely the group of D.J. Amit) and neurobiologists (the group of S. Hochstein and V. Yakovlev in Jerusalem), in order to set experiments which will test these predictions. To our knowledge this is the first serious attempt to test the hypothesis that a collective phenomenon is at the origin of a memory function.