CONTRIBUTIONS TO THE MECHANISMS OF HUMAN MEMORY
JACQUES NINIO
( included in the web site http://www.lps.ens.fr/~ninio )
TOPICS DISCUSSED HERE:
Introduction :
- memory, a central issue
- a wealth of formal or informal observations
- the « learning and memory » confusion
- my itinerary in the field
Architectures for a replicative memory.
Experiments on short term visual memory :
- the 12 bits chunk boundary
- extension to colours and letters
- detection of symmetry violations
Experiments on working memory :
- the time course of information acquisition
- how the retrievable information varies with the number of items
Visual memory experiments with random testing orders :
- error levels as a function of testing rank
- reaction times as a function of testing rank
- patterns on consecutive tests
- towards a localized, architectural model
Reaction time distributions.
INTRODUCTION
Memory, a central issue.
I often begin my seminars on memory by drawing a parallel with the situation in molecular biology. After the initial observations of Mendel on the laws of transmission of inheritable traits, and after decades of studies on formal genetics, there came in 1953 the structure of DNA which provided a simple intuitive framework for organizing all the genetic knowledge. Now, one can initiate a teenager to molecular biology by presenting the structure of DNA and explaining reproduction and heredity from there. We cannot do the same for memory. We cannot draw a simple model, saying here is how memory works, and here is the origin of this property and this one.
Contemporary psychology has a very curious attitude towards the problem of the mechanism of memory. Memory plays a crucial role in most mental processes. If we want to understand how the mind works, understanding how memory really works should be a task with the highest priority. Yet, it seems that most people are not even conscious of the fact that we do not know how memory works, although we can make a long list of its well-established properties.
A wealth of formal or informal properties
We know a lot of things about memory. Pertinent observations have been made informally and reported in the past by a large number of people from all horizons, and it is possible to list many properties upon which most of us will quickly agree.
For instance, we know that our memory allows us to capture almost instantaneously a large number of items, and this does not require repeated presentations. More precisely, it captures a small amount of information about the encountered items. For instance, I run through the pages of an illustrated book which I have in my hands for the first time. One month later, if I am questioned about the book’s illustrations, I may provide only vague and poor information. However, if I turn again the pages of the book, I will recognize without difficulty the illustrations I had seen before. The memory expert will recognize here the theme of the almost infinite size of the memory stores (see, e.g. [1-4]) and the theme of the limited amount of information that is memorized during a rapid presentation (e.g., [5, 6]). An important related theme is that of the distinction between the efficiency of recognition memory (you recognize an illustration when you see it again) versus the inefficiency of explicit recall memory (being able to mention what was exactly in the illustration).
In all likelihood, human memory is content addressable. Here is a metaphor which I often give in seminars to explain the notion of addressability by content. Suppose you enter a room in which there are a large number of people, and you want to know whether or not Mr John Doe is among them. In a scanning process, you would interact with each person in the audience, one after the other, until you eventually locate John Doe. A more efficient way to get an answer would be to shout loudly, « is John Doe present ? ». Everyone in the audience receives the message, and everyone determines, at once, and independently of the others, whether or not he/she is concerned by the message. If John Doe is present, he will eventually reply positively to the inquiry. This is the principle of addressing by content. The input must make a kind of contact with almost everything in long term memory and whatever is related there to the input must signal that it is indeed related to it. Note also how easy it is to answer a question like “is tringob” an English word ?”. The diagnosis of absence of tringob from memory is immediate. Similarly, in forced-choice psychophysical recognition tests, recognition of a target is often indirect, by elimination of the distractor.
We have other formal properties such as the « forgetting curve » which indicates how fast the trace of a memorized item decays [7], or the law saying how the time to memorize N items increases with N (roughly, in fact, as N2 [7, 8]). There are also, in the front scene, elements of knowledge which I now consider to be misleading (see why further down), mainly in relation to the « magical number » and to the «linear memory scanning model » [9].
Face recognition. You have an appointment at a street corner with a person, and you recognize the person coming towards you from afar. Then, the person makes a few more steps in your direction, and now it is, unmistakably, another person, whose face is substantially different from the face you recognized a few seconds earlier. What is remarkable here, is that when the face is not seen with enough detail to be reliably identified, one yet sees it not as an indistinct face, but as the well defined face of the expected person. Thus, we were seeing the face with our memory template for the face of the expected person.
Drawing from memory. If someone with no skilled practice in drawing is asked to draw a bicycle, usually he/she will draw an absurd engine which cannot perform its function. The fact that the drawn bicycle is not correct is well perceived by the person who made the drawing, and this person will in general admit at once the superiority of a correct drawing of the bicycle. So we do not have a photographic picture of a bicycle in memory, which would be sufficiently explicit to guide us in drawing it ; but we have plenty of information about bicycles in memory, allowing us to discriminate between correctly and incorrectly drawn bicycles.
Fingerprints. Here is an observation made by Jean Itard at the beginning of the 19th century, which has profound implications concerning the mechanism of memory. Itard attempted to educate a child who had spent most of his early years in wilderness, the famous “Victor de l’Aveyron” [10]. He taught him to associate objects with their names, written on little sheets of paper. In one of the exercises Victor was shown a piece of paper on which an object’s name was written, then he was asked to fetch the object thus designated in a room upstairs. Once in a while, says Itard, Victor stopped in the middle of his climbing of the stairs, realizing he had forgotten which object he had to fetch. However, when he was down, it was enough for him to give a quick glance at the whole collection of names on the sheets to know what he had to fetch. So, it is much as though the representation of the object’s name in memory had been reduced to a kind of fingerprint. The fingerprint of a person allows you to identify the person when he/she is in front of you. It does not enable you to figure out how a person which is not in view does look. I have developed the argument in [11].
Double hits. A few years ago, I was seated at a theatre, and someone accompanied by a woman sat down in front of me. I was immediately struck by the intonations of his voice when he spoke to the lady, and I spent a good part of the play exploring my memory to identify this person among my old acquaintances. Then, the word “cinema” was uttered by one of the actors, and I immediately identified the man as someone I had met thirty years earlier during a trip in Cuba, and who had a passion for movies. So, it is as though a first search in memory had activated all acquaintances of the past, and independently of this search, the word “cinema” had activated in LTM everything that was there and related to movies, so that the memory trace of this man, having received two independent activations now could make its way down towards consciousness. Thereafter, I investigated on several occasions what made me think abruptly of something while apparently nothing in the circumstances could explain it. Usually, I find that the odd item could have emerged from the conjunction of a remote relationship with the present circumstances, and a subliminal activation during an independent mental process that had occurred previously. Note, in the same order of ideas that responding quickly to the question: “give the name of an animal which begins with ”g” is more difficult than answering the question “give the name of an animal with a long neck beginning with “g”. The Stroop interference effect [12] provides one of the best indications of this mode of functioning of memory, in which any incoming time activates nearly anything related to it in LTM. And “tip of the tongue” situations often reveal the effectiveness of multiple activations.
The “learning and memory” confusion
There are many charlatans, mostly among neurobiologists, who pretend that there is no mystery about memory, that it is merely a sophisticated version of pavlovian conditioning. According to their claims, learning and memory are essentially the same thing. This is a deep mistake. Learning is the acquisition of deterministic connections between an input and an output. In learning, if you receive a signal A, then you must automatically initiate a well determined action B, which has been associated with A through conditioning.
Memory works in a quite different way : it allows you to make links when you do not even suspect that these links might exist. For instance, you are involved in a discussion about a newly raised problem and all of a sudden an element comes to your mind which is highly relevant to the discussed problem, and for sure, this element is brought right when it was needed, and although there was no preexisting link with the problem. Here, memory works as a search mechanism which, given a situation A tries to find elements B which are likely to be pertinent to A.
The confusion between learning and memory is probably deliberate, and in any event, this confusion suits the interests of our rulers. Those who hold the power wish to deal with human beings who can be manipulated like pavlovian dogs. They are highly interested in mastering the recipes which can make the standard citizens work like slaves, buy uncritically the goods which are put on the market, follow blindly the investment advices given by the financial corporations which will rob them, vote for corrupt politicians who will promote legislations to ruin their lives, etc. So, research which equates the workings of the human mind with pavlovian conditioning (or LTP) in a small-brained animal (the leech, for instance), is receiving high priority.
On
the other hand, memory poses a threat to society, as it is organized
today. No politician would be reelected, if the citizens had enough
memory to remember his or her lies. People would not fall again and
again into the trap of compulsive buying for useless goods. They
would rebel against planified obsolescence, or against its more
modern substitute,”upgrading” (e.g., see [13]). Rational consumer
behaviour would generate an economic crash, under the logic of
contemporary economy.
So the thesis that learning and memory are
essentially the same thing is not simply a deep mistake. It is most
probabbly a deliberate confusion propagated to suit the interests of
politicians and other rulers. Cynicism is at its height when slavery
pills (based on LTP promoting agents) are advertised as
memory-enhancing drugs.
My itinerary in the field
I became involved in memory research through a curious itinerary. When I was working in molecular biology in the 1960’s and the 1970’s, I was favouring memory models involving an encoding of the memorized information into linear DNA or RNA sequences. I remember having read a theory proposing that some memory could be encoded in the form of a pattern of methylated nucleotides on DNA [14]. The author, who died very young, will be remembered for having proposed, as early as 1967, how the transmission of the scrapie disease could be mediated by a protein as the infectious agent [15]. Later, I was excited by the reports on transfer of memory (more precisely, aversive behaviour towards light, mediated by the peptide scotophobin) between planarian worms [16-18]. The work fell into oblivion after the death of one of the main investigators, Georges Ungar.
Then came the period of neural network models. I followed with amazement a series of talks delivered by Leon N. Cooper at Collège de France, in Paris, on associative memories, founded on matrix algebra analogies [I9-20]. A few years later, I read one of Kohonen’ books on content addressable memories [21]. I found these ideas sound and seducing, but did not see how I could trace my way in the field.
In the 1980’s I was very interested in communicating science to the public. In particular, I had regular contacts with Bernard Avron, a comedian and author of small theatre plays, who was srtiving to develop an activity on « theatre and science ». One day, possibly in 1986, Bernard Avron told me about his desire to do something on memory, so I stared to collect in a file all articles on memory I could find in « Scientific American », « American Scientist » and « Science », took the file with me on vacations, trying to extract the key ideas and distil them to him. I was impressed by a small set of review articles [22-24] or popular science articles [25-29] which thereafter formed the nucleus of my culture in the field. In December 1986, I started collecting material for a book intended to introduce cognitive sciences in France. The core of the book would be on visual perception, and from my understanding of this field, I wished to go further, and describe some of the mechanisms of the mind. For instance, I had read a number of articles on shape perception in animals, including ants, wasps and bees (e.g., [30-34]) and I was trying to understand how shapes could be encoded in the brain. It seemed natural for me that we recognize a familiar shape by matching the visual input with some construct formed from elements in memory. This is clear enough in the case of face recognition (see the paragraph on face recognition in the preceding section). I also favour the notion that much of abstract shape perception involves the generation, by internal processes, of shapes that match best the incoming shapes (see, e.g. [35, 36]). So there must be an encounter somewhere between memory-derived templates, and the input. (According to Miyashita [37], the inferior temporal complex is the brain’s locus where “perception meets memory”). Then came the question : How to characterize our ability to construct shapes from memory ? Here, I turned to our internal productions, as witnessed in dreams. I spent much time trying to recollect my dreams – Not as done by psychoanalysts, who emphasize the storyboard of the dreams, but in terms of the graphic quality of the images. What kind of details were accurately incorporated into dream images, and what details were not ? You will find in [11], Part III, chapter on dreams, a discussion of the graphic properties of dream images. I also conjectured later (see the 1996 edition of [11]) that while in learning one associates events happening in close spatial and temporal proximity, in dreams one brings together events separated in time or space. One function of dreams could be to explore ways of linking in memory previously unconnected items. This exploration would produce, in general, irrelevant connections, that will usually be discarded except in some primitive cultures; but from time to time, the nearly random association process might bring to the surface real causal connections. In this way, dreams might well be an instrument of rationality ([11],1996 edition). Similar ideas are now surfacing (e.g., [38]).
Soon, my ideas on memory crystallized. I conceived a model ([39], which made use of a neuronal replication mechanism. Its basic postulate was the existence of a neuronal architecture such that the information content of some given neuronal assembly could be copied into another neuronal assembly. An incoming percept would in a sense be replicated into a very large number of copies, which would be sent up along the branches of a ramified tree until they came in contact with the items in the long-term memory store. All items having some relationship with the incoming stimulus would be activated. But only the few best matching items would be selected, and would make their way down towards consciousness (for more explanations and details on the model, see the section on the model below). This contribution was completely ignored by the specialists in the field (and ignored by non-specialists as well). Unfortunately, I did not see clearly at that time how to test the theory.
After moving to the Statistical Physics Laboratory at Ecole Normale Supérieure, within a team of physicists interested in neural network theory and cognitive sciences, I started to initiate an experimental practice on visual illusions (see the website sections on illusions) using interactive computer graphics. I also gained good competence in computer graphics in my work on designing stereograms and autostereograms (see the web site section on stereoscopic vision). My colleagues showed a platonical interest in this work, and never reached the point of becoming partners in a psychophysical work. At last, I decided to launch an experimental work on memory. The first goal was to collect high quality data on memory, which could help to discriminate between distributed memory models and localized models.
A young member of the neural network team, Nicolas Brunel, with whom I was sharing an office, was open to the idea of joining the experimental work. Our experimental results on short-term memory (based upon tests in which images were compared side by side) allowed us to propose a characteristic value of 12 bits for the elementary capture of visual information.
Then we moved to working memory experiments. N images were memorized in succession, then the subject went through a series of N tests. Neural network models generally predicted that error-rates should vary in a sigmoid fashion as a function of the number of images. Below a certain number of images, all images would be equally well recognized. Above a certain number, the network would collapse. In the intermediate range, there would be a sigmoid rise in error-levels, the shape of the sigmoid being specific to the particular network model that was considered. The results turned out to be more complex than predicted by neural network theories (unless one said that there are several neural networks working in parallel, giving rise to “blended curves”). Massive data were required to obtain finer, more conclusive results. Nicolas Brunel abandoned this line of studies, and moved to more secure and institutionally rewarding domains, in which he published brilliant articles (e.g., [40]). I pursued the work alone. Paradoxically, while I had a rather high position within the CNRS, and was about 60 years old, I worked now as I did when I was a doctoral student, so to speak with my bare hands.
At last came the reward. With massive data in hands mostly contributed by myself as a subject, I could see structures which no one (including myself) had anticipated. The structures in the data were strongly in favour of a localized model of memory, and the fine details in the data allowed me to propose a tentative model of how the memory slots were interconnected, and how the memory traces moved from slot to slot.
ARCHITECTURES FOR A REPLICATIVE MEMORY
In my initial architectural model for memory [39], there is a « smart device » sandwiched between sensory memory and long-term memory. It is definitely not a distributed neural network. It is a device with slots. Information about a particular item may be temporarily stored in a slot A, then transfered to another slot B. In part, the model is about an intelligent wiring diagram for this module, allowing the incoming items to be efficiently encoded as a result of a dialogue with long term memory (LTM), and allowing items in LTM which are pertinent to the present situation to make their way to consciousness. The model is also about wiring diagrams at the dendritic level between neurones, in order to achieve, through a kind of neuronal copying mechanism, the transfer of information from one neuronal assembly to another.
If one is prepared to accept the idea of temporary localized storage of an item in a slot, and since the new incoming items must replace the old ones, we must also accept the idea that information about an item may move from one place to another. To be more precise, what moves is not a kind of copy of the sensory item, but some compact encoded description of the item. Now saying that information may be tranferred from one place A to another B, amounts to say that a neuronal assembly located in B can reproduce the information that is stored in A. In current descriptions of brain processes, one would rather consider that a back and forth copying mechanisms cannot exist, and that there are exclusively bottom up or top down processings, during which the nature of the encoding would change at each level. Contrary to this view, I considered that at the core of memory (as distinguished from learning) there is a mechanism for just copying information, making back and forth travels possible, and mental simulation a reality.
When a stimulus, beit visual or auditory is received by the senses, it is transduced and maintained for a short while in a « sensory store ». The brain carries out an interpretative work on the sensory stimulus (for instance, extracting letters of the alphabet from the visual signal, as in Sperling’s experiments [5], or abstract shapes as in Phillips’ experiments [41]). The interpretation of the signal in terms of letters, or of familiar faces, or of known objects obviously involves a dialogue with long term memory. « Seeing » an abstract shape also involves, in my view, a dialogue with areas of LTM. The stimulus is thus identified, or it is assigned a description in terms of LTM categories. Let us say that it is encoded. In visual perception as soon as you do not have a stimulus in front of your eyes (for instance because your eyes are making a saccade), the sensory representation of the stimulus is lost, and you are left with the encoded representation. This feature has been convincingly demonstrated in a large number of experiments. For instance, when you read text, you remember the meaning better than the exact wording [42], and you remember the words rather than the font used to print them [43-45].
Familiar items are very quickly recognized. I view the « calling » stage as follows : Imagine the input (a percept) being propagated along the branches of a highly ramified tree (for instance, some kind of binary tree studied in computer science algorithms). It has a root, then a nod from which two branches stem, then at the end of each branch there are nods from which more branches are issued, etc.. Let us say that the ends of the terminal branches are in contact, in a one to one correspondence, with the neurons or neuronal assemblies encoding the various items stored in long term memory. Wherever there is partial match, the LTM slot is activated and sends back a signal. Most often, a rather large number of items in long term memory will be activated, and there must be a selection mechanism to maintain active only the items matching best the calling item. This is how, in my model, the recall process works, requiring (i) an exponential copying of the input to form a contact with large areas of LTM (ii) a comparison between stored items, and the copies of the input, leading to an activation of the best matching items (iii) a further selection of the matching items (or rather, their copies) in a descending path towards consciousness. As a refinement to the model, I proposed that an incoming item, having activated a number of memory items might then be encoded in terms of what it did activate. The code of a new item could be mainly about the addresses of items in memory to which it relates.
The process I am discussing is that of “complete” recognition of an item, it may be somewhat different from that of fast classification of an object as belonging to a broad class of objects ([46-48]).
Now, how can a neuron (or an assembly of neurons) copy the information encoded by another neuron, or assembly of neurons ? In my first proposal [39] I suggested a neuronal architecture in which dendro-dendritic interactions played a crucial role. In a second publication, I discussed other wiring diagrams, perhaps closer to neurological orthodoxy [49]. Note that in any event, the « shifter circuit » architecture, proposed by highly respected neurobiologists [50] is designed to move, physically, neuronal information from one place to another. So, shifter circuits, if they exist in the brain, could perform neuronal copying.
By the end of 1987, my ideas about memory being subtended by a neuronal copying mechanism were sufficiently mature. I wrote a short note which I thought of submitting to the proceedings of the French Academy of Sciences. The note needed to be evaluated and channelled by a member of the Academy, and I asked the French neurobiologist Jean-Pierre Changeux to handle the note. He first asked me to make room, in my note, to an article he had recently written with Stanislas Dehaene and Jean-Pierre Nadal on a memory module with three cells [51]. Having obtained that his article be quoted, Changeux then found a reason not to handle the manuscript. I then turned to François Jacob, the most highly respected French biologist, who shared the 1965 Nobel prize for medicine and physiology, together with Jacques Monod and André Lwoff. Jacob was very receptive to my theory. He sent my note for review. It came back with a report recommending rejection, in all likelihood written by Elie Bienenstock, a theoretician of learning mechanisms by neural networks. François Jacob received me a second time in his office, and decided to give my theory a chance, so it was published in the Comptes-Rendus de l’Académie des Sciences. The title of the note spoke of a « localized, replicable, associative iconic » model of memory, which was adequate, but the qualifier « iconic » before the word memory was unfortunate. By iconic, I had meant visual and capable of producing vivid images. (I was, at that time, much impressed by work on visual imagery, e.g. [26, 52-54]). However, it must have been taken by colleagues in a hurry as an article dealing with sensory memory.
A few years later, I had an occasion to refine the model, and remove the ambiguity due to the title. A theoretical neurobiology conference was organized by Vincent Torre in Erice (Sicilia) in 1995. My friend Christian Wehrhahn, from the Max-Planck-Institut für Biologische Kybernetik in Tübingen knew the organizer, and obtained an invitation for me. I was invited, I expounded my theory at the conference, and had the occasion to learn a lot and meet many interesting colleagues. The refined theory was subsequently published as a chapter in the book issued after the congress [49].
Nearly twenty years have elapsed since my initial publication, and there has not been as yet a single echo to my work. When I discuss the model with colleagues, they invariably object, as did the reviewer for [39] that it does not agree with neurobiological orthodoxy. My feeling is that this may indeed be the case, but that tomorrow’s orthodoxy may substantially depart from today’s orthodoxy. Furthermore, one should first determine whether or not my theory captures a number of essential properties of memory. The question of the most realistic neuronal implementation can be answered at a later stage.
At a deeper level, I consider that my colleagues are victims of the prejudice that memory is merely a form of learning, and since we have reasonably good models of learning (Hebbian, etc.) there is no need to make room for specific memory mechanisms. One aspect of the prejudice is that people do not pay sufficient attention to the formal properties of memory. It is as though, in molecular biology, we satisfied ourselves with a vague concept of inheritance, and did not care to take the full measure of the precise laws of genetic transmission.
EXPERIMENTS ON SHORT TERM VISUAL MEMORY
The 12 bits chunk boundary
When I initiated, with Nicolas Brunel, the experimental program on human memory, I had already done psychophysical work on stereo vision (e.g., [55] – see the web chapter en contributions to stereoscopic vision) and on geometrical visual illusions ([56]). From these two domains, I had acquired the belief that what was most lacking, in cognitive sciences, was high quality data. So I was determined not to put the cart before the horses. I wished to lay the foundations of a rigorous, precise experimental work, which would lead to solid conclusions.
The first work was designed to measure the capacity limit of what William Phillips christened « short-term visual memory » or STVM (e.g., [57, 58]. Drawing on my past experience with stereoscopic stimuli, I tested visual memory on abstract patterns. Imagine a pair of images, in which the left image is a square lattice filled at random with black or white squares and the right image differs mainly from the right one by a white/black inversion in one of the squares. (See Fig. 1, below). The task of the subject was to compare the two images, until he/she spotted the difference between the two images. This protocol was inspired from a well-known visual game in which two drawings of a complex scene differ at seven positions, and the task is to locate the differences, which can take a surprisingly long time. In searching for the differences the eyes try to capture part of one image, then move to the corresponding part of the other image, and although the two images may differ in this region, the person may not detect the difference. The explanation is that when the person looks at the first image, he/she extracts partial information, which is held in STVM while he/she performs a visual saccade to the other image. Upon visual landing on the appropriate portion of the second image, the person compares the available, detailed visual input with the representation of the first image in STVM. Failure to detect the difference is an indication that the STVM representation was not detailed enough to include pertinent information about the locus of the difference between the two images. From this type of consideration, and from the abundant data we gathered on the time to detect the difference as a function of image complexity, and the error-rates for different classes of images, we derived a straightforward estimate of about 12 bits for the visual information that could be captured and maintained in memory during a saccade. This estimate, we showed, was consistent with various earlier results. However, while earlier results (e.g., [5, 6]) had usually been interpreted in terms of a limit to the number of memorized items, here our results were expressed in terms of bits of information. We also found that making our images more discrete (by introducing separations between the elementary squares) did not help, and that distorting the squares did not impair performance. So, what is stored in memory is not a kind of pixel by pixel representation, but a rough description of the shapes in the images (see Fig. 1)
-------------
Figure 1. Comparing images side by side, differing at a single position.
Click here to
load the jpg file
A pair of images being presented on
the computer monitor, the task of the subject is to locate the
difference between the two images. He moves the mouse until a cursor
on the screen coincides with a position at which the two images
differ, then makes a press on one of the mouse’s buttons. This set
of examples illustrates various questions which were addressed in the
study: complexity, in terms of number of black or white elementary
quadrangles (compare a with c), discreteness (compare a, b, c in
which the shapes are continuous, with d, e, f in which the presence
of a grid discretizes the images, resistance to distortions (compare
a with e) and the effect of correlated versus uncorrelated
distortions (compare e with f).
-----------
An article describing our results was submitted to « Cognitive Brain Research ». Both reviews were quite pertinent and sympathetic to the work. The Editor in chief was Dominick Purpura, and I do not know whether he or someone else did the effective editing work. In any event, this was a case in which the evaluation process was performed honestly and efficiently. We were not penalized for being outsiders, and the article was judged on its scientific merit, although it was written as a « hard science » article, rather than as an article in experimental psychology.
Here is how our results were summarized in the Cognitive Brain Research 1997 article [59] :
-------------------------
The time to locate a difference between two artificial images presented side by side on a CRT screen was studied as a function of their complexity. The images were square lattices of black or white squares or quadrangles, in some cases delineated by a blue grid. Each pair differed at a single position, chosen at random.
For images of size NxN, the median reaction time varied as cN2, from N= 3 to N = 15, with c being around 50 ms in the absence of grid. (i.e., when the quadrangles were associated into continuous shapes). For N <= 9, when the lattice was made irregular, performance did not deteriorate, up to a high level of irregularity. Furthermore the presence of uncorrelated distortions in the left and right images did not affect performance for N <= 6. In the presence of a grid, the reaction time were on average higher by 20%. The results taken together indicate that the detection of differences does not proceed on a point by point basis, and must be mediated by an abstract shape analysis, in agreement with current views on short-term visual memory (e.g., Phillips, W.A. [41]).
In complementary experiments, the subjects had to judge whether two images presented side by side were the same or different, with N varying from 1 to 5. For N <= 3, the same and the different responses were similar in all their statistical aspects. For N >= 4, the « same » responses took a significantly larger time than the « different » responses and were accompanied by a significant increase in errors. The qualitative change from N = 3 to N = 4 is interpreted as a shift from a « single inspection » analysis to an obligatory scanning procedure.
On the whole, we suggest that visual information in our simultaneous comparison task is extracted by chunks of about 12 + or -3 bits, and that the visual processing and matching tasks take about 50 ms per couple of quadrangles. In Section 4, we compare these values to the values obtained through other experimental paradigms.
--------
In 2004, I was at the ECVP congress in Budapest and was presenting other results, pertaining to working memory. Yuri Shelepin, then head of the Pavlov Institute of Physiology in Saint Petersburg was one of the rare persons to find an interest in my poster. He mentioned that he had short-term memory results on the elementary capture of visual information [60], and he asked me if I could guess how many elementary black or white squares this amounted to. I knew of course the result (12 bits), and we have remained into kind contact since that time.
-----------------
Extension to colours and letters.
Several subjects felt that the task would be easier using pairs of coloured, instead of black and white elementary squares. So, I performed similar experiments, with images containing elementary squares having one colour among a choice of four (red, blue, green, yellow) drawn at random. The result was consistent with the 12 bits capacity (a figure with 8 elementary coloured squares is equivalent to a figure with 16 elementary black or white squares). Similarly, when comparing boxes containing letters taken at random from an alphabet of 16 letters, there is a slightly larger capacity limit, around 16-20 bits (4-5 letters). The comparison of the results obtained with black and white shapes, colour patterns, and letter strings confirms the validity of the proposal of a capacity limit expressed in bits, rather than in terms of number of items. The results were presented at the 1999 European Congress on Visual Perception, held in Trieste [61].
Here is the ECVP 1999 abstract (co-authored with Eduardo Mizraji) :
-------------------------
Processing of shapes, colours, and letters in short-term visual memory.
The time to determine whether two images presented side by side are identical or not provides clues on short-term visual memory (STVM). For arrays of random black or white elements, about 12 elements are held in memory during eye movement from one image to the other, and the comparison time is about 60 ms/element (Brunel and Ninio, 1997, Cognitive Brain Research 5 273-282). Here 18 subjects compared arrays of black or white quadrangles (1 bit/element) ; arrays of blue, green, yellow or red quadrangles (2 bits/element) ; or arrays of letters belonging to a 16-letters alphabet (4 bits/letter). The subjects differed in their relative performances on letters versus colours or colours versus shapes. However, the subjects-averaged times to decide that two images are different were remarkably consistent, once they were expressed per bit, and not per element (60, 65, and 55 ms/bit for shapes, colours, and letters, respectively). STVM capacity was deduced from the point at which « same » responses became substantially slower than « different » responses : around 12 bits for shapes or colours, but 20 bits for letters. Thus letter processing is not exceedingly rapid, but once encoded, letters are rather well retained.
-------------------------
Detection of symmetry violation.
There is an abundant literature on symmetry detection – mainly vertical symmetry detection (e.g., [62-66]). When two identical images are presented side by side, it may take you some time to realize that they are identical. However, if one of the images is juxtaposed, along one of its vertical sides, to its symmetrical image, the symmetry jumps to the eyes. The fact that vertical symmetry is far more salient than identity is rather well established.
However, most experiments upon which this conclusion is derived are experiments in which one compares perfectly symmetric shapes versus randomly related shapes, or perfectly identical shapes, versus randomly related images. These experiments do not incorporate the conceptual progress made in my previous work, according to which images are stored as roughly 12 bits of information descriptions. So two nearly symmetric images may appear symmetric when examined at a certain level of detail, and appear dissymmetric when examined at a greater level of detail. I thus extended my previous work to pairs of images which were related by a vertical symmetry axis, and in which one of the elementary squares could be different. So, at a rough level of description, the pairs were always symmetric, and the question became how good we are at detecting a small dissymmetry between the patterns. My provisional conclusions were presented as follows at ECVP 2006 in Saint-Petersburg [67] :
-------------------------
Processing of symmetrical shapes in short-term visual memory.
The time to determine whether two images are identical or not increases with image complexity, by about 60 ms per bit. Above 12 bits, the « same » reaction times (RTs) become significantly larger (Brunel and Ninio, 1997, Cognitive Brain Research 5, 273-282). Here, twenty-five subjects compared 3x3, 4x4, or 5x5 random arrays of black or white quadrangles either identical, or differing by a single element, and related either by a translation or by a symmetry around a vertical axis. The same 12 bits threshold was found for the two types of tests. The RT variations were strikingly parallel, but downshifted in the symmetric case by about 730 ms for the ‘same’ condition, and 650 ms for the ‘different’ condition. The detection of symmetry violation was the fastest and the least error-prone on the middle column in the case of 3x3 images, and on the second and third columns in the case of 5x5 images. These results suggest that symmetry as well as repetition comparisons are initiated with the construction of a 9-12 bits representation of one of the two images. This construction, rather than the comparison itself, would be faster in the case of symmetric images.
-------------------------
I am still progressing in the accumulation of data (to reach about 100,000 reaction time measurements), and I may revise or refine my conclusions in the near future. My experiments confirm the fact that symmetry detection goes faster than repetition detection. However, there are a number of factors which may give an advantage to symmetry detection, even in the absence of a hard-wired symmetry detection mechanism : (i) repetition is in part difficult to detect because the shapes in the two images are seen in different contexts. The left side of the right image is, so to speak, contaminated with the right side of the left image. And the right side of the left image is contaminated with the left side of the right image. This turns out to be perturbing, and it would perhaps be useful to perform experiments dealing specifically with this issue (for instance, using periodic patterns) (ii) An image may be interpreted as a white shape over a black background, or as a black shape over a white background. In order to appreciate repetition, the same choice must be made for the two images. However, in the case of symmetry, if the interpretation builds up from the centre, the same choice is made automatically for the two halves.
My results, in which I studied error-levels and reaction times for comparing dissimilar or dissymmetric images suggest that in both cases the brain works on chunks of about 12 bits. It is indeed remarkable that symmetry violations, in the case of 4x4 or 5x5 images are better detected at the second than at the first column from the symmetry axis.
----------
Figure 2. Reaction times for detecting a violation of symmetry.
Click here to
load the jpg file
The stimuli were pairs of images composed of
3x3, 4x4 or 5x5 random arrays of black or white quadrangles, as in
Figure 1. The quadrangles were slightly distorted, as in Fig. 1b. In
the control experiment, pairs of images either identical, or
differing at a single position, were presented side by side, and the
subject had to make a same/different judgement (top left panel). The
RTs for the “different “ responses are plotted as a function of
the column in which the difference was. There is a clear effect with
5x5 images. Differences are most salient in the centre. There is no
effect of column rank with 3x3 images, confirming that these images
are entirely captured in STVM in a single visual seizure. In the
experiment shown at the bottom similar pairs of 3x3, 4x4 and 5x5
images were generated, and one image was presented juxtaposed along a
vertical side to a mirror image of the other. The subjects had to
determine whether or not the two components were perfectly symmetric.
The RTs recorded in the case of symmetry violations are shown in the
bottom panel. There is no column rank effect with 3x3 images,
suggesting again a single shot capture. In the case of 5x5 images,
differences in column 1 (the closest to the axis of symmetry) are not
detected more rapidly than differences in column 2. The top right
panel reports results for pairs of images that are symmetric, or
nearly symmetric about a vertical axis, but slightly separated, as
for the repeated images of the top left panel. The wide RT difference
between the results for symmetric images with or without separation
is mainly due to the presence of a few very rapid subjects in the
series on symmetry without separation.
--------
Since the data on symmetry and repetition are so much alike, except for the speed effect, I am tempted to make an iconoclastic proposal. Remember that a shape is hard to distinguish from its mirror-image. Caelli and Julesz had shown for instance that when an array of randomly oriented letters is embedded within an array of the mirror images of these letters, the two arrays are hard to segregate [68]. I will go one step further. Consider that there is a common representation for a shape and its mirror image in memory, and that the distinction between the shape and its mirror image is merely a single determinant appended to this representation (in the same way that you could represent a shape, and add a colour specification). I suggest that when one looks at a symmetrical image, the two halves activate the common representation, and they activate it more efficiently precisely because they act on both variants of the representation.
EXPERIMENTS ON WORKING MEMORY
The time course of information acquisition.
Having determined the amount of visual information that is extracted in a single shot and maintained in short term memory, I then explored the properties of visual information storage in a longer time range. I first studied visual memory of single images. An image similar in design to the images which were used in the previous section (slightly distorted square lattices of black or white quadrangles) was memorized for a certain amount of time, then it disappeared from the screen. Then a couple of images was presented side by side. One of the images was the memorized image, and the other one differed from this one at one or more positions, in which the quadrangles’ colours were changed from black to white, or vice-versa. The task was to determine which of the two images corresponded to the memorized image. The amount of memorized information was deduced from the error-rate (I am indebted to Nicolas Brunel for proposing the method and supplying the computer program [Note #1]).
I tested myself to determine how much I memorized of an image as a function of the presentation time, and the result was clearcut. The number n of memorized bits varied roughly as the square root of the presentation time. (More precisely, the exponent x of the power law n = tx could be 0.56 rather than 0.5). The power law applied from a few seconds presentation time to at least 100 seconds. To turn it differently, in order to double the number of memorized bits, I needed a presentation duration multiplied by four. Such a power law had in fact been described a century ago by Binet in his observations on mnemonists who memorized large lists of numbers [8]. I also determined the amount of memorized bits as a function of presentation time when 2, 3 or 4 images were memorized consecutively, the images being tested in their presentation order. The number of memorized bits per image was smaller than in the case of single images, but the power law applied, with the same exponent. So, I confirmed with myself (a non exceptional subject) and in the visual domain what had been observed with mnemonists in the 19th century. This power law should stand as one of the basic experimental laws of memory. Yet, it is largely ignored by memory specialists (but see [69]).
I then ran a series of experiments to determine what happened in a shorter time range, actually from 1 second to 8 seconds. This time, 30 naive subjects took part in the experiments. I determined the average number of bits memorized per image, for presentation times of 1, 2, 3, 5 and 8 seconds. I obtained a sigmoid dependency. At one second, about 12 bits were memorized - just the amount which had been found for short term visual memory and, presumably, acquisition times in the range of 300 mseconds. So, one could say that there is an initial capture of about 12 bits of information in about 300 msec, and no clear gain up to one second. Then, at 2 sec presentation time, there is a small gain. The number of memorized bits rises to about 15. I interpret this increment as follows: With a 2 sec. presentation time, the subject can make a rapid exploration of the image, and chooses a part which looks simple to memorize, and on which he may fix his attention. So the gain would have little to do with the workings of memory. From 2 to 6 seconds, the curve is nearly horizontal, there is very little gain, but after six seconds, the curve starts ascending clearly. The stability in performance of the naive subjects between 2 and 6 seconds is amazing. Furthermore, it contradicts the subjective feeling of acquiring information all along the presentation duration. In my opinion, what happens is that there is a first acquisition of visual information at the 12-15 bits level, up to two seconds presentation time. Beyond this first seizure of information, during which a few salient shapes within the image (for instance, a cross, a square, the letter T) were perhaps noticed one needs to establish a dialog with long term memory to be able to construct a more detailed representation of the image. This is where an experienced subject can do better, because he has a larger store of readily accessible patterns in memory, that can match the patterns in the image, and a large store of criteria (are there alignments ? is there symmetry ? do the elementary patterns touch a border of the image ? and so on). A similar time-course was observed in parallel experiments in which two images had to be memorized, instead of one.
In any event, the results are there (see Fig. 3) and, to my knowledge, they are absolutely original in the field of memory. Again, nobody seems to have paid attention to these findings.
--------
Figure 3. Memorizing one or two images as a function of viewing time.
Click here to
load the jpg file
Central panel: results of two independent
experiments. The first one involved 15 subjects, single images, and
viewing times from 1 to 5 seconds (unfilled circles). The second
experiment involved 14 subjects, and either single images (black
disks) or two consecutive images tested in the same order (squares).
Each result by each subject was converted in bits independently, then
the averages were taken over the subjects. Right panel: experiment
performed by the author as a subject under similar conditions. The
errors scored in the left panel correspond to the same experiments as
the central panel, represented with the same symbols.
---------
How the retrievable information varies with the number of items
Having cleared the ground, I then proceeded to the more ambitious task of determining how the amount of memorized information varied with the number of memorized images. Would we, at last, find some evidence in favour of the mythical « magical number » [70-73]? I performed comparative experiments with several subjects attending to 1, 2, 3, 6, 12 images, found the results disappointing, and decided to perform exhaustive, lengthy experiments on a few subjects. There were in fact three subjects in this new series, a young psychophysicist, at the phD level (CA), a psychology student with no laboratory experience (PP) and myself (JN). The curve describing the average number of bits memorized as a function of the number of images presented consecutively was determined from n = 1 to n = 40 images for the two youngsters, and from n = 1 to n = 100 images for myself. These curves are shown in Fig. 4 below.
Subject CA has, as expected, a drop of performance from 1 to 2 images. The remaining of the curve was unexpected. Performance remained stable from 2 to 10 or 12 images, and declined only afterwards.
Subject PP had also a drop of performance between 1 and 2 images (which is masked in the published curve, because PP was already at ceiling performance on 2 images under the conditions of the study). Then the performances were similar for 2 and 3 images, and dropped sharply from 3 to 4 images. The magnitude of the drop was surprising : It implied that the total amount of information memorized on 4 images was less than the total memorized on three images. So, something destructive happened, in the case of subject PP, when he attempted to memorize a fourth image. After the drop from 3 to 4 images, performance remained at a steady level from 4 to about 10 or 12 images. The existence of this second plateau, after the 2-3 images plateau, seems to me hard to reconcile with most current neural network models.
In my case I had the classical drop from 1 to 2 images, then a plateau on 2-3 images, and a second drop, but of much smaller amplitude than that observed with PP. There was then a second plateau from 4 to about 10-12 images, then another drop. In the 20-100 images range, performance declined moderately, and 7 bits on average remained memorized per image in runs of 100 consecutive images. I performed the same experiment one year later and had very similar results (see Fig. 4).
---------
Figure 4. Memorization as a function of the number of consecutive images.
Click here to
load the jpg file
The four panels show the results of
case studies with three subjects. Top left: early experiments by the
author, involving mostly 10x10 but also 8x8 images, viewed during 8
seconds. Testing was always in the presentation order. All the
results for a given number of consecutive images are regrouped as a
single point. Increasing m values were explored several times. 7000
images were viewed and 1279 errors were scored. Bottom left:
experiments by the author 11 months later, using 8x8 images viewed
during 6 seconds, with testing in the presentation order. Different
symbols represent different series of experiments. Within each
series, increasing and decreasing m values were explored an equal
number of times. 3866 images were viewed, and 887 errors scored. Top
right: experiment by subject CA, with 8x8 images, viewed during 6
seconds and tested in the same presentation order. In each series, 4
to 6 m values were explored in increasing order, and the successive
series formed overlapping sets gliding progressively towards the
higher m’s. 4048 images were viewed, and 894 errors were scored.
Bottom right: experiments by subject PP. The conditions were the same
as for the bottom left experiment, except that the images were
tested, in equal numbers at each m > 1 value, in the direct and
the reverse presentation orders. 4494 images were viewed, and 1178
errors were scored.
-------
Although the results from the 3 subjects differed substantially, I thought of a conceptual framework in which all three types of curves could be given meaning. Imagine an amphitheatre with several rows that accommodate an increasing number of seats, from the front to the bottom. Let us say that an incoming item in memory is like a person entering the amphitheatre, then taking a seat in one of the rows. Let us assume that the quality of the seats decrease from the front to the bottom. According to this metaphor, the quality of storage (as measured, for instance, by the number of retrievable bits of information) decreases from the front rows to the back rows. The differences between the 3 subjects can now be interpreted in terms of placing strategies. Subject PP would start by filling the first row until it is complete, which happens with the arrival of the third item to be memorized. Now, when the fourth image is memorized, it must be accommodated at the front, because the last memorized images goes to STVM. PP’s brain sends the three images which were in the front row, to some row behind. So, PP has excellent memory for three consecutive images (they are all seated at the front), and he goes through a memory catastrophe when memorizing a fourth image. Subject CA would place the incoming images randomly, within a certain number of close rows. Then the quality of memorization may remain constant from 2 to 12 images. In my case, if one observes carefully the 2 – 12 images plateau (Fig. 4, panels a and b) one notices that the slope may even be ascending. So,there is the possibility that 9 images are better memorized, on average than 4 images ! The effect may not be significant, or it may be due to some kind of experimental artefact. Still, this behaviour is conceivable, in the amphitheatre model. It would correspond to a strategy of filling the back rows first.
Whatever the criticisms which could apply to my experimental results, it could not be negated that, if they were correct, they were rather important. They were quite discriminative in terms of memory models. They could not be predicted by any of the current neural netwok models, and they seemed to lead to an architectural model of localized memory, with several rows containing slots with different qualities of storage.
A manuscript was written, and submitted again to Cognitive Brain Research [74]. Dominick Purpura acted again as Editor. Two reviews were communicated to the author. A very favourable one said:
“This paper reports a series of studies to characterize certain aspects of the operation of working memory. The experiments are carefully designed, and the results are generally clear-cut. The author has a broad view of the literature, and does a good job of placing his results into context”.
A second review was broadly favourable. But some speculations in the paper made the reviewer jump to the ceiling. In particular, I had written in the manuscript that I favoured the existence of an array with 9 slots, “because of the widespread occurrence of ninefold symmetry in cellular structures. This organizational theme is found in cilia and flagella, in kinetosomes, in the cillium-based sensory cells of the retinas, and also in chemoreceptors and mecano-receptors”. This sentence elicited the following comment:
“This is, in my view, complete and utter nonsense, and has no place in an supposedly serious scientific article aimed at a serious journal like Cognitive Brain Research. Who could seriously accept that the capacity of visual memory should be constrained by the same developmental rules that govern the formation of cilia and the like. The author will have to greatly improve the strength of his arguments before people will take him seriously.” I guess that the friends of the reviewer will identify him easily. (Christoph Koch, perhaps?).
Incidentally, I became aware of the biological importance of ninefold symmetries reading the book by Lynn Margulis “Symbiosis in cell evolution”. I thank my colleague Michel Bornens, a distinguished cell cytologist, for further insights on 9-fold symmetry in subcellular or cellular structures.
After the publication of this article, I tested two more subjects, under slightly different conditions (the presentation time was lowered to 4 seconds). The drop after 10-12 images was also observed with these subjects. However, there was no drop after 3 images, and instead there was one around 4 to 6 images. So, all subjects have a plateau starting at 2 images. The location of this plateau’s terminus is highly variable: 3 for PP and JN [74], 4-6 for EM and SK (unpublished results), and 12 for CA [74].
Following a suggestion by Nicolas Brunel I also carried out experiments in which memorization and testing were strictly alternating. After memorizing an image, there was a test related to an image that was memorized n steps before (an “n-back” paradigm, related to the paradigm used by Amit and co-workers in monkey’s working memory studies, e.g. [75, 76]). I tested many subjects, using three different ranges of the « exploration depth » n. There was a series with tests on images that had been memorized 2 to 5 steps back, another one with n ranging from 2 to 9, and a third one with n ranging from 2 to 13. More recently I also carried out experiments with more deterministic rules on the number of steps back. The results are curious, and I have not yet reached a satisfactory understanding of what was going on, in these « steady-state » protocols. I just note that testing under these conditions appears more strenuous than it was with the former « block trial » learning protocols.
I also tried to obtain detailed information on the quality of memorization of each of the images memorized within a set of n images in block-trial experiments. Consider for instance the case of subject PP memorizing 4 images. I have said that the total number of bits memorized on 4 images was less than the total number of bits memorized on 3 images. So, how is each of the 4 images memorized ? One can think of several possibilities. (i) if there is a linear arrangement of the items in memory slots, and if each acquisition of an item pushes the previous items one slot forward, we may expect that the last 3 items in the case of 4 images are perhaps nearly as well memorized as the 3 items in the case of 3 images, and the 1st item is pushed out of working memory. (ii) the situation would resemble that of a juggler who juggles easily with three balls. But when a fourth ball enters the game, one of the balls, at random, escapes (iii) we might be as in a phone cabin, occupied by three persons. A fourth person may also enter the cabin, but then everyone feels uncomfortable. The first two cases correspond to a localized memory model, with slots. The third case is in line with distributed memory models.
The data did not seem to follow any of these patterns. First, there were incomprehensible discrepancies between error-rates (from which the numbers of memorized bits are computed) and the reaction times. Second, when the testing order was reversed (from 1, 2, … n to n, n-1, … 1) the error-rates and the RTs for each image in a set of 4 could not be anticipated from the error-rates and RTs in the standard order. Furthermore, I did test myself systematically, using various testing order (e.g., 3-1-4-2, 2-3-4-1, etc). The results seemed erratic. It was not possible to characterize a « memorization quality » for each of the 4 images memorized in a row. It was not possible to predict the results obtained with one testing order from the results obtained with other testing orders. There were also discrepancies between the error-rates and the RT variations. Conceivably, when the experiments made use of a particular testing order, memory was adapting to this order and somewhat optimizing its placement and retrieval strategy to this order. So I decided to run experiments in which the testing orders were randomized.
VISUAL MEMORY EXPERIMENTS WITH
RANDOM TESTING ORDERS
Error levels as a function of testing rank
If a subject views N images numbered 1, 2, …., according to their presentation order, there are 2 possible testing orders for two images, 6 possible testing orders for 3 images, 24 testing orders for 4 images, N! possible testing orders for N images.
I focused first on experiments with 3 images. There were experimental blocks of 90 images, comprising 30 series of three images. Each of the 6 testing orders occurred 5 times within a block. Two subjects (SH and myself) were tested systematically, and some data were also contributed by a third subject (AT). There were unexpected, quite interesting structures in the results (see further down below).
I then performed experiments on memorizing 4 images, and testing them in each of the 24 possible permutations. Within an experimental block of 96 images, each testing order occurred once. There were again two subjects, SH and myself. The structure of the data, for both error rates and RTs was strikingly similar to that observed with the 3 images experiments.
Then I started testing on 5 images. Since there are 120 possible permutations, in this case, an experimental block containing all the permutations would require 600 images, which was too large. Furthermore a rather large number of measurements would have been required to obtain statistically significant observations on each condition. So I restrained the testing to a representative set of 20 different permutations. The results showed some evolution between the 4 images and the 5 images tests, which was not expected on the basis of mere extrapolation.
The results on error-rates for 3-5 images were quite interesting (See Fig. 5). They were described in [77] as follows:
---------------
More than 37,000 errors were scored involving over 290,000 tests. They follow simple rules:
(i) The error-rate on image N is initially very low, then rises progressively to the level of the other images. So, the "recency" effect of the last image fades gradually
(ii) There is perhaps a mild recency effect for image N-1, which soon disappears in the 3 and the 4 images experiments, and a clearly more pronounced effect in the 5 images experiments.
(iii) the error-rates on the other images are nearly equal, and remarkably independent of the testing stage in both the 3 and the 4 images data sets. Errors slightly increase with the testing stage in the 5 images data sets.
------
Figure 5. Recognition errors recorded in the 3, 4 and 5 images experiments.
Click here to
load the pdf file
The labels on the horizontal segment indicates the
presentation rank of the tested image. Results for a same image (same
label) tested at a same testing stage (rank given in abscissa) but
belonging to different permutations (for instance, image 2 in testing
permutations 123 and 321) were pooled. The number of trials
corresponding to each result, represented by a horizontal segment, is
the total number of responses, indicated on the top of the panel,
divided by the number of cases, thus 59,000/9, 115,000/16 and around
122,000/25 for the left, central and right panels respectively. The
figures in the ordinate are absolute numbers. The lower error counts
seen in the 5 images experiments is due to the lower number of trials
per result. The presentation times ranged from 1.65 to 1.9 sec, and
the image sizes were, depending on the subject, 5x4 or 5x5.
------
Two points are worth being highlighted, in relation to current concepts. The privilege of the last viewed, first tested image, over all other images, is extremely strong, in agreement with Phillips' picture of STVM (Phillips, 1983). On the other hand, the persistence of this privilege beyond the first testing stage was quite unexpected. Should we not expect each couple of images used in the tests, to occupy STVM one after the other? If this happened, the last memorized image should have lost its privilege immediately after the first test.
If the constancy of the error-rates, and their equality for images 1 and 2 in the 3 and 4 images data are not mere coincidences, but reflect deep constraints in the operations of memory, then we can reformulate the results as follows: In the N = 3 and N = 4 images random permutation experiments, there are exactly 4 characteristic error-levels. Numbering them 1, 2, 3, 4 from the lowest to the highest, levels 1, 2, 3 apply to image N at testing stages 1, 2, 3 and level 4 applies to all other cases.
People in the field have two ingredients, “recency effects” and “primacy effects”(e.g. [78-80]),with which they pretend to account for any kind of results. Invoking such effects is a mere restatement of the primary observation of, say, a U shaped relation between error-rates and rank of the memorized image. Similarly, in other domains of biology, one explains any observation in terms of “inhibition” and “stimulation” effects. Looking lucidly at the data, free from the ad hoc “primacy effects” and “recency effects”, the structure of the data given in points (i) – (iii) above emerges with great clarity (Fig. 5). After that, if one is really a recency-primacy addict, one can translate the real structure of the results into a recency-primacy description. The reverse translation does not make sense.
Reaction times as a function of testing rank.
The results on reaction times contained even more exciting structural details which were not present in the error-rates results. The RT for responding to a test on a given image at testing stage 2 or 3 depended significantly on which image was tested just before. For instance, the RT for a test on image 3 at testing stage 2 was shorter when image 2 was tested at stage 1 than when image 1 was tested at stage 1. The difference in RTs was observed despite the equality in error-rates for the two conditions. So, it is as though image 3 was maintained at a certain quality level at stage 2, but was more accessible to a memory search after a test on image 2 than after a test on image 1.
Dissociations between RT's and error-rates have also been observed in verbal memory studies (cf. [81-83]).
----------
Figure 6. Reaction time data on tests with 3 images.
Click here to
load the pdf file
The reaction times for image j tested at rank t > 1 are
split, in this representation, according to the image j
tested at rank t-1. The horizontal bars indicate the values of the
RTs, and the labels connected obliquely to the bars indicate the
couple ji, j being in smaller type. The standard deviations indicated
by the icons apply to t > 1. The RTs for image 3 at stage 1 are
0.682, 0.716, 0.574, 0.476, 0.642 and 0.508 for diagrams (a) to (f),
respectively. In (e) and (f) there were only 2 different testing
orders per experimental block of 90 images. The RTs were normalized
in such a way that the average RT, in an experimental block of 90
images, excluding the « last in first out »
cases, would be equal to one second.
---------
Patterns on consecutive tests.
You can think of many models to account for such an observation. For instance, in the case of 3 images, 1 2 and 3 are like aligned cars in a parking space. If you walk from one car to the other, it may be easier to reach car 3 from the location of car 2, than from the location of car 1. This is just one crude model to account for a single observation. However, the 6 testing permutations generate 12 different RTs on succesive tests (6 for each stage 1- stage 2 succession, 6 for each stage 2 – stage 3 succession). The set of the 12 RTs was not as simply structured as the example chosen above suggests.
Actually, the dominant pattern can be conceptualized by putting two images on one line, and the third image on another line, thus :
---- 1 --------- 2 ----
----------- 3 ---------
Imagine that downward motion is easier than upward motion, and that lateral motion is even more difficult than upward motion. Then you would have the following ranking for the 6 RTs :
23 and 13 : fast
31 and 32 : intermediate`
12 and 21 : slow
This ranking can be noted concisely in terms of inequalities between pairs of RTs, thus:
(23, 13) < (31, 32) < (12, 21)
where the figure in bold type gives the presentation rank of the image being tested at a certain testing stage, and the figure preceding it indicates the presentation rank of the image which was tested at the previous testing stage.
This ranking is indeed observed on the stage 2 RTs. But at stage 3 there is a clearly different ranking. Note also that the observation that 23 is faster than 13 is treated here as an additional detail, the main picture being provided by the classification of the 6 RTs into 3 classes.
With the same logic, a "horizontal" arrangement, with image 2 closer to image 3 than to image 1:
--- 3 --- 2 ------- 1 ----
would produce the inequalities:
(32, 23) < (12, 21) < (13, 31)
assuming that the three images are stored with equal accuracy.
A vertical arrangement, with image 2 at equal distances from image 1 and image 3:
---- 1 ----
---- 2 ----
---- 3 ----
could produce the inequalities:
(12, 23) < (21, 32) < 13 < 31
again, assuming that all three images are stored at equal levels of accuracy.
These considerations give a rough idea of how the experimental RT's could be used to deduce hypothetical configurations formed by memorized items. There are, however, two difficulties with this mapping strategy. The first relates to the fact that a distance such as "ab" relates to the slot occupied by image a at a given testing stage, and by image b at the next testing stage. So, the analysis works well on condition that the positions of a and b do not change from one testing stage to the next. Otherwise, the analysis provides only a rough sketch, to be refined by taking into account the complete sequence of events. The second difficulty has to do with the "storage at equal accuracy level" clause. We need a good theory connecting RT's and storage accuracy. With this theory, we could correct the RT's for the accuracy effect, and use them with greater confidence to deduce the geometry of the model.
In a preliminary work, in which I did not correct the RT's for the accuracy effect, I devised reasonably consistent sequence of events which could explain the observed RT patterns in the 3 images experiments, and in the 4 images experiments, up to testing rank 3 included. I expect that with more experimental work, in particular with 5 images, we will be in possession of decisive arguments on the geometry of memory, if there is one. In particular, I find it important to determine whether or not there is a small number of discrete accuracy and RT levels.
Towards a localized, architectural model
I propose now a broad interpretative framework within which the results may be tentatively explained. I consider a temporary store containing several slots (Fig. 7). There are slots providing good quality of storage, located near the entrance (at the bottom, on Fig. 7), and slots providing less detailed storage, located at the back (on the top, in Fig. 7). An item, after being stored in STVM moves upwards to a lower quality store, then sideways to leave the passage free for the next memorized items. Now, and this is a crucial hypothesis, the item may also move downwards, if space permits, in which case, there is no further loss of information. So an item may migrate from slot to slot, following a complex path, loosing information when going upwards, and maintaining the information constant when going sideways or downwards.
----------
Fig. 7. Model for visual memory.
Click here to
load the pdf file
This minimal structure with
seven slots A-G plus the STVM slot is proposed to account for the
patterns of reaction times observed in the 3 and 4 images
experiments. After residing in STVM, the memory traces would move to
A, then move upwards along the midline ACF and fill the rows by
moving sideways. During recognition tests, the traces would move, if
space permits, downwards from the EFG to the BCD row. One or two more
rows would be required to account for the 5 images results.
-------------
There are four rows of slots in the model of Fig. 7. The storage accuracy is assumed to decrease from row one which contains STVM to row four. At the end of the active memorization phase, the last image is in the first row, and all previous images occupy the fourth row, if space permits. Metaphorically, you may imagine a parking space with one entrance. When cars are coming in, you accommodate them by lining them at the back. Late comers are parked closer to the entrance.
As testings proceed, the last image moves up along the midline A-C-F. This description accounts well for the observations, in the 3 and 4 images experiments, of a nearly constant error-rate on all but the last image, and for a gradual increase of the error-rate on the last image. The situation is somewhat paradoxical, because if we reason in terms of a steady-state, we expect the fourth image to take the place of the third, the third to take the place of the second, and the second to take the place of the first. The results go clearly against such a steady-state view of memory.
The 5 images results are compatible with the idea that only three slots are available on the fourth row. The first three images would still be initially accommodated in the fourth row, but the fourth image would be initially accommodated in the third row. This neatly accounts for the observation of the differentiated status of the penultimage image, in the 5 images experiments, as contrasted with its behaviour in the 3 and the 4 images exepriments. Furthermore, as testings proceed, and more images are passively memorized, a crowding problem arises, and some images are eventually pushed to a fifth row. This accounts for the way error-rates increase with testing stage in the 5 images experiments, as compared with the situation observed in the 3 and the 4 images experiments.
Main features of the model
I discuss now some salient features of the model:
(i) The existence of rows of slots with different storage qualities: This was already postulated in [74], using the term "layer" instead of "row", on the basis of the stepwise character of the function relating storage accuracy to the number of memorized items. This aspect is further substantiated here by the results suggesting the existence of four error-rate levels in the 3 images and 4 images experiments.
(ii) The presence of only three slots in the fourth row: This is in line with the special status of the 4th image in the 5 images experiments.
(iii) The existence of horizontal connections between slots in a same row. There is no decisive argument for this feature. The connections may be indirect. In this case, we would have a tree-like geometry, and the row structure would be an abstract way of representing slots at the same depth in a tree. However, my attempts at explaining the RT patterns as reflecting searches on a tree were less satisfactory than those based on the model of Fig. 7.
(iv) (about vertical or oblique connections, see [77]).
(v) The triangular mesh geometry. If we discard the central slot C and connect slots B and D directly, we have a triangular mesh. Anatomically, we can view each slot as a column, represented by its cross-section in the model of Fig. 7. A triangular mesh arrangement is suitable for packing columns. I thus attempted to understand the RT patterns in terms of a pure triangular mesh. This worked well with the 3 images data set for JN, but there were problems with the other data sets. Most problems were alleviated by introducing the central slot C. The introduction of this slot, which was a late addition to the model, allowed me to rationalize the migration paths of the memorized items.
(vi) The mobile cursor. In early attempts at interpreting the data, I imagined that there was a fixed testing position, and that the memorized items would migrate to this position to be tested, one after the other. These attempts failed to produce anything convincing. On the other hand, the principle of testing the image where it is (or at an adjacent empty slot) allowed me to better rationalize the data, and in particular, explain “the first test singularity” (point vi).
(vi) The first test singularity, or why the RTs for some images may strongly decrease from the first to the second testing stage (see Figure 6 here, first and second tests on images 1 and 2, and [77]).
(vii) Beyond the 4th row. Using the same kind of connectivity (triangular mesh + central vertical path), we can add a fifth row containing 5 slots, which would bring the total to 12 slots + STVM. This would account for the sharp decline in recall accuracy observed beyond 9 or 10 memorized images [74]. However, adding more and more layers according to the same principle of connectivity does not make sense. In the triangular mesh geometry, the numbers of units per row are in arithmetic progression. Ultimately, we want to connect our memory module to the deep layers of LTM, in which huge numbers of items are stored. A geometric progression is more appropriate. So, we may imagine that each slot of the fifth row is at the origin of a binary tree, without horizontal connectivity, linking the slot to a section of LTM. We may even imagine an evolutionary script, in which memory was initially organized as a binary tree. The horizontal connections at the bottom would have been a very late invention, which created working memory and allowed it to operate in a very flexible manner.
Neural mechanisms
If my structural interpretations are taken literally, the memory traces must migrate from one location to another. It is common to speak, in the multi-store memory models, of information being transferred from one store to another. In a depiction of such statements, one imagines some neuronal module, in a store, encoding some information through the state of activation of its synapses, and some other neuronal module, copying or translating this information, through modifications of its own synapses. Then, we are led to think about neuronal mechanisms for copying or translating information and ask whether or not there may be smart neuronal chips for performing such tasks [39, 49]. Actually, the concept of "neuronal replication" is found, in disguise, in the field of stereoscopic vision. The task there is to compare two nearly identical images, and one way to do so, in theory, is to translate one image over the other in search of the best local matches. Indirect psychophysical results have been interpreted in terms of a neuronal superimposition mechanism ([55]). Perhaps then, the major implication of our results is that memory -as distinguished from learning - might well make use of a neuronal duplication mechanism.
--------------
This massive work (more than 300,000 memory tests, generating an equal number of RTs, and more than 40,000 errors) provided a picture of (visual) working memory, far more detailed than anything known before, and it made obsolete most of the previous publications on primacy and recency effects in visual memory. In my ingenuity, I thought that this big leap forward would be acclaimed as a seminal achievement, but it was, in fact, received with indifference or hostility by most of the established « leaders » in the field. The article describing the work was submitted successively to Nature, Nature Neurosciences, Cognitive Brain Research and Psychological Science and was, in all four cases, rejected with arguments which amounted to a negation of reality. I will say more about this in the forthcoming web section on science and society, and scientific ethics.
The successive titles were:
Structural model for visual memory deduced from reaction time patterns
(Nature and Nature Neurosciences)
Stratification of reaction times in visual memory experiments: clues for a structural model (Cognitive Brain Research – Editor: George Mangun)
Testing sequence effects in visual memory (Psychological Science – Editor: James Cutting)
The situation was quite unusual for me. I knew, by experience, that original theoretical work was rather more difficult to publish than vacuous bla-bla. On the other hand, it seemed to me that when a researcher came with massive, high quality data, there was no other attitude than to bow down and accept the facts. In particular, I remember the status of crystallographic work in molecular biology. A whole team of crystallographers needed years and years of strenuous labour to establish the low resolution structure of a protein. Once the structure was there, it could not be negated, and everyone working on the protein acknowledged the existence of this high quality data. The reactions of my peers in memory research (with the exception of William Phillips) who failed to acknowledge the existence of my data reveals the state of infantilism in which research on memory is being carried out all over the world.
One point of resemblance between my work on memory, and crystallographic work, besides their massive character, was the indirect relationship between the primary data and the interpretations. In crystallography, one collects X-ray diffraction spots. The collection of the spots do not provide a direct « photographic » picture of the molecule. From the location of the spots, their intensities, and additional information needed to solve the « phase problem », the 3d structure of the molecule may (hopefully) be deduced. There is a well established body of theory which predicts, with extreme accuracy, the relationship between a molecular structure, and its X-ray diffraction pattern. All the mathematical difficulty is in performing the reverse inferences, from diffraction spots to 3d structure. In the case of my memory experiments, I wished to deduce a geometrical arrangement of slots in memory from reaction times data, the assumption being that RTs reflected the length of the path from one slot to another. It was a working hypothesis, and it did not have the status of an established mathematical procedure. So, one is free to seek alternative interpretations (and I am myself open to a reinterpretation of the results). However, the results being there, it seems to me that on cannot discard them, and keep doing experiments on primacy and recency effects in the old fashioned way, or else keep producing “generic” models which carefully avoid confrontation with detailed experimental results.
In October 2003, I was invited at a meeting in Trieste (organized by Walter Gerbino and Corrado Caudek) and on this occasion, I presented my work on memory. One person in the audience was immediately seduced : Michael Kubovy. He stood up and said that this was the most interesting talk in the field to which he had listened in 40 years. I also had the occasion to discuss various aspects of my work with Johan Wagemans. Michael Kubovy happened to have good relations with Johan Wagemans, and Wagemans happened to be the recently appointed editor in chief of Acta Psychologica. So, I submitted my work to Acta Psychologica, and it was finally accepted there.
After that, I took a breath, and attempted to incite several colleagues, in France and abroad, to engage into this line of research. I proposed to send all my programs for performing the tests and analysing the data, in a completely gratuitous manner. I explained that the work they would perform would be their property, and that I would not put my name as a co-author on their articles. So far, I do not think that anyone pursued the line seriously.
REACTION TIME DISTRIBUTIONS
I had envisioned to perform more testing on myself, but I considered that (1) such extensive testing could become damaging to my health (both mental and physical) and (2) that it would be more appropriate to extract more refined information from the data, so as to focus on important issues before launching new experiments. In this respect, I am paying attention to reaction time distributions. All the previous analyses were based upon mean or median RTs. There is certainly more information in the whole RT distributions. My first incursion in this domain is reported in the abstract for ECVP 2007 in Arezzo, reproduced below :
-------------
Mental kinetics in visual memory.
Over 50 reaction time (RT) distributions were generated from a set of 500,000 RTs measured during published (e.g., Ninio, 2004 Acta Psychologica 116 263-283) or unpublished studies on visual memory. Most distributions for single subjects and restricted experimental conditions clearly deviate from the shifted lognormal distributions, classically supported by data pooled from various subjects or experimental conditions. The distributions were simulated with a 4-parameters model involving a lag, two consecutive elementary kinetic steps (exponential decays), and a Gaussian widening. The second kinetic step fits rather well the descending parts of the distributions. The steep ascending parts are controlled by the Gaussian and the first kinetic step. A number of issues, previously discussed in terms of comparative mean RTs, are re-examined in terms of shape modifications in the RT distributions. Discussed topics will include same versus different judgements and symmetry versus repetition recognition in short-term visual memory, serial position effects in working memory, stimulus complexity effects, left/right idiosyncrasies, and how the ratios of erroneous versus correct responses vary with time.
-------
Complexity.
Random images generated under uniform conditions nevertheless offer a range of difficulties in a memorization task. Furthermore, the differences within the target/distractor couples in the tests may be more or less salient. This introduces one more source of dispersion in the RT data. This is why I am attempting to develop complexity and saliency indicators. In particular, I am decomposing the images into separate all black or all white blocks. Up until now, such criteria have a very moderate impact on RTs, and a more marked one on error levels.
+++++++++++++++++++++
NOTES
[#1] From error-rates in memory experiments to an estimation of the number of memorized bits. The principle of the (simplified)model is the following. Assume that you have images of N binary elements drawn at random. If you never make recognition errors on these images, it implies that you are memorizing at least N bits on each image. If you make 50% errors, it implies that ou are choosing at random between the correct image and the distractor, so you are memorizing 0 bit per image. Error-rates intermediate between 0% and 50% correspond to average bit values memorized per image between N and 0. In order to assign a value, we make the assumption that a subject memorizes a surface of M quadrangles (M < N) in the image. Then if there are d differences between the correct image and the distractor, and the differences occur at random spatial positions, it is easy to compute the probability that none of the differences occur within the memorized area. If d = 1, p(1) = (N-M)/N ; if d = 2, p(2) = p(1)x(N-1-M)/(N-1), and so on. In this situation, in which all the differences lie outside the memorized region, the subject cannot identify the correct image, so his error-rate is p/2.
++++++++++++++++++++++++
[1] Shepard, R.N. (1967) Recognition memory for words, sentences and pictures. Journal of Verbal Learning and verbal behaviour behaviour 6, 156-163.
[2] Nickerson, R.S. (1968) A note on long-term recognition memory for pictorial material. Psychonomic Science 11, 58.
[3] Haber, R. N. (1970) How we remember what we see. Scientific American 222, N°5, 104-112.
[4] Standing, L. (1973) Learning 10,000 pictures. Quarterly Journal of Experimental Psychology 25, 207-222.
[5] Sperling, G. (1960) The information available in brief visual presentations. Psychological monographs, 74, 1-29. Se also Gegenfurtner, K.R. and Sperling, G. (1993) Information transfer in iconic memory experiments. Journal of Experimental Psychology 19, 845-866.
[6] Butler, B.E. (1981) Identification and localization in tachistocopic recognition: the effects of data and resource limitations. Canadian Journal of Psychology 35, 36-51.
[7] Ebbinghaus, H. (1885) Über das Gedächtnis. Unterschungen zur expeimentellen Psychologie. Duncker & Humboldt, Leipzig.
[8] Binet, A. (1894) Psychologie des grands calculateurs et joueurs d’échec. Hachette, Paris.
[9] Sternberg, S. (1966) High speed scanning in human memory. Science 153, 652-654.
[10] Itard, J.-M. (1807) Rappport fait à S.E. le ministre de l’Intérieur sur les nombreux développements et l’état actuel du sauvage de l’Aveyron. Imprimerie impériale, Paris.
[11] Ninio, J. (1989, 1991, 1996) L'empreinte des sens. Odile Jacob, Paris. (1991). Portuguese translation (1996): La impregnacão dos sentidos Instituto Piaget, Lisbon.
[12] Stroop, J. R. (1935) Studies of interference in serial verbal reactions. Journal of Experimental Psychology 18, 643-662.
[13] de Kerchkove, D. (1991) Brainframes. Technology, mind and business. Bosch & Keuning – BSO/ORIGIN Utrecht, The Nertherlands.
[14] Griffith, J.S. and Mahler, H.R. (1969) DNA ticketing theory of memory. Nature 223, 580-582.
[15] Griffith, J.S. (1967) Self-replication and scrapie. Nature 215, 1043-1044.
[16] McConnell, J.V. (1962) Memory transfer through cannibalism in planarians. J. Neuropsychiatry 3 (suppl.1), 42-48.
[17] Ungar, G. and Oceguera-Navarro, C. (1965) Transfer of habituation by materials extracted from brain. Nature 207, 301.
[18] Ungar, G., Galvan, L. and Clark, R.H. (1968) Chemical transfers of learned fear. Nature 217, 1259-1261.
[19] Anderson, J.A. (1972) A simple neural model generating an interactive memory. Mathematical Biosciences 14, 197-220.
[20] Cooper, L.N. (1973) A possible organization of animal memory and learning. Proceedings of the Nobel Symposium on Collective Properties of Physical Systems, Aspenasgarden, Sweden (Lindquist, B. and Lindquist, S., eds), pp. 252-264.
[21] Kohonen, T. (1977) Associative memory. Springer-Verlag, Berlin.
[22] Squire, L. R. (1986) Mechanims of memory. Science 232, 1612-1619.
[23] Kihlstrom, J. F. (1987) The cognitive unconscious. Science 237, 1445-1452.
[24] Waldrop, M. M. (1987) The workings of working memory. Science 237, 1564-1567.
[25] Kolers, P.A. (1968) Bilinguism and information processing. Scientific American 218 N°3, 78-86.
[26] Haber, R.N. (1969) Eidetic images. Scientific American 220, N°4, 36-44.
[27] Buckhout, R. (1974) Eyewitness testimony. Scientific American 231 N°6, 23-29.
[28] Loftus, E. F. (1979) The malleability of human memory. American Scientist 67, 312-320.
[29] Estes, W.K. (1980) Is human memory obsolete? American Scientist 68, 62-69.
[30] Hertz, M. (1929) Die Organisation der optischen Feldes bei der Biene. I: Z. vergl. Physiol. 8, 693-748. II: Ibid 11, 107-145.
[31] Wehner, R. and Räber, F. (1979) Visual spatial memory in desert ants, cataglyphis bicolor(Hymenoptera: Formicidae) Experientia 35, 1569- 1571.
[32] Anderson, A.M. (1979) A model for landmark learning in the honey-bee. Journal of Comparative Physiology 114, 335-355.
[33] Hölldobler, B. (1980) Canopy orientation: a new kind of orientation in ants. Science 210, 86-88.
[34] Collett, T.S. and Cartwright, B.A. (1983) Eidetic images in insects: their role in navigation. Trends in Neurosciences 6, 101-105.
[35] Ninio, J. (1998) Comment le cerveau donne forme à l'image. La Recherche 305, 103-109. Spanish translation (1998) Como el cerebro da forma a la imagen. Mundo cientifico 188, 29-35.
[36] Ninio, J. (2000) Aspects of human shape perception. In "Pattern formation in biology, vision and dynamics" (A. Carbone, M. Gromov and P. Prusinkiewicz, eds), World Scientific, Singapore, pp. 365-381.
[37] Miyashita, Y. (1993) Inferior temporal cortex: Where perception meets memory. Annual Review of Neurosciences 16, 245-263.
[38] Ellenbogen, J.M., Hu, P.T., Payne, J.D., Titone, D., and Walker, M.P. (2007) Human relational memory requires time and sleep. Proc. Nat. Acad. Sci. USA 104, 7723-7728.
[39] Ninio, J. (1988) Modèle de mémoire iconique localisée, réplicable et associative, utilisant le temps partagé et trois modes de communications neuronales. Comptes-Rendus Acad. Sci. Paris Série III, 306, 545-550.
[40] Amit, D.J. and Brunel, N. (1997) Dynamics of a recurrent network of spiking neurons before and following learning. Network 8, 373-404.
[41] Phillips, W.A. (1974) On the distinction between sensory storage and short-term visual memory. Perception & Psychophysics 16, 283-290.
[42] Bartlett, F.C. (1932) Remembering. Cambridge University Press, Cambridge.
[43] Carlson-Radvansky, L.A. and Irwin, D.E. (1995) Memory for structural information across eye movements. Journal of Experimental Psychology: Learning, Memory and Cognition 21, 1441-1458.
[44] McConkie, G. and Zola, D. (1979) Is visual information integrated across successive fixations in reading? Perception and Psychophysics 25, 221-224.
[45] Just, M.A. and Carpenter, P.A. (1980) A theory of reading: From eye fixation to comprehension. Psychological Review 87, 329-354.
[46] Potter, M.C. (1975) Meaning in visual search. Science 187, 965-966.
[47] Potter, M.C. (1993) Very short-term conceptual memory. Memory and cognition 21, 156-161.
[48] Thorpe, S., Fize, D. and Marlot, C. (1996) Speed of processing in the human visual system. Nature 381, 520-522.
[49] Ninio, J. (1996). Architecture for a replicative memory. In: V. Torre, F. Conti (Eds), Neurobiology: Ionic channels, neurons, and the brain (pp. 309-315). New-York: Plenum.
[50] Anderson, C.H. &. van Essen, D.C, (1987). Shifter circuits: A computational strategy for dynamic aspects of visual processing. Proc. Nat. Acad. Sci. U.S.A., 84, 6297-6301.
[51] Dehaene, S., Changeux, J.-P. and Nadal, J.-P. (1987) Neural networks that learn temporal sequences by selection. Proc. Nat. Acad. Sci. U.S.A 84, 2727-2731.
[52] Perky, C.W. (1910) An experimental study of imagination. American Journal of Psychology 21, 422-452.
[53] Kosslyn, S.M. (1986) Ghosts in the mind’s machine. W.W. Norton and Co., London.
[54] Luria, A.R. (1970) Une prodigieuse mémoire. Etude psychobiographique. Traduit du russe. Delachaux et Niestlé, Lausanne, Suisse.
[55] Herbomel, P. & Ninio, J. (1993). Processing of linear elements in stereopsis: Effects of positional and orientational distinctiveness. Vision Research, 33, 1813-1825
[56] Ninio, J. and O’Regan, J.K. (1996) The half-Zöllner illusion. Perception 25, 77-94.
[57] Phillips, W.A. (1983). Short-term visual memory. Philosophical Transactions of the Royal Society, London, B 302, 295-309.
[58] Phillips, W.A. & Christie, F.M. (1976). Components of visual memory. Quarterly Journal of Experimental Psychology, 29, 117-133.
[59] Brunel, N. & Ninio, J. (1997). Time to detect the difference between two images presented side by side. Cognitive Brain Research, 5,
273-282.
[60] Shelepin, Y.E., Kolesnikova, L.N. and Levkovich, Y.I. (1985) Recognition of objects and spatial frequency band of visual system transmission (in Russian), pp. 33-41.
[61] Ninio, J. and Mizraji, E. (1999) Processing of shapes, colours, and letters in short-term visual memory. Perception 28 supplement, p. 6.
[62] Barlow, H.B. and Reeves, B.C. (1979) The versatility and absolute efficiency of detecting mirror symmetry in random dot displays. Vision Research 19, 783-793.
[63] Wagemans, J. (1993) Skewed symmetry: a nonaccidental property used to perceive visual forms. J. Exp. Psychol. Hum. Percept. Perform. 19, 364-390.
[63a] van der Vloed, G., Csathó, A. and van der Helm, P.A. (2005) Symmetry and repetition in perspective. Acta Psychologica 120, 74-92.
[64] Tyler, C.W., editor (1996) Human symmetry perception and its computational analysis. VSP, Utrecht, the Netherlands.
[65] Evans, C. S., Wenderoth, P. and Cheng, K. (2000) Detection of bilateral symmetry in complex biological images. Perception 29, 31-42.
[66] Niimi, R., Watanabe, K. and Yokosawa, K. (2005) The role of visible persistence for perception of visual bilateral symmetry. Japanese Psychological Research 47, 262-270.
[67] Ninio, J. (2006) Processing of symmetrical shapes in short-term visual memory. Perception 35, supplement, 70-71.
[68] Caelli, T. and Julesz, B. (1978) On perceptual analyzers underlying visual texture discrimination : Part I. Biological Cybernetics 28, 167-175.
[69] (Reference to a book on exceptional memories). I forgot the title and the name of the author.
[70] Murray, D.J. (1976) Research on human memory in the nineteenth century. Canadian Journal of Psychology, 30, 201-220.
[71] Miller, G.A. ((1956) The magical number seven plus or minus two: some limits on our capacity for processing information. Psychological Review 63, 81-97.
[72] Cavanagh, J.P. (1972). Relation between the immediate memory span and the memory search rate. Psychological Review , 79, 525-530.
[73] Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacities. Behavioural and Brain Sciences, 24, 87-185.
[74] Ninio, J. (1998). Acquisition of shape information in working memory, as a function of viewing time and number of consecutive images: evidence for a succession of discrete storage classes. Cognitive Brain Research, 7, 57-69.
[75] Amit, D.J., Bernacchia, A. and Yakovlev, V. (2003) Multiple-object working memory – A model for behavioral performance. Cerebral Cortex 13, 435-443.
[76] Yakovlev, V., Bernacchia, A., Orlov, T., Hochstein, S. and Amit, D. (2005) Multi-item working memory – A behavioral study. Cerebral cortex 15, 602-615.
[77] Ninio, J. (2004) Testing sequence effects in visual memory: clues for a structural model. Acta Psychologica, 116, 263-283.
[78] Crowder, R. (1976) Principles of Learning and Memory.Erlbaum, Hillsdale, New-Jersey.
[79] Knoedler, A.J., Hellwig, K.A. & Neath, I. (1999) The shift from recency to primacy with increasing delay. Journal of Experimental Psychology: Learning, Memory and Cognition, 25, 474-487.
[80] Avons, S.E. (1998) Serial report and item recognition of novel visual patterns. British Journal of Psychology 89, 285-308.
[81] Anderson, J.R. & Matessa, M. (1997). A production system theory of serial memory. Psychological Review, 104, 728-748.
[82] Maybery, M.T., Parmentier, F.B.R. & Jones, D.M. (2002). Grouping of list items reflected in the timing of recall: implications for models of serial verbal memory. Journal of Memory and Language, 47, 360-385.
[83] Oberauer, K. (2003). Understanding serial position curves in short-term recognition and recall. Journal of Memory and Language, 49, 469-483.
Xxxxxxxxxxxxxxxxxxxxxxxxx
A FEW OTHER REFERENCES ON MEMORY
Baddeley, A.D. (1986).Working memory. Oxford: Clarendon Press.
Baddeley, A.D. & Hitch, G. (1974). Working memory. In Bower, G. (Ed.), The Psychology of Learning and Motivation: Advances in Research and Theory (Vol. 8, pp 47-90). New-York: Academic Press.
Brandimonte, M.A., Hitch, G.J., & Bishop, D.M. (1992) Verbal recoding of visual stimuli impairs mental images transformations. Memory and Cognition, 20, 449-455.
Cowan, N., Saults, S.J., Elliott, E.M. & Moreno, M.V. (2002). Deconfounding serial recall. Journal of Memory and Language , 46, 153-177.
Luck, S.J. and Vogel, E.K. (1997) The capacity of visual working memory for features and conjunctions. Nature 390, 279-280.
Malsburg, Ch. Von der, and Willshaw, D.J. (1977) How to label nerve cells so that they can interconnect in an ordered fashion. Proc. Nat. Acad. Sci. USA 74, 5176-5178.
Morton, J., Hammerseley, R. D., and Bekerian, D.A. (1985) Headed records : A model for memory and its failures. Cognition 20, 1-23.
Tulving, E. and Schacter, D. L. (1990) Priming and human memory systems. Science 247, 301-306.
