Neural Associative Memories


    Neural associative memories (NAM) are neural network models consisting of neuron-like and synapse-like elements. At any given point in time the state of the neural network is given by the vector of neural activities, it is called the activity pattern. Neurons update their activity values based on the inputs they receive (over the synapses). In the simplest neural network models the input-output function of a neuron is the identitiy function or a threshold operation. Note that in the latter case the neural activity state is binary: active or inactive. Information to be processed by the neural network is represented by activity patterns (for instance, the representation of a tree can an activity pattern where active neurons display a tree's picture). Thus, activity patterns are the representations of the elements processed in the network. A representation is called sparse if the ratio between active and inactive neurons is small.

    The synapses in a neural network are the links between neurons or between neurons and fibers carrying external input. A synapse transmits presynaptic activity to the postsynaptic site. In the simplest neural network models each synapse has a weight value and the postsynaptic signal is simply the product of presynaptic activity and weight. The input of a neuron is then the sum of the postsynaptic signals of all synapses connecting the neuron. Thus, information is processed in a neural network by activity spread. How activity spreads, and by this, which algorithm is implemented in the network depends on how the synaptic structure, the matrix of synaptic weights in the network is shaped by learning. In neural associative memories the learning provides the storage of a (large) set of activity patterns during learning, the memory patterns.
    Different memory functions are defined by the way how learned patterns can be selectively accessed by an input pattern. The function of pattern recognition means to classify input patterns in two classes, familiar patterns and the rest. Pattern association describes the function to associate certain input patterns with certain memory patterns, i.e., each stored memory consists of a pair of input and desired output pattern.
    The advantage of neural associative memories over other pattern storage algorithms like lookup tables of hash codes is that the memory access can be fault tolerant with respect to variation of the input  pattern. For pattern association this means that an output pattern can be produced for a set of input patterns that are (with respect to a metric like the Hamming distance) closest to the input pattern presented during learning. Fault tolerance is key in the function of content-addressed pattern retrieval or pattern completion. Here the pattern pairs used during learning have to consist of two identical patterns which one also calls autoassociative learning. In this case if the input patterns are noisy versions of the learned patterns, the pattern association can be seen as pattern completion, i.e., the noisy version is replaced by the noiseless version of the pattern.


    One discerns two different retrieval mechanisms corresponding to feed-forward or feed-back architectures of the neural network: In one-step retrieval each network neuron evaluates its input just once. In iterative retrieval activity flows through feedback connections and neurons evaluate their inputs several times during the retrieval process.

    Training or learning in the network means that neural activity patterns can influence the synaptic structure. The rules governing this influence are called synaptic learning rules. The simplest form is a local learning rule where synaptic weight change depends only on pre- and postsynaptic activity, i.e., the signals locally available at the synapse.
    In associative memories many associations can be stored at the same time. There are different schemes of superposition of the memory traces formed by the different associations. The superposition can be simple linear addition of the synaptic changes required for each association (like in the Hopfield model) or nonlinear. The Willshaw model, for instance employs a clipped superposition (resulting in binary weights) that is nonlinear. There are different models of associative memory with respect to the learning procedure. It can either be one-shot where each learning pattern is presented once or recurrent. In recurrent training the pattern retrieval is checked and the patterns are presented multiple times until a desired level of retrieval quality is achieved. 

    The performance of  neural associative memories is usually measured by a quantity called information capacity, that is, the information content that can be learned and retrieved, devided by the number of synapses required.