### Neural Associative Memories

Neural associative memories (NAM) are neural network
models consisting of neuron-like and synapse-like elements. At any given
point in time the state of the neural network is given by the vector of neural
activities, it is called the *activity pattern*. Neurons
update their activity values based on the inputs they receive (over the synapses).
In the simplest neural network models the input-output function of a neuron
is the identitiy function or a threshold operation. Note that in the latter
case the neural activity state is binary: active or inactive. Information
to be processed by the neural network is represented by activity patterns
(for instance, the representation of a tree can an activity pattern where
active neurons display a tree's picture). Thus, activity patterns are the
*representations* of the elements processed in the network. A representation
is called *sparse*
if the ratio between active and inactive neurons is small.

The synapses in a neural network are the links between
neurons or between neurons and fibers carrying external input. A synapse
transmits presynaptic activity to the postsynaptic site. In the simplest
neural network models each synapse has a weight value and the postsynaptic
signal is simply the product of presynaptic activity and weight. The input
of a neuron is then the sum of the postsynaptic signals of all synapses
connecting the neuron. Thus, information is processed in a neural network
by activity spread. How activity spreads, and by this, which algorithm is
implemented in the network depends on how the synaptic structure, the matrix
of synaptic weights in the network is shaped by learning. In neural associative
memories the learning provides the storage of a (large) set of activity patterns
during learning, the memory patterns.

Different memory functions are defined by the way how
learned patterns can be selectively accessed by an input pattern. The function
of *pattern recognition* means to classify input patterns in two classes,
familiar patterns and the rest. *Pattern association* describes the
function to associate certain input patterns with certain memory patterns,
i.e., each stored memory consists of a pair of input and desired output
pattern.

The advantage of neural associative memories over other
pattern storage algorithms like lookup tables of hash codes is that the
memory access can be fault tolerant with respect to variation of the input
pattern. For pattern association this means that an output pattern
can be produced for a set of input patterns that are (with respect to a
metric like the Hamming distance) closest to the input pattern presented
during learning. Fault tolerance is key in the function of content-addressed pattern
retrieval or *pattern completion*. Here the pattern pairs used during
learning have to consist of two identical patterns which one also calls autoassociative
learning. In this case if the input patterns are noisy versions of the learned
patterns, the pattern association can be seen as pattern completion, i.e.,
the noisy version is replaced by the noiseless version of the pattern.

One discerns two different retrieval mechanisms corresponding
to feed-forward or feed-back architectures of the neural network: In one-step
retrieval each network neuron evaluates its input just once. In *iterative
retrieval* activity flows through feedback connections and neurons evaluate
their inputs several times during the retrieval process.

Training or learning in the network means that neural activity
patterns can influence the synaptic structure. The rules governing this influence
are called synaptic learning rules. The simplest form is a *local learning
rule* where synaptic weight change depends only on pre- and postsynaptic
activity, i.e., the signals locally available at the synapse.

In associative memories many associations can be stored
at the same time. There are different *schemes of superposition of the memory
traces* formed by the different associations. The superposition can be
simple linear addition of the synaptic changes required for each association
(like in the Hopfield model) or nonlinear. The Willshaw model, for instance
employs a clipped superposition (resulting in binary weights) that is nonlinear.
There are different models of associative memory with respect to the learning
procedure. It can either be one-shot where each learning pattern is presented
once or recurrent. In recurrent training the pattern retrieval is checked
and the patterns are presented multiple times until a desired level of retrieval
quality is achieved.

The performance of neural associative memories
is usually measured by a quantity called *information capacity*, that
is, the information content that can be learned and retrieved, devided by
the number of synapses required.