This page is under construction, as I put papers on the
Internet.  If you have problems with printing -- for example,
if the printer complains about paper size -- please let me know
<pkanerva (AT) rni (DOT) org> and I will try to make a more
accommodating version.

_______________________________________________________
Kanerva, P.  Sparse Distributed Memory (with Foreword by Douglas
    R. Hofstadter).  Cambridge, Mass.: MIT Press, 1988.

ABSTRACT.  This is my PhD thesis, plus a chapter on the Organization
of an Autonomous Learning System.  What kind of a model--i.e.,
mathematical space and operations--could explain human memory?
Large bit patterns (i.e., long bit strings, high-dimensional spaces)
have properties that correspond to properties of memory recall and
recognition.  A neural-net architecture is outlined for building very
large memories.  Such an architecture resembles the wiring of the
cerebellum and is very similar to the Marr and Albus models of the
cerebellum.  The final chapter on autonomous learning systems suggests
areas of research toward a better understanding of living systems
controlled by brains.

_______________________________________________________
Kanerva, P.  "Sparse Distributed Memory and related models."  In
    M.H. Hassoun (ed.), Associative Neural Memories: Theory and
    Implementation; 50-76.  New York: Oxford University Press, 1993.

ABSTRACT.  The chapter describes Sparse Distributed Memory (SDM)
as a neural-net associative memory.  It is characterized by two weight
matrices and by a large internal dimensionality--the number of hidden
units is orders of magnitude larger than the number of input or output
units.  The first matrix, A, is fixed and possibly random, and the
second matrix, C, is modifiable.  The paper compares and contrasts SDM
to (1) computer memory (RAM), (2) correlation-matrix memory (CMM),
(3) feed-forward artificial neural network, (4) cortex of the cerebellum,
(5) Marr and Albus models of the cerebellum, and (6) Albus' Cerebellar
Model Arithmetic Computer (CMAC).  Several variants of the basic
design are discussed: the selected-coordinate design and the
hyperplane design of Jaeckel, the pseudorandom associative neural
memory of Hassoun, and SDM with real-valued input variables by Prager
and Fallside.  SDM research conducted mainly at the Research Institute
for Advanced Computer Science at NASA Ames in 1986-1991 is
highlighted.

The on-line version is faithful to the book, with one or two updates.
The text and the illustrations are in separate files.
  Text:
  http://www.rni.org/kanerva/sdmchapter-text.pdf (226 KB)
  http://www.rni.org/kanerva/sdmchapter-text.ps.gz (189 KB)
  Illustrations:
  http://www.rni.org/kanerva/sdmchapter-figs.pdf (141 KB)
  http://www.rni.org/kanerva/sdmchapter-figs.ps.gz (75 KB)

_______________________________________________________
Kanerva, P., Sjoedin, G., Kristoferson, J., Karlsson, R., Levin, B.,
    Holst, A., Karlgren, J., and Sahlgren, M.  "Computing with large
    random patterns."  In Uesaka, Y., Kanerva, P., and Asoh, H. (eds.)
    Foundations of Real-World Intelligence.  Stanford, Calif.: CSLI
    Publications, 2001.

ABSTRACT.  The chapter describes a style of computing that differs
from traditional numeric and symbolic computing and is suited for
neural networks.  The focus is on one aspect of "neurocomputing,"
namely, computing with large random patterns, or high-dimensional
random vectors.  What kind of computing can be done with them and
how might that help us understand information processing by the brain
and the workings of the mind?  Rapidly developing hardware technology
will soon be able to produce the massive circuits that this style of
computing requires, but we also need a theory on which the computing
can be founded.
    Simple settings are used to illustrate principles of pattern
computing.  The first section describes the spatter code, which is a
binary realization of Plate's Holographic Reduced Representation and
demonstrates its use in modeling analogy.  The key idea is uniform
representation, so that superficially everything looks the same:
a high-dimensional binary vector can represent a simple object, or
a complex object made of simpler ones, or a property, or a relation
between objects, or a mapping of objects.  Dense binary vectors (0s
and 1s are equally likely) work poorly in large systems.  The problem
is avoided with sparse representations, which are explained and
explored mathematically in the second and third sections.  An item
memory or "clean-up" memory is an essential part of any computing
architecture based on large random patterns.  The next section
describes its efficient simulation on a conventional computer.  The
last section takes on challenges presented by language.  First, sparse
random vectors are used for capturing meanings of words from text,
and finally the ultimate challenge for brainlike computing is outlined,
namely, dealing with the remarkable flexibility with which we use
language.

  http://www.rni.org/kanerva/rwi-sics.pdf (806 Kbytes)
  http://www.rni.org/kanerva/rwi-sics.ps.gz (493 Kbytes)

For other titles by our publisher, see
  http://csli-publications.stanford.edu

_______________________________________________________
Kanerva, P., Kristoferson, J., and Holst, A.  "Random indexing of
    text samples for Latent Semantic Analysis."  In L.R. Gleitman and
    A.K. Josh (eds.), Proc. 22nd Annual Conference of the Cognitive
    Science Society (Philadelphia), p. 1036.  Mahwah, New Jersey:
    Erlbaum, 2000.

ABSTRACT.  In statistical study of natural language, word frequencies
are often collected into a huge matrix, which is then analyzed
mathematically.  However, matrices for very large text corpora can be
too large for computers to handle.  This poster describes a simple
method of limiting the size of the matrix without limiting the size of
the corpus.  The method is based on sparse distributed representation
and is explained with reference to Latent Semantic Analysis/Indexing
(LSA/LSI).

One-page extended abstract from the Proceedings (has some errors):
  http://www.rni.org/kanerva/cogsci2k-abstract.ps (59 KB)

The actual poster, which is more informative than the above.  For
viewing in fixed-width (Courier) font:
  http://www.rni.org/kanerva/cogsci2k-poster.txt (13 KB)

_______________________________________________________
Kanerva, P.  "The Spatter Code for encoding concepts at many levels."
    In M. Marinaro and P.G. Morasso (eds.), ICANN '94, Proceedings of
    International Conference on Artificial Neural Networks (Sorrento,
    Italy), vol. 1, pp. 226-229.  London: Springer-Verlag, 1994.

ABSTRACT.  The first paper on the Spatter Code, which is a
high-dimensional (e.g., N = 10,000), random code that encodes
"high-level concepts" in terms of their "low-level attributes" so that
concepts at different levels can be mixed freely.  The binary spatter
code is the simplest.  It has two N-bit codewords for each concept or
item, a "high-level" or dense word with many randomly placed 1s, and
a "low-level" or sparse word with a few 1s (that are contained in the
many).  The dense codewords can be used as inputs to an associative
memory.  The sparse codewords are used in encoding new concepts.

  http://www.rni.org/kanerva/icann94.pdf (39 Kbytes)
  http://www.rni.org/kanerva/icann94-a4.ps.gz (21 Kbytes)
  http://www.rni.org/kanerva/icann94-letter.ps.gz (36 KB)

_______________________________________________________
Kanerva, P.  "Fully distributed representation."  Proc. 1997 Real
    World Computing Symposium (RWC'97, Tokyo).  Report TR-96001,
    pp. 358-365.  Real World Computing Partnership, Tsukuba-city,
    Japan, 1997.

ABSTRACT.  A more complete paper on the spatter code.  It shows
how the information of a conventional computer record with fields is
converted into a long random bit string, or a holistic record, that
has no fields, and how the "fields" are extracted from the holistic
record.  It is argued that holistic representation should be used in
modeling high-level mental functions.

  http://www.rni.org/kanerva/rwc97.ps.gz (45 Kbytes)