Friday, October 24, 2003

12 noon

Redwood Neuroscience Institute

Title: "Energy-based Models for Sparse Overcomplete Representations"

Yee Whye Teh

Computer Science, University of California at Berkeley

Abstract:

I will present a new way of extending independent components analysis (ICA) to overcomplete representations. In contrast to the usual causal generative extensions of ICA which maintain marginal independence of sources, we define features as deterministic (linear) functions of the inputs. This assumption results in marginal dependencies among the features, but conditional independence of the features given the inputs. By assigning energies to the features a probability distribution over the input states is defined through the Boltzmann distribution. Free parameters of this model are trained using the contrastive divergence objective (Hinton 2002). When the number of features is equal to the number of input dimensions this energy-based model reduces to noiseless ICA and we show experimentally that the proposed learning algorithm is able to perform blind source separation on speech data. In additional experiments we train overcomplete energy-based models to extract features from various standard data-sets containing speech, natural images, hand-written digits and faces.

This is work done in collaboration with Geoff Hinton, Max Welling and Simon Osindero. A draft of the paper on which this talk is based on can be obtained at:

http://www.cs.berkeley.edu/~ywteh/research/ebm/ebmica.ps.gz