Title: ICA Using Spacings Estimates of Entropy

Erik Miller

University of California at Berkeley

Abstract:
We present a new algorithm for the ICA problem based on efficient entropy estimators from the statistics literature. In extensive testing, the method outperforms all current methods of which we are aware, including the recently published Kernel-ICA method. Like many previous methods, this algorithm directly minimizes the measure of departure from independence according to the estimated Kullback-Leibler divergence between the joint distribution and the product of the marginal distributions. The entropy estimator used to achieve this minimization is consistent (in the statistical sense) and is asymptotically efficient. It is also computationally simple, requiring only O(N log N) time to compute. We show how the method can be used in higher dimensions, with experiments on up to 16 sources.

Time permitting, we will briefly discuss recent work in which we generalize "spacings" methods to higher dimensions. We show how to "paint" uniform discrete measures onto arbitrary continuous probability spaces. We give an application to estimating the KL-divergence between two high-dimensional distributions.

(Joint work with John Fisher, MIT)