The Information Bottleneck Method (Tishby, Pereira, Bialek 1999) is a principled information theoretic framework for extracting relevant features of one random variable about another one. It has been applied successfully to a variety of problems, such as text categorization, neural coding, and bioinformatics. In this talk I will discuss the principle, its theoretical basis, and its recent multivariate extension, with emphasis on possible application to multivariate feature selection. In particular this extension suggests possible interpretation of good features as "approximate sufficient partitions" of datasets. It may also provide a geometric interpretation of features as sufficient projection to lower dimensions.
For related references see:
http://www.cs.huji.ac.il/labs/learning/Papers/IBM_list.html