Publications | Max Planck Institute for Intelligent Systems

63 results (View BibTeX file of all listed publications)

2005

Kernel Methods for Measuring Independence

Gretton, A., Herbrich, R., Smola, A., Bousquet, O., Schölkopf, B.

Journal of Machine Learning Research, 6, pages: 2075-2129, December 2005 (article)

Abstract

We introduce two new functionals, the constrained covariance and the kernel mutual information, to measure the degree of independence of random variables. These quantities are both based on the covariance between functions of the random variables in reproducing kernel Hilbert spaces (RKHSs). We prove that when the RKHSs are universal, both functionals are zero if and only if the random variables are pairwise independent. We also show that the kernel mutual information is an upper bound near independence on the Parzen window estimate of the mutual information. Analogous results apply for two correlation-based dependence functionals introduced earlier: we show the kernel canonical correlation and the kernel generalised variance to be independence measures for universal kernels, and prove the latter to be an upper bound on the mutual information near independence. The performance of the kernel dependence functionals in measuring independence is verified in the context of independent component analysis.

PDF PostScript PDF [BibTex]

2005

ei Gretton, A., Herbrich, R., Smola, A., Bousquet, O., Schölkopf, B. Kernel Methods for Measuring Independence Journal of Machine Learning Research, 6, pages: 2075-2129, December 2005 (article)

PDF PostScript PDF [BibTex]

Some thoughts about Gaussian Processes

Chapelle, O.

NIPS Workshop on Open Problems in Gaussian Processes for Machine Learning, December 2005 (talk)

PDF Web [BibTex]

ei Chapelle, O. Some thoughts about Gaussian Processes NIPS Workshop on Open Problems in Gaussian Processes for Machine Learning, December 2005 (talk)

PDF Web [BibTex]

A taxonomy of semi-supervised learning algorithms

Chapelle, O.

Yahoo!, December 2005 (talk)

PDF [BibTex]

ei Chapelle, O. A taxonomy of semi-supervised learning algorithms Yahoo!, December 2005 (talk)

PDF [BibTex]

A Unifying View of Sparse Approximate Gaussian Process Regression

Quinonero Candela, J., Rasmussen, C.

Journal of Machine Learning Research, 6, pages: 1935-1959, December 2005 (article)

Abstract

We provide a new unifying view, including all existing proper probabilistic sparse approximations for Gaussian process regression. Our approach relies on expressing the effective prior which the methods are using. This allows new insights to be gained, and highlights the relationship between existing methods. It also allows for a clear theoretically justified ranking of the closeness of the known approximations to the corresponding full GPs. Finally we point directly to designs of new better sparse approximations, combining the best of the existing strategies, within attractive computational constraints.

PDF [BibTex]

ei Quinonero Candela, J., Rasmussen, C. A Unifying View of Sparse Approximate Gaussian Process Regression Journal of Machine Learning Research, 6, pages: 1935-1959, December 2005 (article)

PDF [BibTex]

Maximal Margin Classification for Metric Spaces

Hein, M., Bousquet, O., Schölkopf, B.

Journal of Computer and System Sciences, 71(3):333-359, October 2005 (article)

Abstract

In order to apply the maximum margin method in arbitrary metric spaces, we suggest to embed the metric space into a Banach or Hilbert space and to perform linear classification in this space. We propose several embeddings and recall that an isometric embedding in a Banach space is always possible while an isometric embedding in a Hilbert space is only possible for certain metric spaces. As a result, we obtain a general maximum margin classification algorithm for arbitrary metric spaces (whose solution is approximated by an algorithm of Graepel. Interestingly enough, the embedding approach, when applied to a metric which can be embedded into a Hilbert space, yields the SVM algorithm, which emphasizes the fact that its solution depends on the metric and not on the kernel. Furthermore we give upper bounds of the capacity of the function classes corresponding to both embeddings in terms of Rademacher averages. Finally we compare the capacities of these function classes directly.

PDF PDF DOI [BibTex]

ei Hein, M., Bousquet, O., Schölkopf, B. Maximal Margin Classification for Metric Spaces Journal of Computer and System Sciences, 71(3):333-359, October 2005 (article)

PDF PDF DOI [BibTex]

Selective integration of multiple biological data for supervised network inference

Kato, T., Tsuda, K., Asai, K.

Bioinformatics, 21(10):2488 , October 2005 (article)

PDF [BibTex]

ei Kato, T., Tsuda, K., Asai, K. Selective integration of multiple biological data for supervised network inference Bioinformatics, 21(10):2488 , October 2005 (article)

PDF [BibTex]

Assessing Approximate Inference for Binary Gaussian Process Classification

Kuss, M., Rasmussen, C.

Journal of Machine Learning Research, 6, pages: 1679 , October 2005 (article)

Abstract

Gaussian process priors can be used to define flexible, probabilistic classification models. Unfortunately exact Bayesian inference is analytically intractable and various approximation techniques have been proposed. In this work we review and compare Laplace‘s method and Expectation Propagation for approximate Bayesian inference in the binary Gaussian process classification model. We present a comprehensive comparison of the approximations, their predictive performance and marginal likelihood estimates to results obtained by MCMC sampling. We explain theoretically and corroborate empirically the advantages of Expectation Propagation compared to Laplace‘s method.

PDF PDF [BibTex]

ei Kuss, M., Rasmussen, C. Assessing Approximate Inference for Binary Gaussian Process Classification Journal of Machine Learning Research, 6, pages: 1679 , October 2005 (article)

PDF PDF [BibTex]

Clustering on the Unit Hypersphere using von Mises-Fisher Distributions

Banerjee, A., Dhillon, I., Ghosh, J., Sra, S.

Journal of Machine Learning Research, 6, pages: 1345-1382, September 2005 (article)

Abstract

Several large scale data mining applications, such as text categorization and gene expression analysis, involve high-dimensional data that is also inherently directional in nature. Often such data is L2 normalized so that it lies on the surface of a unit hypersphere. Popular models such as (mixtures of) multi-variate Gaussians are inadequate for characterizing such data. This paper proposes a generative mixture-model approach to clustering directional data based on the von Mises-Fisher (vMF) distribution, which arises naturally for data distributed on the unit hypersphere. In particular, we derive and analyze two variants of the Expectation Maximization (EM) framework for estimating the mean and concentration parameters of this mixture. Numerical estimation of the concentration parameters is non-trivial in high dimensions since it involves functional inversion of ratios of Bessel functions. We also formulate two clustering algorithms corresponding to the variants of EM that we derive. Our approach provides a theoretical basis for the use of cosine similarity that has been widely employed by the information retrieval community, and obtains the spherical kmeans algorithm (kmeans with cosine similarity) as a special case of both variants. Empirical results on clustering of high-dimensional text and gene-expression data based on a mixture of vMF distributions show that the ability to estimate the concentration parameter for each vMF component, which is not present in existing approaches, yields superior results, especially for difficult clustering tasks in high-dimensional spaces.

PDF [BibTex]

ei Banerjee, A., Dhillon, I., Ghosh, J., Sra, S. Clustering on the Unit Hypersphere using von Mises-Fisher Distributions Journal of Machine Learning Research, 6, pages: 1345-1382, September 2005 (article)

PDF [BibTex]

Support Vector Machines for 3D Shape Processing

Steinke, F., Schölkopf, B., Blanz, V.

Computer Graphics Forum, 24(3, EUROGRAPHICS 2005):285-294, September 2005 (article)

Abstract

We propose statistical learning methods for approximating implicit surfaces and computing dense 3D deformation fields. Our approach is based on Support Vector (SV) Machines, which are state of the art in machine learning. It is straightforward to implement and computationally competitive; its parameters can be automatically set using standard machine learning methods. The surface approximation is based on a modified Support Vector regression. We present applications to 3D head reconstruction, including automatic removal of outliers and hole filling. In a second step, we build on our SV representation to compute dense 3D deformation fields between two objects. The fields are computed using a generalized SVMachine enforcing correspondence between the previously learned implicit SV object representations, as well as correspondences between feature points if such points are available. We apply the method to the morphing of 3D heads and other objects.

PDF [BibTex]

ei Steinke, F., Schölkopf, B., Blanz, V. Support Vector Machines for 3D Shape Processing Computer Graphics Forum, 24(3, EUROGRAPHICS 2005):285-294, September 2005 (article)

PDF [BibTex]

Fast Protein Classification with Multiple Networks

Tsuda, K., Shin, H., Schölkopf, B.

Bioinformatics, 21(Suppl. 2):59-65, September 2005 (article)

Abstract

Support vector machines (SVM) have been successfully used to classify proteins into functional categories. Recently, to integrate multiple data sources, a semidefinite programming (SDP) based SVM method was introduced Lanckriet et al (2004). In SDP/SVM, multiple kernel matrices corresponding to each of data sources are combined with weights obtained by solving an SDP. However, when trying to apply SDP/SVM to large problems, the computational cost can become prohibitive, since both converting the data to a kernel matrix for the SVM and solving the SDP are time and memory demanding. Another application-specific drawback arises when some of the data sources are protein networks. A common method of converting the network to a kernel matrix is the diffusion kernel method, which has time complexity of O(n^3), and produces a dense matrix of size n x n. We propose an efficient method of protein classification using multiple protein networks. Available protein networks, such as a physical interaction network or a metabolic network, can be directly incorporated. Vectorial data can also be incorporated after conversion into a network by means of neighbor point connection. Similarly to the SDP/SVM method, the combination weights are obtained by convex optimization. Due to the sparsity of network edges, the computation time is nearly linear in the number of edges of the combined network. Additionally, the combination weights provide information useful for discarding noisy or irrelevant networks. Experiments on function prediction of 3588 yeast proteins show promising results: the computation time is enormously reduced, while the accuracy is still comparable to the SDP/SVM method.

PDF Web DOI [BibTex]

ei Tsuda, K., Shin, H., Schölkopf, B. Fast Protein Classification with Multiple Networks Bioinformatics, 21(Suppl. 2):59-65, September 2005 (article)

PDF Web DOI [BibTex]

Iterative Kernel Principal Component Analysis for Image Modeling

Kim, K., Franz, M., Schölkopf, B.

IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(9):1351-1366, September 2005 (article)

Abstract

In recent years, Kernel Principal Component Analysis (KPCA) has been suggested for various image processing tasks requiring an image model such as, e.g., denoising or compression. The original form of KPCA, however, can be only applied to strongly restricted image classes due to the limited number of training examples that can be processed. We therefore propose a new iterative method for performing KPCA, the Kernel Hebbian Algorithm which iteratively estimates the Kernel Principal Components with only linear order memory complexity. In our experiments, we compute models for complex image classes such as faces and natural images which require a large number of training examples. The resulting image models are tested in single-frame super-resolution and denoising applications. The KPCA model is not specifically tailored to these tasks; in fact, the same model can be used in super-resolution with variable input resolution, or denoising with unknown noise characteristics. In spite of this, both super-resolution a nd denoising performance are comparable to existing methods.

Web DOI [BibTex]

ei Kim, K., Franz, M., Schölkopf, B. Iterative Kernel Principal Component Analysis for Image Modeling IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(9):1351-1366, September 2005 (article)

Web DOI [BibTex]

Phenotypic characterization of chondrosarcoma-derived cell lines

Schorle, C., Finger, F., Zien, A., Block, J., Gebhard, P., Aigner, T.

Cancer Letters, 226(2):143-154, August 2005 (article)

Abstract

Gene expression profiling of three chondrosarcoma derived cell lines (AD, SM, 105KC) showed an increased proliferative activity and a reduced expression of chondrocytic-typical matrix products compared to primary chondrocytes. The incapability to maintain an adequate matrix synthesis as well as a notable proliferative activity at the same time is comparable to neoplastic chondrosarcoma cells in vivo which cease largely cartilage matrix formation as soon as their proliferative activity increases. Thus, the investigated cell lines are of limited value as substitute of primary chondrocytes but might have a much higher potential to investigate the behavior of neoplastic chondrocytes, i.e. chondrosarcoma biology.

Web [BibTex]

ei Schorle, C., Finger, F., Zien, A., Block, J., Gebhard, P., Aigner, T. Phenotypic characterization of chondrosarcoma-derived cell lines Cancer Letters, 226(2):143-154, August 2005 (article)

Web [BibTex]

Local Rademacher Complexities

Bartlett, P., Bousquet, O., Mendelson, S.

The Annals of Statistics, 33(4):1497-1537, August 2005 (article)

Abstract

We propose new bounds on the error of learning algorithms in terms of a data-dependent notion of complexity. The estimates we establish give optimal rates and are based on a local and empirical version of Rademacher averages, in the sense that the Rademacher averages are computed from the data, on a subset of functions with small empirical error. We present some applications to classification and prediction with convex function classes, and with kernel classes in particular.

PDF PostScript Web [BibTex]

ei Bartlett, P., Bousquet, O., Mendelson, S. Local Rademacher Complexities The Annals of Statistics, 33(4):1497-1537, August 2005 (article)

PDF PostScript Web [BibTex]

Building Sparse Large Margin Classifiers

Wu, M., Schölkopf, B., BakIr, G.

The 22nd International Conference on Machine Learning (ICML), August 2005 (talk)

PDF [BibTex]

ei Wu, M., Schölkopf, B., BakIr, G. Building Sparse Large Margin Classifiers The 22nd International Conference on Machine Learning (ICML), August 2005 (talk)

PDF [BibTex]

Learning from Labeled and Unlabeled Data on a Directed Graph

Zhou, D.

The 22nd International Conference on Machine Learning, August 2005 (talk)

Abstract

We propose a general framework for learning from labeled and unlabeled data on a directed graph in which the structure of the graph including the directionality of the edges is considered. The time complexity of the algorithm derived from this framework is nearly linear due to recently developed numerical techniques. In the absence of labeled instances, this framework can be utilized as a spectral clustering method for directed graphs, which generalizes the spectral clustering approach for undirected graphs. We have applied our framework to real-world web classification problems and obtained encouraging results.

PDF [BibTex]

ei Zhou, D. Learning from Labeled and Unlabeled Data on a Directed Graph The 22nd International Conference on Machine Learning, August 2005 (talk)

PDF [BibTex]

Learning the Kernel with Hyperkernels

Ong, CS., Smola, A., Williamson, R.

Journal of Machine Learning Research, 6, pages: 1043-1071, July 2005 (article)

Abstract

This paper addresses the problem of choosing a kernel suitable for estimation with a Support Vector Machine, hence further automating machine learning. This goal is achieved by defining a Reproducing Kernel Hilbert Space on the space of kernels itself. Such a formulation leads to a statistical estimation problem similar to the problem of minimizing a regularized risk functional. We state the equivalent representer theorem for the choice of kernels and present a semidefinite programming formulation of the resulting optimization problem. Several recipes for constructing hyperkernels are provided, as well as the details of common machine learning problems. Experimental results for classification, regression and novelty detection on UCI data show the feasibility of our approach.

PDF [BibTex]

ei Ong, CS., Smola, A., Williamson, R. Learning the Kernel with Hyperkernels Journal of Machine Learning Research, 6, pages: 1043-1071, July 2005 (article)

PDF [BibTex]

Machine-Learning Approaches to BCI in Tübingen

Bensch, M., Bogdan, M., Hill, N., Lal, T., Rosenstiel, W., Schölkopf, B., Schröder, M.

Brain-Computer Interface Technology, June 2005, Talk given by NJH. (talk)

[BibTex]

ei Bensch, M., Bogdan, M., Hill, N., Lal, T., Rosenstiel, W., Schölkopf, B., Schröder, M. Machine-Learning Approaches to BCI in Tübingen Brain-Computer Interface Technology, June 2005, Talk given by NJH. (talk)

[BibTex]

Image Reconstruction by Linear Programming

Tsuda, K., Rätsch, G.

IEEE Transactions on Image Processing, 14(6):737-744, June 2005 (article)

Abstract

One way of image denoising is to project a noisy image to the subspace of admissible images derived, for instance, by PCA. However, a major drawback of this method is that all pixels are updated by the projection, even when only a few pixels are corrupted by noise or occlusion. We propose a new method to identify the noisy pixels by l1-norm penalization and to update the identified pixels only. The identification and updating of noisy pixels are formulated as one linear program which can be efficiently solved. In particular, one can apply the upsilon trick to directly specify the fraction of pixels to be reconstructed. Moreover, we extend the linear program to be able to exploit prior knowledge that occlusions often appear in contiguous blocks (e.g., sunglasses on faces). The basic idea is to penalize boundary points and interior points of the occluded area differently. We are also able to show the upsilon property for this extended LP leading to a method which is easy to use. Experimental results demonstrate the power of our approach.

PDF DOI [BibTex]

ei Tsuda, K., Rätsch, G. Image Reconstruction by Linear Programming IEEE Transactions on Image Processing, 14(6):737-744, June 2005 (article)

PDF DOI [BibTex]

RASE: recognition of alternatively spliced exons in C.elegans

Rätsch, G., Sonnenburg, S., Schölkopf, B.

Bioinformatics, 21(Suppl. 1):i369-i377, June 2005 (article)

PDF Web DOI [BibTex]

ei Rätsch, G., Sonnenburg, S., Schölkopf, B. RASE: recognition of alternatively spliced exons in C.elegans Bioinformatics, 21(Suppl. 1):i369-i377, June 2005 (article)

PDF Web DOI [BibTex]

Matrix Exponentiated Gradient Updates for On-line Learning and Bregman Projection

Tsuda, K., Rätsch, G., Warmuth, M.

Journal of Machine Learning Research, 6, pages: 995-1018, June 2005 (article)

Abstract

We address the problem of learning a symmetric positive definite matrix. The central issue is to design parameter updates that preserve positive definiteness. Our updates are motivated with the von Neumann divergence. Rather than treating the most general case, we focus on two key applications that exemplify our methods: on-line learning with a simple square loss, and finding a symmetric positive definite matrix subject to linear constraints. The updates generalize the exponentiated gradient (EG) update and AdaBoost, respectively: the parameter is now a symmetric positive definite matrix of trace one instead of a probability vector (which in this context is a diagonal positive definite matrix with trace one). The generalized updates use matrix logarithms and exponentials to preserve positive definiteness. Most importantly, we show how the derivation and the analyses of the original EG update and AdaBoost generalize to the non-diagonal case. We apply the resulting matrix exponentiated gradient (MEG) update and DefiniteBoost to the problem of learning a kernel matrix from distance measurements.

PDF [BibTex]

ei Tsuda, K., Rätsch, G., Warmuth, M. Matrix Exponentiated Gradient Updates for On-line Learning and Bregman Projection Journal of Machine Learning Research, 6, pages: 995-1018, June 2005 (article)

PDF [BibTex]

Texture and haptic cues in slant discrimination: Reliability-based cue weighting without statistically optimal cue combination

Rosas, P., Wagemans, J., Ernst, M., Wichmann, F.

Journal of the Optical Society of America A, 22(5):801-809, May 2005 (article)

Abstract

A number of models of depth cue combination suggest that the final depth percept results from a weighted average of independent depth estimates based on the different cues available. The weight of each cue in such an average is thought to depend on the reliability of each cue. In principle, such a depth estimation could be statistically optimal in the sense of producing the minimum variance unbiased estimator that can be constructed from the available information. Here we test such models using visual and haptic depth information. Different texture types produce differences in slant discrimination performance, providing a means for testing a reliability-sensitive cue combination model using texture as one of the cues to slant. Our results show that the weights for the cues were generally sensitive to their reliability, but fell short of statistically optimal combinationwe find reliability-based re-weighting, but not statistically optimal cue combination.

PDF Web [BibTex]

ei Rosas, P., Wagemans, J., Ernst, M., Wichmann, F. Texture and haptic cues in slant discrimination: Reliability-based cue weighting without statistically optimal cue combination Journal of the Optical Society of America A, 22(5):801-809, May 2005 (article)

PDF Web [BibTex]

Bayesian inference for psychometric functions

Kuss, M., Jäkel, F., Wichmann, F.

Journal of Vision, 5(5):478-492, May 2005 (article)

Abstract

In psychophysical studies, the psychometric function is used to model the relation between physical stimulus intensity and the observers ability to detect or discriminate between stimuli of different intensities. In this study, we propose the use of Bayesian inference to extract the information contained in experimental data to estimate the parameters of psychometric functions. Because Bayesian inference cannot be performed analytically, we describe how a Markov chain Monte Carlo method can be used to generate samples from the posterior distribution over parameters. These samples are used to estimate Bayesian confidence intervals and other characteristics of the posterior distribution. In addition, we discuss the parameterization of psychometric functions and the role of prior distributions in the analysis. The proposed approach is exemplified using artificially generated data and in a case study for real experimental data. Furthermore, we compare our approach with traditional methods based on maximum likelihood parameter estimation combined with bootstrap techniques for confidence interval estimation and find the Bayesian approach to be superior.

PDF PDF DOI [BibTex]

ei Kuss, M., Jäkel, F., Wichmann, F. Bayesian inference for psychometric functions Journal of Vision, 5(5):478-492, May 2005 (article)

PDF PDF DOI [BibTex]

A gene expression map of Arabidopsis thaliana development

Schmid, M., Davison, T., Henz, S., Pape, U., Demar, M., Vingron, M., Schölkopf, B., Weigel, D., Lohmann, J.

Nature Genetics, 37(5):501-506, April 2005 (article)

Abstract

Regulatory regions of plant genes tend to be more compact than those of animal genes, but the complement of transcription factors encoded in plant genomes is as large or larger than that found in those of animals. Plants therefore provide an opportunity to study how transcriptional programs control multicellular development. We analyzed global gene expression during development of the reference plant Arabidopsis thaliana in samples covering many stages, from embryogenesis to senescence, and diverse organs. Here, we provide a first analysis of this data set, which is part of the AtGenExpress expression atlas. We observed that the expression levels of transcription factor genes and signal transduction components are similar to those of metabolic genes. Examining the expression patterns of large gene families, we found that they are often more similar than would be expected by chance, indicating that many gene families have been co-opted for specific developmental processes.

PDF DOI [BibTex]

ei Schmid, M., Davison, T., Henz, S., Pape, U., Demar, M., Vingron, M., Schölkopf, B., Weigel, D., Lohmann, J. A gene expression map of Arabidopsis thaliana development Nature Genetics, 37(5):501-506, April 2005 (article)

PDF DOI [BibTex]

Experimentally optimal v in support vector regression for different noise models and parameter settings

Chalimourda, A., Schölkopf, B., Smola, A.

Neural Networks, 18(2):205-205, March 2005 (article)

PDF DOI [BibTex]

ei Chalimourda, A., Schölkopf, B., Smola, A. Experimentally optimal v in support vector regression for different noise models and parameter settings Neural Networks, 18(2):205-205, March 2005 (article)

PDF DOI [BibTex]

Kernel Constrained Covariance for Dependence Measurement

Gretton, A., Smola, A., Bousquet, O., Herbrich, R., Belitski, A., Augath, M., Murayama, Y., Schölkopf, B., Logothetis, N.

AISTATS, January 2005 (talk)

Abstract

We discuss reproducing kernel Hilbert space (RKHS)-based measures of statistical dependence, with emphasis on constrained covariance (COCO), a novel criterion to test dependence of random variables. We show that COCO is a test for independence if and only if the associated RKHSs are universal. That said, no independence test exists that can distinguish dependent and independent random variables in all circumstances. Dependent random variables can result in a COCO which is arbitrarily close to zero when the source densities are highly non-smooth. All current kernel-based independence tests share this behaviour. We demonstrate exponential convergence between the population and empirical COCO. Finally, we use COCO as a measure of joint neural activity between voxels in MRI recordings of the macaque monkey, and compare the results to the mutual information and the correlation. We also show the effect of removing breathing artefacts from the MRI recording.

PostScript [BibTex]

ei Gretton, A., Smola, A., Bousquet, O., Herbrich, R., Belitski, A., Augath, M., Murayama, Y., Schölkopf, B., Logothetis, N. Kernel Constrained Covariance for Dependence Measurement AISTATS, January 2005 (talk)

PostScript [BibTex]

Composite adaptive control with locally weighted statistical learning

Nakanishi, J., Farrell, J. A., Schaal, S.

Neural Networks, 18(1):71-90, January 2005, clmc (article)

Abstract

This paper introduces a provably stable learning adaptive control framework with statistical learning. The proposed algorithm employs nonlinear function approximation with automatic growth of the learning network according to the nonlinearities and the working domain of the control system. The unknown function in the dynamical system is approximated by piecewise linear models using a nonparametric regression technique. Local models are allocated as necessary and their parameters are optimized on-line. Inspired by composite adaptive control methods, the proposed learning adaptive control algorithm uses both the tracking error and the estimation error to update the parameters. We first discuss statistical learning of nonlinear functions, and motivate our choice of the locally weighted learning framework. Second, we begin with a class of first order SISO systems for theoretical development of our learning adaptive control framework, and present a stability proof including a parameter projection method that is needed to avoid potential singularities during adaptation. Then, we generalize our adaptive controller to higher order SISO systems, and discuss further extension to MIMO problems. Finally, we evaluate our theoretical control framework in numerical simulations to illustrate the effectiveness of the proposed learning adaptive controller for rapid convergence and high accuracy of control.

link (url) [BibTex]

am Nakanishi, J., Farrell, J. A., Schaal, S. Composite adaptive control with locally weighted statistical learning Neural Networks, 18(1):71-90, January 2005, clmc (article)

link (url) [BibTex]

Invariance of Neighborhood Relation under Input Space to Feature Space Mapping

Shin, H., Cho, S.

Pattern Recognition Letters, 26(6):707-718, 2005 (article)

Abstract

If the training pattern set is large, it takes a large memory and a long time to train support vector machine (SVM). Recently, we proposed neighborhood property based pattern selection algorithm (NPPS) which selects only the patterns that are likely to be near the decision boundary ahead of SVM training [Proc. of the 7th Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD), Lecture Notes in Artificial Intelligence (LNAI 2637), Seoul, Korea, pp. 376387]. NPPS tries to identify those patterns that are likely to become support vectors in feature space. Preliminary reports show its effectiveness: SVM training time was reduced by two orders of magnitude with almost no loss in accuracy for various datasets. It has to be noted, however, that decision boundary of SVM and support vectors are all defined in feature space while NPPS described above operates in input space. If neighborhood relation in input space is not preserved in feature space, NPPS may not always be effective. In this paper, we sh ow that the neighborhood relation is invariant under input to feature space mapping. The result assures that the patterns selected by NPPS in input space are likely to be located near decision boundary in feature space.

PDF PDF [BibTex]

ei Shin, H., Cho, S. Invariance of Neighborhood Relation under Input Space to Feature Space Mapping Pattern Recognition Letters, 26(6):707-718, 2005 (article)

PDF PDF [BibTex]

Theory of Classification: A Survey of Some Recent Advances

Boucheron, S., Bousquet, O., Lugosi, G.

ESAIM: Probability and Statistics, 9, pages: 323 , 2005 (article)

Abstract

The last few years have witnessed important new developments in the theory and practice of pattern classification. We intend to survey some of the main new ideas that have lead to these important recent developments.

PDF DOI [BibTex]

ei Boucheron, S., Bousquet, O., Lugosi, G. Theory of Classification: A Survey of Some Recent Advances ESAIM: Probability and Statistics, 9, pages: 323 , 2005 (article)

PDF DOI [BibTex]

Support Vector Machines and Kernel Algorithms

Schölkopf, B., Smola, A.

In Encyclopedia of Biostatistics (2nd edition), Vol. 8, 8, pages: 5328-5335, (Editors: P Armitage and T Colton), John Wiley & Sons, NY USA, 2005 (inbook)

[BibTex]

ei Schölkopf, B., Smola, A. Support Vector Machines and Kernel Algorithms In Encyclopedia of Biostatistics (2nd edition), Vol. 8, 8, pages: 5328-5335, (Editors: P Armitage and T Colton), John Wiley & Sons, NY USA, 2005 (inbook)

[BibTex]

Moment Inequalities for Functions of Independent Random Variables

Boucheron, S., Bousquet, O., Lugosi, G., Massart, P.

To appear in Annals of Probability, 33, pages: 514-560, 2005 (article)

Abstract

A general method for obtaining moment inequalities for functions of independent random variables is presented. It is a generalization of the entropy method which has been used to derive concentration inequalities for such functions cite{BoLuMa01}, and is based on a generalized tensorization inequality due to Lata{l}a and Oleszkiewicz cite{LaOl00}. The new inequalities prove to be a versatile tool in a wide range of applications. We illustrate the power of the method by showing how it can be used to effortlessly re-derive classical inequalities including Rosenthal and Kahane-Khinchine-type inequalities for sums of independent random variables, moment inequalities for suprema of empirical processes, and moment inequalities for Rademacher chaos and $U$-statistics. Some of these corollaries are apparently new. In particular, we generalize Talagrands exponential inequality for Rademacher chaos of order two to any order. We also discuss applications for other complex functions of independent random variables, such as suprema of boolean polynomials which include, as special cases, subgraph counting problems in random graphs.

PDF [BibTex]

ei Boucheron, S., Bousquet, O., Lugosi, G., Massart, P. Moment Inequalities for Functions of Independent Random Variables To appear in Annals of Probability, 33, pages: 514-560, 2005 (article)

PDF [BibTex]

Visual perception I: Basic principles

Wagemans, J., Wichmann, F., de Beeck, H.

In Handbook of Cognition, pages: 3-47, (Editors: Lamberts, K. , R. Goldstone), Sage, London, 2005 (inbook)

[BibTex]

ei Wagemans, J., Wichmann, F., de Beeck, H. Visual perception I: Basic principles In Handbook of Cognition, pages: 3-47, (Editors: Lamberts, K. , R. Goldstone), Sage, London, 2005 (inbook)

[BibTex]

A tutorial on v-support vector machines

Chen, P., Lin, C., Schölkopf, B.

Applied Stochastic Models in Business and Industry, 21(2):111-136, 2005 (article)

Abstract

We briefly describe the main ideas of statistical learning theory, support vector machines (SVMs), and kernel feature spaces. We place particular emphasis on a description of the so-called -SVM, including details of the algorithm and its implementation, theoretical results, and practical applications. Copyright © 2005 John Wiley & Sons, Ltd.

PDF [BibTex]

ei Chen, P., Lin, C., Schölkopf, B. A tutorial on v-support vector machines Applied Stochastic Models in Business and Industry, 21(2):111-136, 2005 (article)

PDF [BibTex]

Robust EEG Channel Selection Across Subjects for Brain Computer Interfaces

Schröder, M., Lal, T., Hinterberger, T., Bogdan, M., Hill, J., Birbaumer, N., Rosenstiel, W., Schölkopf, B.

EURASIP Journal on Applied Signal Processing, 2005(19, Special Issue: Trends in Brain Computer Interfaces):3103-3112, (Editors: Vesin, J. M., T. Ebrahimi), 2005 (article)

Abstract

Most EEG-based Brain Computer Interface (BCI) paradigms come along with specific electrode positions, e.g.~for a visual based BCI electrode positions close to the primary visual cortex are used. For new BCI paradigms it is usually not known where task relevant activity can be measured from the scalp. For individual subjects Lal et.~al showed that recording positions can be found without the use of prior knowledge about the paradigm used. However it remains unclear to what extend their method of Recursive Channel Elimination (RCE) can be generalized across subjects. In this paper we transfer channel rankings from a group of subjects to a new subject. For motor imagery tasks the results are promising, although cross-subject channel selection does not quite achieve the performance of channel selection on data of single subjects. Although the RCE method was not provided with prior knowledge about the mental task, channels that are well known to be important (from a physiological point of view) were consistently selected whereas task-irrelevant channels were reliably disregarded.

Web DOI [BibTex]

ei Schröder, M., Lal, T., Hinterberger, T., Bogdan, M., Hill, J., Birbaumer, N., Rosenstiel, W., Schölkopf, B. Robust EEG Channel Selection Across Subjects for Brain Computer Interfaces EURASIP Journal on Applied Signal Processing, 2005(19, Special Issue: Trends in Brain Computer Interfaces):3103-3112, (Editors: Vesin, J. M., T. Ebrahimi), 2005 (article)

Web DOI [BibTex]

Nonlinear optical spectroscopy of chiral molecules

Fischer, P., Hache, F.

CHIRALITY, 17(8):421-437, 2005 (article)

Abstract

We review nonlinear optical processes that are specific to chiral molecules in solution and on surfaces. In contrast to conventional natural optical activity phenomena, which depend linearly on the electric field strength of the optical field, we discuss how optical processes that are nonlinear (quadratic, cubic, and quartic) functions of the electromagnetic field strength may probe optically active centers and chiral vibrations. We show that nonlinear techniques open entirely new ways of exploring chirality in chemical and biological systems: The cubic processes give rise to nonlinear circular dichroism and nonlinear optical rotation and make it possible to observe dynamic chiral processes at ultrafast time scales. The quadratic second-harmonic and sum-frequency-generation phenomena and the quartic processes may arise entirely in the electric-dipole approximation and do not require the use of circularly polarized light to detect chirality: They provide surface selectivity and their observables can be relatively much larger than in linear optical activity. These processes also give rise to the generation of light at a new color, and in liquids this frequency conversion only occurs if the solution is optically active. We survey recent chiral nonlinear optical experiments and give examples of their application to problems of biophysical interest. (C) 2005 Wiley-Liss, Inc.

DOI [BibTex]

pf Fischer, P., Hache, F. Nonlinear optical spectroscopy of chiral molecules CHIRALITY, 17(8):421-437, 2005 (article)

DOI [BibTex]

Negative refraction at optical frequencies in nonmagnetic two-component molecular media

Chen, Y., Fischer, P., Wise, F.

PHYSICAL REVIEW LETTERS, 95(6), 2005 (article)

Abstract

There is significant motivation to develop media with negative refractive indices at optical frequencies, but efforts in this direction are hampered by the weakness of the magnetic response at such frequencies. We show theoretically that a nonmagnetic medium with two atomic or molecular constituents can exhibit a negative refractive index. A negative index is possible even when the real parts of both the permittivity and permeability are positive. This surprising result provides a route to isotropic negative-index media at optical frequencies.

DOI [BibTex]

pf Chen, Y., Fischer, P., Wise, F. Negative refraction at optical frequencies in nonmagnetic two-component molecular media PHYSICAL REVIEW LETTERS, 95(6), 2005 (article)

DOI [BibTex]

A model of smooth pursuit based on learning of the target dynamics using only retinal signals

Shibata, T., Tabata, H., Schaal, S., Kawato, M.

Neural Networks, 18, pages: 213-225, 2005, clmc (article)

Abstract

While the predictive nature of the primate smooth pursuit system has been evident through several behavioural and neurophysiological experiments, few models have attempted to explain these results comprehensively. The model we propose in this paper in line with previous models employing optimal control theory; however, we hypothesize two new issues: (1) the medical superior temporal (MST) area in the cerebral cortex implements a recurrent neural network (RNN) in order to predict the current or future target velocity, and (2) a forward model of the target motion is acquired by on-line learning. We use stimulation studies to demonstrate how our new model supports these hypotheses.

link (url) [BibTex]

am Shibata, T., Tabata, H., Schaal, S., Kawato, M. A model of smooth pursuit based on learning of the target dynamics using only retinal signals Neural Networks, 18, pages: 213-225, 2005, clmc (article)

link (url) [BibTex]

Parametric and Non-Parametric approaches for nonlinear tracking of moving objects

Hidaka, Y, Theodorou, E.

Technical Report-2005-1, 2005, clmc (article)

PDF [BibTex]

am Hidaka, Y, Theodorou, E. Parametric and Non-Parametric approaches for nonlinear tracking of moving objects Technical Report-2005-1, 2005, clmc (article)

PDF [BibTex]

2002

Constructing Boosting algorithms from SVMs: an application to one-class classification.

Rätsch, G., Mika, S., Schölkopf, B., Müller, K.

IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(9):1184-1199, September 2002 (article)

Abstract

We show via an equivalence of mathematical programs that a support vector (SV) algorithm can be translated into an equivalent boosting-like algorithm and vice versa. We exemplify this translation procedure for a new algorithmone-class leveragingstarting from the one-class support vector machine (1-SVM). This is a first step toward unsupervised learning in a boosting framework. Building on so-called barrier methods known from the theory of constrained optimization, it returns a function, written as a convex combination of base hypotheses, that characterizes whether a given test point is likely to have been generated from the distribution underlying the training data. Simulations on one-class classification problems demonstrate the usefulness of our approach.

DOI [BibTex]

2002

ei Rätsch, G., Mika, S., Schölkopf, B., Müller, K. Constructing Boosting algorithms from SVMs: an application to one-class classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(9):1184-1199, September 2002 (article)

DOI [BibTex]

Forward models in visuomotor control

Mehta, B., Schaal, S.

J Neurophysiol, 88(2):942-53, August 2002, clmc (article)

Abstract

In recent years, an increasing number of research projects investigated whether the central nervous system employs internal models in motor control. While inverse models in the control loop can be identified more readily in both motor behavior and the firing of single neurons, providing direct evidence for the existence of forward models is more complicated. In this paper, we will discuss such an identification of forward models in the context of the visuomotor control of an unstable dynamic system, the balancing of a pole on a finger. Pole balancing imposes stringent constraints on the biological controller, as it needs to cope with the large delays of visual information processing while keeping the pole at an unstable equilibrium. We hypothesize various model-based and non-model-based control schemes of how visuomotor control can be accomplished in this task, including Smith Predictors, predictors with Kalman filters, tapped-delay line control, and delay-uncompensated control. Behavioral experiments with human participants allow exclusion of most of the hypothesized control schemes. In the end, our data support the existence of a forward model in the sensory preprocessing loop of control. As an important part of our research, we will provide a discussion of when and how forward models can be identified and also the possible pitfalls in the search for forward models in control.

link (url) [BibTex]

am Mehta, B., Schaal, S. Forward models in visuomotor control J Neurophysiol, 88(2):942-53, August 2002, clmc (article)

link (url) [BibTex]

The contributions of color to recognition memory for natural scenes

Wichmann, F., Sharpe, L., Gegenfurtner, K.

Journal of Experimental Psychology: Learning, Memory and Cognition, 28(3):509-520, May 2002 (article)

Abstract

The authors used a recognition memory paradigm to assess the influence of color information on visual memory for images of natural scenes. Subjects performed 5-10% better for colored than for black-and-white images independent of exposure duration. Experiment 2 indicated little influence of contrast once the images were suprathreshold, and Experiment 3 revealed that performance worsened when images were presented in color and tested in black and white, or vice versa, leading to the conclusion that the surface property color is part of the memory representation. Experiments 4 and 5 exclude the possibility that the superior recognition memory for colored images results solely from attentional factors or saliency. Finally, the recognition memory advantage disappears for falsely colored images of natural scenes: The improvement in recognition memory depends on the color congruence of presented images with learned knowledge about the color gamut found within natural scenes. The results can be accounted for within a multiple memory systems framework.

PDF Web DOI [BibTex]

ei Wichmann, F., Sharpe, L., Gegenfurtner, K. The contributions of color to recognition memory for natural scenes Journal of Experimental Psychology: Learning, Memory and Cognition, 28(3):509-520, May 2002 (article)

PDF Web DOI [BibTex]

Training invariant support vector machines

DeCoste, D., Schölkopf, B.

Machine Learning, 46(1-3):161-190, January 2002 (article)

Abstract

Practical experience has shown that in order to obtain the best possible performance, prior knowledge about invariances of a classification problem at hand ought to be incorporated into the training procedure. We describe and review all known methods for doing so in support vector machines, provide experimental results, and discuss their respective merits. One of the significant new results reported in this work is our recent achievement of the lowest reported test error on the well-known MNIST digit recognition benchmark task, with SVM training times that are also significantly faster than previous SVM methods.

PDF DOI [BibTex]

ei DeCoste, D., Schölkopf, B. Training invariant support vector machines Machine Learning, 46(1-3):161-190, January 2002 (article)

PDF DOI [BibTex]

Contrast discrimination with sinusoidal gratings of different spatial frequency

Bird, C., Henning, G., Wichmann, F.

Journal of the Optical Society of America A, 19(7), pages: 1267-1273, 2002 (article)

Abstract

The detectability of contrast increments was measured as a function of the contrast of a masking or pedestal grating at a number of different spatial frequencies ranging from 2 to 16 cycles per degree of visual angle. The pedestal grating always had the same orientation, spatial frequency and phase as the signal. The shape of the contrast increment threshold versus pedestal contrast (TvC) functions depend of the performance level used to define the threshold, but when both axes are normalized by the contrast corresponding to 75% correct detection at each frequency, the (TvC) functions at a given performance level are identical. Confidence intervals on the slope of the rising part of the TvC functions are so wide that it is not possible with our data to reject Webers Law.

PDF [BibTex]

ei Bird, C., Henning, G., Wichmann, F. Contrast discrimination with sinusoidal gratings of different spatial frequency Journal of the Optical Society of America A, 19(7), pages: 1267-1273, 2002 (article)

PDF [BibTex]

Support Vector Machines and Kernel Methods: The New Generation of Learning Machines

Cristianini, N., Schölkopf, B.

AI Magazine, 23(3):31-41, 2002 (article)

[BibTex]

ei Cristianini, N., Schölkopf, B. Support Vector Machines and Kernel Methods: The New Generation of Learning Machines AI Magazine, 23(3):31-41, 2002 (article)

[BibTex]

Contrast discrimination with pulse-trains in pink noise

Henning, G., Bird, C., Wichmann, F.

Journal of the Optical Society of America A, 19(7), pages: 1259-1266, 2002 (article)

Abstract

Detection performance was measured with sinusoidal and pulse-train gratings. Although the 2.09-c/deg pulse-train, or line gratings, contained at least 8 harmonics all at equal contrast, they were no more detectable than their most detectable component. The addition of broadband pink noise designed to equalize the detectability of the components of the pulse train made the pulse train about a factor of four more detectable than any of its components. However, in contrast-discrimination experiments, with a pedestal or masking grating of the same form and phase as the signal and 15% contrast, the noise did not affect the discrimination performance of the pulse train relative to that obtained with its sinusoidal components. We discuss the implications of these observations for models of early vision in particular the implications for possible sources of internal noise.

PDF [BibTex]

ei Henning, G., Bird, C., Wichmann, F. Contrast discrimination with pulse-trains in pink noise Journal of the Optical Society of America A, 19(7), pages: 1259-1266, 2002 (article)

PDF [BibTex]

Chirality-specific nonlinear spectroscopies in isotropic media

Fischer, P., Albrecht, A.

BULLETIN OF THE CHEMICAL SOCIETY OF JAPAN, 75(5):1119-1124, 2002, 10th International Conference on Time-Resolved Vibrational Spectroscopy (TRVS 2001), OKAZZAKI, JAPAN, MAY 21-25, 2001 (article)

Abstract

Sum or difference frequency generation (SFG or DFG) in isotropic media is in the electric-dipole approximation only symmetry allowed for optically active systems. The hyperpolarizability giving rise to these three-wave mixing processes features only one isotropic component. It factorizes into two terms, an energy (denominator) factor and a triple product of transition moments. These forbid degenerate SFG, i.e., second harmonic generation, as well as the existence of the linear electrooptic effect (Pockels effect) in isotropic media. This second order response also has no static limit, which leads to particularly strong resonance phenomena that are qualitatively different from those usually seen in the ubiquitous even-wave mixing spectroscopies. In particular, the participation of two (not the usual one) excited states is essential to achieve dramatic resonance enhancement, We report our first efforts to see such resonantly enhanced chirality specific SFG.

DOI [BibTex]

pf Fischer, P., Albrecht, A. Chirality-specific nonlinear spectroscopies in isotropic media BULLETIN OF THE CHEMICAL SOCIETY OF JAPAN, 75(5):1119-1124, 2002, 10th International Conference on Time-Resolved Vibrational Spectroscopy (TRVS 2001), OKAZZAKI, JAPAN, MAY 21-25, 2001 (article)

DOI [BibTex]

The chiral specificity of sum-frequency generation in solutions

Fischer, P., Beckwitt, K., Wise, F., Albrecht, A.

CHEMICAL PHYSICS LETTERS, 352(5-6):463-468, 2002 (article)

Abstract

Sum-frequency generation in isotropic media is in the electric-dipole approximation the only symmetry allowed for chiral systems. We demonstrate that the sum-frequency intensity from an optically active liquid depends quadratically on the difference in concentration of the two enantiomers. The dominant contribution to the signal is found to be due to the chirality specific electric-dipolar three-wave mixing nonlinearity. Selecting the polarization of all fields allows the chiral electric-dipolar contributions to the bulk sum-frequency signal to be discerned from any achiral magnetic-dipolar and electric-quadrupolar contributions. (C) 2002 Published by Elsevier Science B.V.

DOI [BibTex]

pf Fischer, P., Beckwitt, K., Wise, F., Albrecht, A. The chiral specificity of sum-frequency generation in solutions CHEMICAL PHYSICS LETTERS, 352(5-6):463-468, 2002 (article)

DOI [BibTex]

On optical rectification in isotropic media

Fischer, P., Albrecht, A.

LASER PHYSICS, 12(8):1177-1181, 2002 (article)

Abstract

Coherent nonlinear optical processes at second-order are only electric-dipole allowed in isotropic media that are optically active. Sum-frequency generation in chiral liquids has recently been observed, and difference-frequency and optical rectification have been predicted to exist in isotropic chiral media. Both Rayleigh-Schrodinger perturbation theory and the density matrix approach are used to discuss the quantum-chemical basis of optical rectification in optically active liquids. For pinene we compute the corresponding orientationally averaged hyperpolarizability, and estimate the light-induced dc electric polarization and the consequent voltage across a measuring capacitor it may give rise to near resonance.

[BibTex]

pf Fischer, P., Albrecht, A. On optical rectification in isotropic media LASER PHYSICS, 12(8):1177-1181, 2002 (article)

[BibTex]

Learning robot control

Schaal, S.

In The handbook of brain theory and neural networks, 2nd Edition, pages: 983-987, 2, (Editors: Arbib, M. A.), MIT Press, Cambridge, MA, 2002, clmc (inbook)

Abstract

This is a review article on learning control in robots.

link (url) [BibTex]

am Schaal, S. Learning robot control In The handbook of brain theory and neural networks, 2nd Edition, pages: 983-987, 2, (Editors: Arbib, M. A.), MIT Press, Cambridge, MA, 2002, clmc (inbook)

link (url) [BibTex]

Arm and hand movement control

Schaal, S.

In The handbook of brain theory and neural networks, 2nd Edition, pages: 110-113, 2, (Editors: Arbib, M. A.), MIT Press, Cambridge, MA, 2002, clmc (inbook)

Abstract

This is a review article on computational and biological research on arm and hand control.

link (url) [BibTex]

am Schaal, S. Arm and hand movement control In The handbook of brain theory and neural networks, 2nd Edition, pages: 110-113, 2, (Editors: Arbib, M. A.), MIT Press, Cambridge, MA, 2002, clmc (inbook)

link (url) [BibTex]

Scalable techniques from nonparameteric statistics for real-time robot learning

Schaal, S., Atkeson, C. G., Vijayakumar, S.

Applied Intelligence, 17(1):49-60, 2002, clmc (article)

Abstract

Locally weighted learning (LWL) is a class of techniques from nonparametric statistics that provides useful representations and training algorithms for learning about complex phenomena during autonomous adaptive control of robotic systems. This paper introduces several LWL algorithms that have been tested successfully in real-time learning of complex robot tasks. We discuss two major classes of LWL, memory-based LWL and purely incremental LWL that does not need to remember any data explicitly. In contrast to the traditional belief that LWL methods cannot work well in high-dimensional spaces, we provide new algorithms that have been tested on up to 90 dimensional learning problems. The applicability of our LWL algorithms is demonstrated in various robot learning examples, including the learning of devil-sticking, pole-balancing by a humanoid robot arm, and inverse-dynamics learning for a seven and a 30 degree-of-freedom robot. In all these examples, the application of our statistical neural networks techniques allowed either faster or more accurate acquisition of motor control than classical control engineering.

link (url) [BibTex]

am Schaal, S., Atkeson, C. G., Vijayakumar, S. Scalable techniques from nonparameteric statistics for real-time robot learning Applied Intelligence, 17(1):49-60, 2002, clmc (article)

link (url) [BibTex]

MPI Papers

Departments

Research Groups

Publication Type

Year

2005

2005

2002

2002

Latest News

Links

Contact Us