Inspired by the recent upsurge of interest in Bayesian methods we consider
adaptive regularization. A generalization based scheme for adaptation of
regularization parameters is introduced and compared to Bayesian
regularization.We show that pruning arises naturally within both adaptive
regularization schemes. As model example we have chosen the simplest
possible: estimating the mean of a random variable with known variance.
Marked similarities are found between the two methods in that they both
involve a "noise limit", below which they regularize with infinite weight
decay, i.e., they prune.However, pruning is not always beneficial. We
show explicitly that both methods in some cases may increase the
generalization error. This corresponds to situations where the underlying
assumptions of the regularizer are poorly matched to the environment.