High dimensional regression with many nuisance parameters: Both cases of specified and unspecified parameters of interest

Because of the presence of a large amount of noise in high dimensional data due to so many unimportant or less important variables, say nuisance variables, both estimation and inference regarding the variables of interest are difficult in high dimensional data with an overgrown number of nuisance variables. Recent approaches are only capable to handle a low dimensional vector of parameters of interest, often one or just few parameters, and not designed to restrain the estimation variance when the number of parameters is large. To obtain reliable estimation and inference in such high dimensional situations, we propose to separate the variables of interest and the nuisance variables in the regularisation/penalisation process, and apply different shrinkages for the corresponding parameters of interest and nuisance parameters using suitable penalties. We use smooth penalties for parameters of interest to shrink them gently to control their estimation variance and simultaneously use non-smooth penalties for nuisance parameters to shrink them sharply and banish all possibly small nuisance parameters. We demonstrate the differences and advantages of this approach over existing approaches. We develop our proposal further to handle the more difficult situation where no parameters of interest are specified a priori. We investigate the theoretical and empirical properties of our method and provide the R package diffShrinkHDR for its implementation.