Do we really have to consider covariance matrices for image features?

Many studies have been made in the past for optimization using covariance matrices of feature points. We first describe how to compute the covariance matrix of a feature point from the gray levels by integrating existing methods. Then, we experimentally examine if thus computed covariance matrices really reflect the accuracy of the feature points. To test this, we do subpixel template matching and compute the homography and the fundamental matrix. Our conclusion is rather surprising, pointing out important elements often overlooked.