If you're looking at the cell means, you're not looking at only the interaction (unless all main effects are zero).

This reply to Meyer explains again that cell means, although usually the results of greatest interest, should not be confused with interaction effects. Unless all main effects are 0, one cannot accurately interpret an interaction by plotting the cell means. To interpret an interaction, it is the residuals remaining after removal of constituent effects (e.g., row and column effects in 2-factor analyses) that must be examined