Exploring Multimodal Sensing Across the Stack for Robot Manipulation