When relevant knowledge is provided as additional context to commonsense question answering, it can substantially enhance the performance even on top of state-of-the-art. Rainier is a reinforced knowledge introspector that learns to generate contextually relevant knowledge in response to given questions.