Can You Retrofit Explainability Later?