Captum - Model Interpretability for PyTorch

Captum (“comprehension” in Latin) is an open source, extensible library for model interpretability built on PyTorch.

Link