Abstract
Deep learning models have demonstrated remarkable performance in video understanding tasks, yet their internal reasoning mechanisms remain opaque — a challenge widely known as the “black box” problem. This lack of transparency raises significant concerns, particularly in high-stakes applications where there is low tolerance for mistakes, which normally result in significant consequences. Action recognition tasks, a narrow subdomain of video understanding tasks, require the model to learn from both spatial appearance and temporal motion across frames and to distinguish the human actions from the input videos. Moreover, video classification models suffer from a large number of learned parameters and high-dimensional latent-space representations that are difficult for humans to interpret, limiting trust and accountability.
To address this, we proposed the B-cosified I3D model, to our knowledge, the first adaptation of the inherently interpretable B-cos framework (Böhle et al., 2024) to the video domain. Unlike post-hoc methods, which explain the model’s behaviour after the fact, this inherently interpretable method explains the model’s reasoning mechanism through its very structure. It produces pixel-level, input-resolution explanations without any upsampling. The proposed model achieves Top-1 accuracies of 90.8% and 71.0% with and without Kinetics (Carreira and Zisserman, 2017) pre-training on the UCF-101 dataset (Soomro, Zamir and Shah, 2012), respectively, comparedtothebaselineI3Dmodel(CarreiraandZisserman,2017), whichachieves 95.6% and 84.5% with the same evaluation metrics. We argue that this performance reduction is an acceptable cost of providing reliable predictions, and that the lack of hyperparameter fine-tuning exacerbates the performance decay, as evidenced by empirical evaluations revealing a static bias in the UCF-101 dataset that causes the baseline I3D model to inflate its performance during evaluation by exploiting its bias terms.