This is an extremely misleading headline.

From the abstract:

… applying the L-Mul operation in tensor processing hardware can potentially reduce 95% energy cost by element-wise floating point tensor multiplications and 80% energy cost of dot products.

In other words… This method of computation could save 95% of the energy spent on floating point multiplication (and save 80% on dot products)… Not 95% of total energy.

It’s an improvement (potentially), but I don’t see any analysis of how this would impact total energy.