Norms

#include <popnn/Norms.hpp>

Normalisation operations.

namespace popnn

Functions used in neural networks.

Functions

std::uint64_t getNormFwdFlops(std::size_t statisticsSize, std::size_t numActsElements, bool computeStats = true)

Calculate the floating point operations required for the forward pass of a norm layer.

For inference with batchNormalise(), computeStats should be set to false if the batch statistics are not computed. This is because averaged batch statistics may be combined with the norm parameters.

Parameters

  • statisticsSize – The size of the statistics vector.

  • numActsElements – The number of elements in the activation inputs.

  • computeStats – Set to false for inference with batch norm.

Returns

Number of floating point operations.

std::uint64_t getNormBwdFlops(std::size_t statisticsSize, std::size_t numActsElements)

Calculate the floating point operations required for computation of the gradient with respect to the activations for a norm layer.

Parameters

  • statisticsSize – The size of the statistics vector.

  • numActsElements – The number of elements in the activation inputs.

Returns

Number of floating point operations.

std::uint64_t getNormWuFlops(std::size_t paramsSize, std::size_t numActsElements)

Calculate the floating point operations required for parameter update for a norm layer.

Parameters

  • paramsSize – The size of the parameter vector.

  • numActsElements – The number of elements in the activation inputs.

Returns

Number of floating point operations.

poplar::Tensor createNormGamma(poplar::Graph &graph, const poplar::Tensor &acts, const poplar::DebugContext &debugContext = {})
Parameters

  • graph – The graph that the normalisation operation is added to.

  • acts – Activations that are inputs to the norm.

  • debugContext – Optional debug information.

Returns

The gamma values for the activations.

poplar::Tensor createNormBeta(poplar::Graph &graph, const poplar::Tensor &acts, const poplar::DebugContext &debugContext = {})
Parameters

  • graph – The graph that the normalisation operation is added to.

  • acts – Activations that are inputs to the norm.

  • debugContext – Optional debug information.

Returns

The beta values for the activations.

std::pair<poplar::Tensor, poplar::Tensor> createNormParams(poplar::Graph &graph, const poplar::Tensor acts, const poplar::DebugContext &debugContext = {})
Parameters

  • graph – The graph that the normalisation operation is added to.

  • acts – Activations that are inputs to the norm.

  • debugContext – Optional debug information.

Returns

A pair of tensors containing the gamma and beta values for the activations.