Functions used in neural networks. More...

Namespaces
namespace	pooling
	Functions for pooling operations.

namespace	rnn
	Functions for Recurrent Neural Networks (RNN)

Enumerations
enum class	NonLinearityType { SIGMOID , HARD_SIGMOID , RELU , TANH , GELU , SWISH , SOFTMAX , SOFTMAX_STABLE , SOFTMAX_SCALED }

enum class	PoolingType
	Pooling types.

Functions
void	logSoftmaxInPlace (poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext={})
	Update tensor `t` by computing log of softmax in-place. More...

poplar::Tensor	logSoftmax (poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext={})
	Compute the log of the softmax to tensor `t` and return the result. More...

std::uint64_t	getNormFwdFlops (std::size_t statisticsSize, std::size_t numActsElements, bool computeStats=true)
	Calculate the floating point operations required for the forward pass of a norm layer. More...

std::uint64_t	getNormBwdFlops (std::size_t statisticsSize, std::size_t numActsElements)
	Calculate the floating point operations required for computation of the gradient with respect to the activations for a norm layer. More...

std::uint64_t	getNormWuFlops (std::size_t paramsSize, std::size_t numActsElements)
	Calculate the floating point operations required for parameter update for a norm layer. More...

poplar::Tensor	createNormGamma (poplar::Graph &graph, const poplar::Tensor &acts, const poplar::DebugContext &debugContext={})

poplar::Tensor	createNormBeta (poplar::Graph &graph, const poplar::Tensor &acts, const poplar::DebugContext &debugContext={})

std::pair< poplar::Tensor, poplar::Tensor >	createNormParams (poplar::Graph &graph, const poplar::Tensor acts, const poplar::DebugContext &debugContext={})

std::pair< poplar::Tensor, poplar::Tensor >	spatialSoftMax2D (poplar::Graph &graph, poplar::program::Sequence &prog, const poplar::Tensor &fields, float temperature, bool disableSoftmax=false, const poplar::DebugContext &debugContext={})
	Implements a spatial softmax specialised for 2D input fields. More...

Detailed Description

Functions used in neural networks.

Enumeration Type Documentation

◆ NonLinearityType

enum class popnn::NonLinearityType

strong

Enumerator
SIGMOID	Sigmoid: y = 1 / (1 + e^(-x))
HARD_SIGMOID	Hard Sigmoid: y = max(0, min(1, 0.2*x + 0.5)
RELU	Rectified Linear Unit: x >= 0 -> y = x x < 0 -> y = 0
TANH	Hyperbolic tangent: y = tanh(x)
GELU	Gaussian Error Linear Unit: y = x * Phi(x) where Phi(x) is the cumulative distribution function of normal gaussian distribution. Phi(x) is approximated as: Phi(x) = 0.5 * (1 + (tanh(x * 0.7978845608 * (1 + 0.044715 * x * x))))
SOFTMAX	Softmax: Always applied over the innermost dimension of the given tensor. Outer dimensions are independent of one another.
SOFTMAX_STABLE	Same as SOFTMAX, but slower more numerically stable algorithm used.
SOFTMAX_SCALED	Same as SOFTMAX, but slower more numerically stable algorithm used. Outputs are scaled to allow use of greater dynamic range in outputs.

Function Documentation

◆ createNormBeta()

poplar::Tensor popnn::createNormBeta	(	poplar::Graph &	graph,
		const poplar::Tensor &	acts,
		const poplar::DebugContext &	debugContext = `{}`
	)

Parameters

graph	The graph that the normalisation operation is added to.
acts	Activations that are inputs to the norm.
debugContext	Optional debug information.

Returns: The beta values for the activations.

◆ createNormGamma()

poplar::Tensor popnn::createNormGamma	(	poplar::Graph &	graph,
		const poplar::Tensor &	acts,
		const poplar::DebugContext &	debugContext = `{}`
	)

Parameters

graph	The graph that the normalisation operation is added to.
acts	Activations that are inputs to the norm.
debugContext	Optional debug information.

Returns: The gamma values for the activations.

◆ createNormParams()

std::pair< poplar::Tensor, poplar::Tensor > popnn::createNormParams	(	poplar::Graph &	graph,
		const poplar::Tensor	acts,
		const poplar::DebugContext &	debugContext = `{}`
	)

Parameters

graph	The graph that the normalisation operation is added to.
acts	Activations that are inputs to the norm.
debugContext	Optional debug information.

Returns: A pair of tensors containing the gamma and beta values for the activations.

◆ getNormBwdFlops()

std::uint64_t popnn::getNormBwdFlops	(	std::size_t	statisticsSize,
		std::size_t	numActsElements
	)

Calculate the floating point operations required for computation of the gradient with respect to the activations for a norm layer.

Parameters

statisticsSize	The size of the statistics vector.
numActsElements	The number of elements in the activation inputs.

Returns: Number of floating point operations.

◆ getNormFwdFlops()

std::uint64_t popnn::getNormFwdFlops	(	std::size_t	statisticsSize,
		std::size_t	numActsElements,
		bool	computeStats = `true`
	)

Calculate the floating point operations required for the forward pass of a norm layer.

For inference with batchNormalise(), computeStats should be set to false if the batch statistics are not computed. This is because averaged batch statistics may be combined with the norm parameters.

Parameters

statisticsSize	The size of the statistics vector.
numActsElements	The number of elements in the activation inputs.
computeStats	Set to false for inference with batch norm.

Returns: Number of floating point operations.

◆ getNormWuFlops()

std::uint64_t popnn::getNormWuFlops	(	std::size_t	paramsSize,
		std::size_t	numActsElements
	)

Calculate the floating point operations required for parameter update for a norm layer.

Parameters

paramsSize	The size of the parameter vector.
numActsElements	The number of elements in the activation inputs.

Returns: Number of floating point operations.

◆ logSoftmax()

poplar::Tensor popnn::logSoftmax	(	poplar::Graph &	graph,
		poplar::Tensor	t,
		poplar::program::Sequence &	prog,
		const poplar::DebugContext &	debugContext = `{}`
	)

Compute the log of the softmax to tensor t and return the result.

Parameters

graph	The graph to add the operation to.
t	The tensor to apply the non-linearity to.
prog	The sequence to add the operation to.
debugContext	Optional debug information.

Returns: A new tensor containing the contents of t with the given log of the softmax applied.

◆ logSoftmaxInPlace()

void popnn::logSoftmaxInPlace	(	poplar::Graph &	graph,
		poplar::Tensor	t,
		poplar::program::Sequence &	prog,
		const poplar::DebugContext &	debugContext = `{}`
	)

Update tensor t by computing log of softmax in-place.

Parameters

graph	The graph to add the operation to.
t	The tensor to apply the log of softmax to.
prog	The sequence to add the operation to.
debugContext	Optional debug information.

◆ spatialSoftMax2D()

std::pair< poplar::Tensor, poplar::Tensor > popnn::spatialSoftMax2D	(	poplar::Graph &	graph,
		poplar::program::Sequence &	prog,
		const poplar::Tensor &	fields,
		float	temperature,
		bool	disableSoftmax = `false`,
		const poplar::DebugContext &	debugContext = `{}`
	)

Implements a spatial softmax specialised for 2D input fields.

This computes the expected coordinates (normalised to be in [-1.0, 1.0]) for every 2D field in the input tensor. A (trainable) temperature scalar is added which normalises the softmax across the fields.

The output of the spatial softmax (first tensor in the returned pair) is a set of expected x and y coordinates for the maximum activation in each field. This result has shape {F, 2} where F is the number of fields. Y-coordinates run down the first column and X-coordinates down the second column to preserve (row,column) indexing order into the original fields.

Parameters

graph	Graph to which variables and vertices will be added.
prog	Program to which operations will be added.
fields	The input Tensor. Must have rank 3. Interpretation is a set of 2D scalar fields of identical height (H) and width (W) given by the two inner dimensions (so shape is {F, H, W} where F is the number of fields).
temperature	Initial value for the softmax scaling/normalisation parameter.
debugContext	Optional debug information.
disableSoftmax	Turns off softmax computation in this function. This is useful if you have already computed a softmax over all the fields due to other processing or for test/debug.

Returns: A pair of tensors. First is the output of the spatial-softmax, second is scalar temperature variable.

Namespaces

Enumerations

Functions

Detailed Description

Enumeration Type Documentation

◆ NonLinearityType

Function Documentation

◆ createNormBeta()

◆ createNormGamma()

◆ createNormParams()

◆ getNormBwdFlops()

◆ getNormFwdFlops()

◆ getNormWuFlops()

◆ logSoftmax()

◆ logSoftmaxInPlace()

◆ spatialSoftMax2D()