NonLinearity

#include <popnn/NonLinearity.hpp>

Non-linearity operations.

Defines

DEF_NONLINEARITY_INPLACE(fn, nlType)

DEF_NONLINEARITY_(fn, nlType)

DEF_NONLINEARITY(fn, nlType)

namespace popnn

Functions used in neural networks.

Functions

void nonLinearityInPlace(poplar::Graph &graph, NonLinearityType nonLinearityType, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

Update tensor t by applying the given non-linearity in-place.

Parameters

• graph – The graph to add the operation to.

• nonLinearityType – The type of non-linearity to apply to t.

• t – The tensor to apply the non-linearity to.

• prog – The sequence to add the operation to.

• debugContext – Optional debug information.

void nonLinearityInPlace(poplar::Graph &graph, NonLinearityType nonLinearityType, poplar::Tensor t, poplar::ComputeSet &cs, const poplar::DebugContext &debugContext = {})

Update tensor t by applying the given non-linearity in-place.

Parameters

• graph – The graph to add the operation to.

• nonLinearityType – The type of non-linearity to apply to t.

• t – The tensor to apply the non-linearity to.

• cs – The compute set to add vertices to.

• debugContext – Optional debug information.

void nonLinearityInPlace(poplar::Graph &graph, NonLinearityType nonLinearityType, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

Update tensor t by applying the given non-linearity in-place and return the scaling factor by which outputs from this operation are multiplied in nonLinearityScaling.

For NonLinearityType other than SOFTMAX_SCALED nonLinearityScaling will be 1.0f upon return.

Parameters

• graph – The graph to add the operation to.

• nonLinearityType – The type of non-linearity to apply to t.

• t – The tensor to apply the non-linearity to.

• nonLinearityScaling – Reference to a float which will be overwritten with the scaling factor by which outputs from this operation in t are multiplied.

• prog – The sequence to add the operation to.

• debugContext – Optional debug information.

void nonLinearityInPlace(poplar::Graph &graph, NonLinearityType nonLinearityType, poplar::Tensor t, float &nonLinearityScaling, poplar::ComputeSet &cs, const poplar::DebugContext &debugContext = {})

Update tensor t by applying the given non-linearity in-place and return the scaling factor by which outputs from this operation are multiplied in nonLinearityScaling.

For NonLinearityType other than SOFTMAX_SCALED nonLinearityScaling will be 1.0f upon return.

Parameters

• graph – The graph to add the operation to.

• nonLinearityType – The type of non-linearity to apply to t.

• t – The tensor to apply the non-linearity to.

• nonLinearityScaling – Reference to a float which will be overwritten with the scaling factor by which outputs from this operation in t are multiplied.

• cs – The compute set to add vertices to.

• debugContext – Optional debug information.

poplar::Tensor nonLinearity(poplar::Graph &graph, NonLinearityType nonLinearityType, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

Apply the given non-linearity to tensor t and return the result.

Parameters

• graph – The graph to add the operation to.

• nonLinearityType – The type of non-linearity to apply.

• t – The tensor to apply the non-linearity to.

• prog – The sequence to add the operation to.

• debugContext – Optional debug information.

Returns

A new tensor containing the contents of t with the given non-linearity applied.

poplar::Tensor nonLinearity(poplar::Graph &graph, NonLinearityType nonLinearityType, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

Apply the given non-linearity to tensor t and return the result.

Also returns the scaling factor by which outputs from this operation are multiplied in nonLinearityScaling.

For NonLinearityType other than SOFTMAX_SCALED nonLinearityScaling will be 1.0f upon return.

Parameters

• graph – The graph to add the operation to.

• nonLinearityType – The type of non-linearity to apply to t.

• t – The tensor to apply the non-linearity to.

• nonLinearityScaling – Reference to a float which will be overwritten with the scaling factor by which outputs from this operation in t are multiplied.

• prog – The sequence to add the operation to.

• debugContext – Optional debug information.

Returns

A new tensor containing the contents of t with the given non-linearity applied.

inline void sigmoidInPlace(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline void sigmoidInPlace(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline poplar::Tensor sigmoid(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline poplar::Tensor sigmoid(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline void reluInPlace(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline void reluInPlace(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline poplar::Tensor relu(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline poplar::Tensor relu(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline void tanhInPlace(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline void tanhInPlace(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline poplar::Tensor tanh(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline poplar::Tensor tanh(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline void geluInPlace(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline void geluInPlace(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline poplar::Tensor gelu(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline poplar::Tensor gelu(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline void swishInPlace(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline void swishInPlace(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline poplar::Tensor swish(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline poplar::Tensor swish(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline void softmaxInPlace(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline void softmaxInPlace(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline poplar::Tensor softmax(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline poplar::Tensor softmax(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline void softmaxStableInPlace(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline void softmaxStableInPlace(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline poplar::Tensor softmaxStable(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline poplar::Tensor softmaxStable(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline void scaledSoftmaxStableInPlace(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline void scaledSoftmaxStableInPlace(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline poplar::Tensor scaledSoftmaxStable(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

inline poplar::Tensor scaledSoftmaxStable(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

poplar::Tensor nonLinearityInputGradient(poplar::Graph &graph, NonLinearityType nonLinearityType, poplar::Tensor act, poplar::Tensor outGradient, poplar::ComputeSet &cs, const poplar::DebugContext &debugContext = {})

Computes and returns the input gradient for a non-linearity from the activations and gradients at the output of the non-linearity.

Parameters

• graph – The graph to add the operation to.

• nonLinearityType – The type of non-linearity to compute the input gradient for.

• act – The output activations from the non-linearity. For the GELU non-linearity only this is the input to the non-linearity.

• outGradient – The gradients at the output of the non-linearity.

• cs – The compute set to add vertices to.

• debugContext – Optional debug information.

Returns

A new tensor with the calculated gradient for the input of the non-linearity.

poplar::Tensor nonLinearityInputGradient(poplar::Graph &graph, NonLinearityType nonLinearityType, poplar::Tensor act, poplar::Tensor outGradient, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

Computes and returns the input gradient for a non-linearity from the activations and gradients at the output of the non-linearity.

Parameters

• graph – The graph to add the operation to.

• nonLinearityType – The type of non-linearity to compute the input gradient for.

• act – The output activations from the non-linearity. For the GELU and SWISH non-linearity only this is the input to the non-linearity.

• outGradient – The gradients at the output of the non-linearity.

• prog – The sequence to add the operation to.

• debugContext – Optional debug information.

Returns

A new tensor with the calculated gradient for the input of the non-linearity.