NonLinearity

#include <popnn/NonLinearity.hpp>

Non-linearity operations.

Defines

DEF_NONLINEARITY_INPLACE(fn, nlType)
DEF_NONLINEARITY_(fn, nlType)
DEF_NONLINEARITY(fn, nlType)
namespace popnn

Functions used in neural networks.

Functions

void nonLinearityInPlace(poplar::Graph &graph, NonLinearityType nonLinearityType, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

Update tensor t by applying the given non-linearity in-place.

Parameters
  • graph – The graph to add the operation to.

  • nonLinearityType – The type of non-linearity to apply to t.

  • t – The tensor to apply the non-linearity to.

  • prog – The sequence to add the operation to.

  • debugContext – Optional debug information.

void nonLinearityInPlace(poplar::Graph &graph, NonLinearityType nonLinearityType, poplar::Tensor t, poplar::ComputeSet &cs, const poplar::DebugContext &debugContext = {})

Update tensor t by applying the given non-linearity in-place.

Parameters
  • graph – The graph to add the operation to.

  • nonLinearityType – The type of non-linearity to apply to t.

  • t – The tensor to apply the non-linearity to.

  • cs – The compute set to add vertices to.

  • debugContext – Optional debug information.

void nonLinearityInPlace(poplar::Graph &graph, NonLinearityType nonLinearityType, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

Update tensor t by applying the given non-linearity in-place and return the scaling factor by which outputs from this operation are multiplied in nonLinearityScaling.

For NonLinearityType other than SOFTMAX_SCALED nonLinearityScaling will be 1.0f upon return.

Parameters
  • graph – The graph to add the operation to.

  • nonLinearityType – The type of non-linearity to apply to t.

  • t – The tensor to apply the non-linearity to.

  • nonLinearityScaling – Reference to a float which will be overwritten with the scaling factor by which outputs from this operation in t are multiplied.

  • prog – The sequence to add the operation to.

  • debugContext – Optional debug information.

void nonLinearityInPlace(poplar::Graph &graph, NonLinearityType nonLinearityType, poplar::Tensor t, float &nonLinearityScaling, poplar::ComputeSet &cs, const poplar::DebugContext &debugContext = {})

Update tensor t by applying the given non-linearity in-place and return the scaling factor by which outputs from this operation are multiplied in nonLinearityScaling.

For NonLinearityType other than SOFTMAX_SCALED nonLinearityScaling will be 1.0f upon return.

Parameters
  • graph – The graph to add the operation to.

  • nonLinearityType – The type of non-linearity to apply to t.

  • t – The tensor to apply the non-linearity to.

  • nonLinearityScaling – Reference to a float which will be overwritten with the scaling factor by which outputs from this operation in t are multiplied.

  • cs – The compute set to add vertices to.

  • debugContext – Optional debug information.

poplar::Tensor nonLinearity(poplar::Graph &graph, NonLinearityType nonLinearityType, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

Apply the given non-linearity to tensor t and return the result.

Parameters
  • graph – The graph to add the operation to.

  • nonLinearityType – The type of non-linearity to apply.

  • t – The tensor to apply the non-linearity to.

  • prog – The sequence to add the operation to.

  • debugContext – Optional debug information.

Returns

A new tensor containing the contents of t with the given non-linearity applied.

poplar::Tensor nonLinearity(poplar::Graph &graph, NonLinearityType nonLinearityType, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

Apply the given non-linearity to tensor t and return the result.

Also returns the scaling factor by which outputs from this operation are multiplied in nonLinearityScaling.

For NonLinearityType other than SOFTMAX_SCALED nonLinearityScaling will be 1.0f upon return.

Parameters
  • graph – The graph to add the operation to.

  • nonLinearityType – The type of non-linearity to apply to t.

  • t – The tensor to apply the non-linearity to.

  • nonLinearityScaling – Reference to a float which will be overwritten with the scaling factor by which outputs from this operation in t are multiplied.

  • prog – The sequence to add the operation to.

  • debugContext – Optional debug information.

Returns

A new tensor containing the contents of t with the given non-linearity applied.

inline void sigmoidInPlace(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline void sigmoidInPlace(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline poplar::Tensor sigmoid(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline poplar::Tensor sigmoid(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline void reluInPlace(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline void reluInPlace(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline poplar::Tensor relu(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline poplar::Tensor relu(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline void tanhInPlace(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline void tanhInPlace(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline poplar::Tensor tanh(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline poplar::Tensor tanh(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline void geluInPlace(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline void geluInPlace(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline poplar::Tensor gelu(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline poplar::Tensor gelu(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline void swishInPlace(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline void swishInPlace(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline poplar::Tensor swish(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline poplar::Tensor swish(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline void softmaxInPlace(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline void softmaxInPlace(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline poplar::Tensor softmax(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline poplar::Tensor softmax(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline void softmaxStableInPlace(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline void softmaxStableInPlace(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline poplar::Tensor softmaxStable(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline poplar::Tensor softmaxStable(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline void scaledSoftmaxStableInPlace(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline void scaledSoftmaxStableInPlace(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline poplar::Tensor scaledSoftmaxStable(poplar::Graph &graph, poplar::Tensor t, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
inline poplar::Tensor scaledSoftmaxStable(poplar::Graph &graph, poplar::Tensor t, float &nonLinearityScaling, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})
poplar::Tensor nonLinearityInputGradient(poplar::Graph &graph, NonLinearityType nonLinearityType, poplar::Tensor act, poplar::Tensor outGradient, poplar::ComputeSet &cs, const poplar::DebugContext &debugContext = {})

Computes and returns the input gradient for a non-linearity from the activations and gradients at the output of the non-linearity.

Parameters
  • graph – The graph to add the operation to.

  • nonLinearityType – The type of non-linearity to compute the input gradient for.

  • act – The output activations from the non-linearity. For the GELU non-linearity only this is the input to the non-linearity.

  • outGradient – The gradients at the output of the non-linearity.

  • cs – The compute set to add vertices to.

  • debugContext – Optional debug information.

Returns

A new tensor with the calculated gradient for the input of the non-linearity.

poplar::Tensor nonLinearityInputGradient(poplar::Graph &graph, NonLinearityType nonLinearityType, poplar::Tensor act, poplar::Tensor outGradient, poplar::program::Sequence &prog, const poplar::DebugContext &debugContext = {})

Computes and returns the input gradient for a non-linearity from the activations and gradients at the output of the non-linearity.

Parameters
  • graph – The graph to add the operation to.

  • nonLinearityType – The type of non-linearity to compute the input gradient for.

  • act – The output activations from the non-linearity. For the GELU and SWISH non-linearity only this is the input to the non-linearity.

  • outGradient – The gradients at the output of the non-linearity.

  • prog – The sequence to add the operation to.

  • debugContext – Optional debug information.

Returns

A new tensor with the calculated gradient for the input of the non-linearity.