12. IPU-optimised operations

Several custom versions of operators are provided to target functions available in PopLibs. See the Python API for more details.

12.1. LSTM and GRU

For recurrent neural networks, the custom LSTM (PopnnLSTM) and GRU (PopnnGRU) ops need to be used because they will use significantly less memory on the IPU.

These are also available as Keras layers.

12.2. Dropout

The PopLibs version of dropout does not need to store the dropout mask between the forward and backward parts of the graph, saving memory.

See tensorflow.python.ipu.rand_ops.dropout().

12.3. Embedding lookup

This is a version of embedding lookup that has been optimised for the IPU. It allows the embedding lookup to be serialised into smaller lookups, which can reduce the maximum memory at the cost of extra computation when the embedding tensors are used by multiple operations.

See tensorflow.python.ipu.embedding_ops.embedding_lookup().

12.4. Group normalisation

Group normalisation is an alternative to batch normalisation, and produces smaller and more optimised graphs.

The original paper on group normalisation is “Group Normalization”, Yuxin Wu, Kaiming He.

See tensorflow.python.ipu.normalization_ops.group_norm().

12.5. Instance normalisation

Instance normalisation is another alternative to batch normalisation.

The original paper on instance normalisation is “Instance Normalization: The Missing Ingredient for Fast Stylization” Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky.

See tensorflow.python.ipu.normalization_ops.instance_norm().

12.6. Layer normalisation

Layer normalisation is another alternative to batch normalisation.

The original paper on layer normalisation is “Layer Normalization” Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton.

See tensorflow.python.ipu.normalization_ops.layer_norm().

12.7. GeLU activation

Gaussian error linear units (GeLU) is an alternative to the ReLU non-linearity. This is described in “Gaussian Error Linear Units (GELUs)” Dan Hendrycks, Kevin Gimpel.

See tensorflow.python.ipu.nn_ops.gelu().