9. IPU-optimised operations
Several custom versions of operators are provided to target functions available in PopLibs. See the Python API for more details.
9.1. LSTM and GRU
For recurrent neural networks, the custom LSTM
(PopnnLSTM) and GRU
(PopnnGRU) ops need to be used
because they will use significantly less memory on the IPU.
These are also available as Keras layers.
9.2. Dropout
The PopLibs version of dropout does not need to store the dropout mask between the forward and backward parts of the graph, saving memory.
9.3. Embedding lookup
This is a version of embedding lookup that has been optimised for the IPU. It allows the embedding lookup to be serialised into smaller lookups, which can reduce the maximum memory at the cost of extra computation when the embedding tensors are used by multiple operations.
9.4. Group normalisation
Group normalisation is an alternative to batch normalisation, and produces smaller and more optimised graphs.
The original paper on group normalisation is “Group Normalization”, Yuxin Wu, Kaiming He.
9.5. Instance normalisation
Instance normalisation is another alternative to batch normalisation.
The original paper on instance normalisation is “Instance Normalization: The Missing Ingredient for Fast Stylization” Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky.
See tensorflow.python.ipu.normalization_ops.instance_norm().
9.6. Layer normalisation
Layer normalisation is another alternative to batch normalisation.
The original paper on layer normalisation is “Layer Normalization” Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton.
9.7. GeLU activation
Gaussian error linear units (GeLU) is an alternative to the ReLU non-linearity. This is described in “Gaussian Error Linear Units (GELUs)” Dan Hendrycks, Kevin Gimpel.