12. IPU-optimised operations¶
Several custom versions of operators are provided to target functions available in PopLibs. See the Python API for more details.
12.1. LSTM and GRU¶
These are also available as Keras layers.
The PopLibs version of dropout does not need to store the dropout mask between the forward and backward parts of the graph, saving memory.
12.3. Embedding lookup¶
This is a version of embedding lookup that has been optimised for the IPU. It allows the embedding lookup to be serialised into smaller lookups, which can reduce the maximum memory at the cost of extra computation when the embedding tensors are used by multiple operations.
12.4. Group normalisation¶
Group normalisation is an alternative to batch normalisation, and produces smaller and more optimised graphs.
The original paper on group normalisation is “Group Normalization”, Yuxin Wu, Kaiming He.
12.5. Instance normalisation¶
Instance normalisation is another alternative to batch normalisation.
The original paper on instance normalisation is “Instance Normalization: The Missing Ingredient for Fast Stylization” Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky.
12.6. Layer normalisation¶
Layer normalisation is another alternative to batch normalisation.
The original paper on layer normalisation is “Layer Normalization” Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton.
12.7. GeLU activation¶
Gaussian error linear units (GeLU) is an alternative to the ReLU non-linearity. This is described in “Gaussian Error Linear Units (GELUs)” Dan Hendrycks, Kevin Gimpel.
12.8. Sequence slice¶
The set of sequence slicing ops provided for the IPU.
The set of histogram ops provided for the IPU.