5.2.1. TensorFlow known issues

2.6.0

  • The program can terminate unexpectedly when using a 0-dimensional Tensor in a tf.Dataset.

2.5.1

Warning

The versions of TensorFlow included in Poplar SDK 2.5 and earlier are not compatible with protobuf version 4 (see TensorFlow issue #56077). When you install a TensorFlow wheel from the Poplar SDK, you must ensure you have a compatible version of protobuf, downgrading if necessary.

  • For TensorFlow 2:

    $ python -m pip install "protobuf>=3.9.2,<3.20" --force-reinstall

  • For TensorFlow 1:

    $ python -m pip install "protobuf>=3.8.0,<3.20" --force-reinstall

You can do this before or after installing the Graphcore TensorFlow wheel.

  • Wrapping Keras layers in ipu.outlined_function can cause compilation errors.

  • The gradient_accumulation_reduction_method feature of Keras models can cause an increase in memory usage when the non-default option is used.

  • Using mixed_precision.Policy('mixed_float16') with pipelined Keras models results in compilation errors.

  • The experimental_normalize_gradients feature of TensorFlow 2 can produce unstable results when the number of replicas or the gradient_accumulation_steps_per_replica is large.