5. TensorFlow 1

• 5.1. Introduction to TensorFlow 1 on the IPU
  • Before you start
  • Other useful resources
• 5.2. TensorFlow 1 on the IPU: training a model using half- and mixed-precision
  • Table of Contents
  • Introduction
  • Using FP16 on the IPU in practice
    • Support for FP16 in TensorFlow
    • Review of common numerical issues
    • Inaccuracies in parameter updates
      • Method 1: Using stochastic rounding
      • Method 2: Store and update the parameters in FP32
    • Underflowing gradients and loss scaling
      • Numerical concerns with loss scaling
      • Techniques for specific optimisers
    • Diagnosing numerical issues
    • Avoiding numerical issues
      • Avoiding underflow
      • Avoiding overflow
      • Unstable operations
    • Other considerations
      • Setting the partials type for convolutions and matrix multiplications
      • Data type arguments for specialised IPU ops
  • Code examples
    • Running the examples
    • Stochastic rounding example
    • FP32 parameter updates example
    • IPUEstimator example
    • Optional command line arguments
    • Other examples
• 5.3. TensorFlow 1 Pipelining
  • Introduction
  • Requirements
  • Key Principles of Model Pipelining
    • Overview
      • 1. Model Parallelism With Sharding
      • 2. Model Parallelism With Pipelining
    • Pipeline Execution Phases
  • Tutorial Walkthrough
    • Tutorial Step 1: The Existing Single IPU Application
    • Tutorial Step 2: Running The Model On Multiple IPUs Using Sharding
    • Tutorial Step 2: Code Changes
    • Tutorial Step 3: Using pipelining for better IPU utilization
      • The TensorFlow 1 Pipelining API
    • Scheduling
      • Sequential Scheduling
      • Interleaved Scheduling
      • Grouped Scheduling
    • Tutorial Step 3: Code Changes
    • Tutorial Step 3 : Extension
      • Pipeline Schedule
      • Repeat count
      • Stages
    • Tutorial Step 4: Run-time Configurable Stages
    • Tutorial Step 4: Code Changes
  • Further Considerations
    • Recomputation
    • Variable Offloading
    • Gradient Accumulation Buffer Data Type
    • Data Parallelism
    • IPUPipelineEstimator