Targeting the IPU from TensorFlow

• 1. Introduction
  • 1.1. Document overview
• 2. Tutorial
  • 2.1. Preliminary graphs
  • 2.2. A basic graph
    • 2.2.1. Selecting hardware to run on
    • 2.2.2. Running on the IPU Model simulator
  • 2.3. Compiling the graph for the IPU
  • 2.4. Sharding a graph
  • 2.5. Adding variables
    • 2.5.1. Troubleshooting
    • 2.5.2. Note on the global_step counter
• 3. Targeting the Poplar XLA device
  • 3.1. Supported types
  • 3.2. Device selection
  • 3.3. Configuring compilation options
    • 3.3.1. TF_POPLAR_FLAGS environment variable
  • 3.4. Caching of compiled executables
  • 3.5. Supported operations
  • 3.6. Unsupported operations
• 4. Training a model
  • 4.1. Training loops, data sets and feed queues
  • 4.2. Accessing outfeed queue results during execution
  • 4.3. Replicated graphs
    • 4.3.1. Selecting the number of replicas
    • 4.3.2. Data feeds
    • 4.3.3. Performing parameter updates
  • 4.4. Pipelined training
    • 4.4.1. Sequential scheduling
    • 4.4.2. Interleaved scheduling
    • 4.4.3. Grouped scheduling
    • 4.4.4. Pipeline stage inputs and outputs
    • 4.4.5. Applying an optimiser to the graph
    • 4.4.6. Device mapping
  • 4.5. Gradient accumulation
    • 4.5.1. Optimizers
    • 4.5.2. Pipelining
  • 4.6. Optimizer state offloading
  • 4.7. Dataset benchmarking
    • 4.7.1. Accessing the JSON data
• 5. Example using IPUEstimator
• 6. Example using IPUPipelineEstimator
• 7. Distributed training
  • 7.1. The input function
  • 7.2. The model function
  • 7.3. Cluster definition
  • 7.4. Complete example
  • 7.5. Distributed training with Horovod
  • 7.6. Horovod Open MPI dependency
  • 7.7. Launching Horovod training
  • 7.8. Complete Horovod example
• 8. Half-precision floating point and stochastic rounding
  • 8.1. Controlling the half-precision floating-point unit
  • 8.2. Resetting the global random number seed
  • 8.3. Debugging numerical issues
• 9. IPU-optimised operations
  • 9.1. LSTM and GRU
  • 9.2. Dropout
  • 9.3. Embedding lookup
  • 9.4. Group normalisation
  • 9.5. Instance normalisation
  • 9.6. Layer normalisation
  • 9.7. GeLU activation
• 10. IPU Functions
  • 10.1. Usage
  • 10.2. Examples
    • 10.2.1. Models with common structures
    • 10.2.2. Serializing large operations
• 11. Custom IPU operations
  • 11.1. Fully customised IPU operations
    • 11.1.1. Metadata
    • 11.1.2. In-place operations
    • 11.1.3. Elementwise operations
    • 11.1.4. Tensor allocation
    • 11.1.5. Gradient operations
    • 11.1.6. Stateless operations
    • 11.1.7. Operation attributes
    • 11.1.8. Example
  • 11.2. Fully customised CPU operations
  • 11.3. Custom elementwise expressions
  • 11.4. API Level Versioning
    • 11.4.1. Changes in API Level
• 12. IPU host embeddings
  • 12.1. Usage
  • 12.2. Example
  • 12.3. Experimental functionality: IPU embeddings in remote buffers
    • 12.3.1. Partitioning strategies
      • Token strategy
      • Encoding strategy
      • Choosing a strategy for your application
• 13. Retrieving information about compilation and execution
  • 13.1. Adding an operation to get compilation and execution events
    • 13.1.1. ipu_event_trace()
    • 13.1.2. ipu_compile_summary(name, [op list])
  • 13.2. Enabling tracing in the hardware configuration options
  • 13.3. Extract the reports from the returned events
  • 13.4. Producing reports for use with the PopVision Graph Analyser
    • 13.4.1. COMPILE_BEGIN
    • 13.4.2. COMPILE_END
      • Tensor map
    • 13.4.3. EXECUTE
  • 13.5. Using the IPU Model device for debugging
  • 13.6. TensorFlow options for reporting
  • 13.7. Reading the Poplar textual summary report
    • 13.7.1. Target
    • 13.7.2. Graph
    • 13.7.3. Memory usage
  • 13.8. Producing an ELF image of the compilation
  • 13.9. Dumping auxiliary Poplar information
    • 13.9.1. Poplar vertex graph
    • 13.9.2. Poplar interval report
  • 13.10. XLA graph file naming
• 14. Python API
  • 14.1. Operations and utilities related to the Graphcore IPU
  • 14.2. Compiler interface
  • 14.3. Scoping contexts
  • 14.4. Infeed queue
  • 14.5. Outfeed queue
  • 14.6. General utilities
  • 14.7. Looping utilities
  • 14.8. Distributed training
  • 14.9. Datasets
    • 14.9.1. Dataset benchmarking
    • 14.9.2. Dataset wrappers
  • 14.10. Estimators
    • 14.10.1. IPUEstimator
    • 14.10.2. IPUPipelineEstimator
    • 14.10.3. Run configs
    • 14.10.4. Session run hooks
  • 14.11. Keras layers
    • 14.11.1. Keras layer specializations for the Graphcore IPU
  • 14.12. Operators
    • 14.12.1. Custom operations
    • 14.12.2. Functional operators
    • 14.12.3. Graphcore utility operations
    • 14.12.4. IPU specific maths operations
    • 14.12.5. Pipelining operators
    • 14.12.6. Popnn primitive neural network operators
    • 14.12.7. Popnn normalization operators
    • 14.12.8. Popnn recurrent neural network operators
    • 14.12.9. Popops all to all and all gather operators
    • 14.12.10. Popops cross replica operators
    • 14.12.11. Popops embedding operators
    • 14.12.12. Popops reduce scatter operator
    • 14.12.13. Poprand operators
    • 14.12.14. Utility operations to be used in replicated mode
    • 14.12.15. Summary operations for IPUs
  • 14.13. Optimisers
    • 14.13.1. Optimizer wrapper for replicated graphs
    • 14.13.2. Optimizer wrappers which perform local gradient accumulation
    • 14.13.3. Optimizer wrapper that modifies gradients before application
    • 14.13.4. Optimizer wrapper for sharded graphs
  • 14.14. Sharding
    • 14.14.1. Automatic graph sharding
    • 14.14.2. Utility functions for sharding graphs
• 15. TensorFlow operators supported by the IPU
• 16. References
• 17. Index
• 18. Trademarks & copyright