PyTorch for the IPU: User Guide

• 1. Introduction
  • 1.1. Data batching
  • 1.2. Parallel and Distributed execution
  • 1.3. Constraints
  • 1.4. Other resources
• 2. Installation
  • 2.1. Version compatibility
  • 2.2. Using a Python virtual environment
  • 2.3. Setting the environment variables
  • 2.4. Validating the setup
• 3. From PyTorch to PopTorch
  • 3.1. Preparing your data
  • 3.2. Creating your model
    • 3.2.1. Training
    • 3.2.2. Inference
  • 3.3. The training loop
  • 3.4. Multiple/custom losses
  • 3.5. Optimizers
  • 3.6. Going further
• 4. Features
  • 4.1. Options
    • 4.1.1. Setting options via config file
  • 4.2. Model wrapping functions
    • 4.2.1. poptorch.trainingModel
    • 4.2.2. poptorch.inferenceModel
    • 4.2.3. poptorch.PoplarExecutor
    • 4.2.4. poptorch.isRunningOnIpu
  • 4.3. Execution strategies
    • 4.3.1. Annotation tools
      • poptorch.Block, poptorch.BeginBlock and poptorch.BlockFunction
      • poptorch.Stage and poptorch.AutoStage
      • poptorch.Phase
      • Advanced annotation with strings
    • 4.3.2. Parallel execution strategies
      • poptorch.ShardedExecution
      • poptorch.PipelinedExecution
      • Phased execution
  • 4.4. Optimizers
    • 4.4.1. Loss scaling
    • 4.4.2. Velocity scaling (SGD combined variant only)
    • 4.4.3. Accumulation types
    • 4.4.4. Constant attributes
  • 4.5. PopTorch ops
    • 4.5.1. poptorch.ctc_beam_search_decoder
    • 4.5.2. poptorch.ipu_print_tensor
    • 4.5.3. poptorch.identity_loss
    • 4.5.4. poptorch.MultiConv
    • 4.5.5. poptorch.nop
    • 4.5.6. poptorch.serializedMatMul
    • 4.5.7. poptorch.set_available_memory
    • 4.5.8. Miscellaneous functions
  • 4.6. Half / float16 support
  • 4.7. Automatic mixed-precision casting
    • 4.7.1. Custom casting policies
  • 4.8. Creating custom ops
    • 4.8.1. Implementing the custom op
    • 4.8.2. Make the op available to PyTorch
    • 4.8.3. Passing attributes to the custom op
  • 4.9. Profiling
  • 4.10. Precompilation and caching
    • 4.10.1. Caching
    • 4.10.2. Precompilation
  • 4.11. Environment variables
    • 4.11.1. Logging level
    • 4.11.2. Profiling
    • 4.11.3. IPU Model
    • 4.11.4. Wait for an IPU to become available
    • 4.11.5. Enable executable caching
• 5. Efficient data batching
  • 5.1. poptorch.DataLoader
  • 5.2. poptorch.AsynchronousDataAccessor
    • 5.2.1. Rebatching iterable datasets
  • 5.3. poptorch.Options.deviceIterations
  • 5.4. poptorch.Options.replicationFactor
  • 5.5. poptorch.Options.Training.gradientAccumulation
  • 5.6. poptorch.Options.Training.anchorReturnType
• 6. IPU supported operations
  • 6.1. Torch operations
    • 6.1.1. Tensor operations
      • Creation ops
      • Indexing, slicing, joining and mutating ops
      • Random samplers
    • 6.1.2. Math operations
      • Pointwise ops
      • Reduction ops
      • Comparison ops
      • Other ops
      • BLAS and LAPACK Operations
  • 6.2. Torch.nn operations
    • 6.2.1. Containers
    • 6.2.2. Convolution layers
    • 6.2.3. Pooling layers
    • 6.2.4. Padding layers
    • 6.2.5. Activations
    • 6.2.6. Normalization layers
    • 6.2.7. Recurrent layers
    • 6.2.8. Linear layers
    • 6.2.9. Dropout
    • 6.2.10. Sparse layers
    • 6.2.11. Loss functions
    • 6.2.12. Vision Layers
  • 6.3. Float 16 operations
    • 6.3.1. Casting
    • 6.3.2. Creation functions
    • 6.3.3. Normalization
• 7. Debugging your model
  • 7.1. Inspecting tensors
  • 7.2. Anchoring tensors
  • 7.3. Retrieving tensors
• 8. Examples
  • 8.1. MNIST example
• 9. Experimental features
  • 9.1. Distributed execution without PopRun
  • 9.2. torch.nn.CTCLoss
• 10. Reference
  • 10.1. Options
  • 10.2. Helpers
  • 10.3. PopTorch Ops
  • 10.4. Model wrapping functions
  • 10.5. Parallel execution
  • 10.6. Optimizers
  • 10.7. Data batching
  • 10.8. Enumerations
  • 10.9. Autocasting
• 11. Index
• 12. Trademarks & copyright
• 13. Changelog
  • 13.1. v2.1 (Poplar SDK 2.1)
    • 13.1.1. New features
    • 13.1.2. API changes
    • 13.1.3. Known issues
  • 13.2. v2.0 (Poplar SDK 2.0)
    • 13.2.1. New features
    • 13.2.2. API changes
  • 13.3. v1.0 (Poplar SDK 1.4)
    • 13.3.1. New features
    • 13.3.2. Known issues
  • 13.4. v0.1 (Poplar SDK 1.3)
    • 13.4.1. New features