Logo
PopDist and PopRun: User Guide
Version: latest
  • 1. Introduction
    • 1.1. Installation
    • 1.2. Validating the installation
    • 1.3. Replicas and instances
  • 2. Launching applications with PopRun
    • 2.1. Launch modes
    • 2.2. Multi-host setup
      • 2.2.1. SSH setup
        • Install and configure OpenSSH on all hosts
        • Create SSH key pair
        • Authorise key pair on all hosts
        • Verify SSH setup
      • 2.2.2. PopRun SSH distribution tool
      • 2.2.3. Network interfaces
      • 2.2.4. File system setup
    • 2.3. Caching compiled executables
    • 2.4. Application launches
      • 2.4.1. Single instance
      • 2.4.2. Multi instance / single host
      • 2.4.3. Multi instance / multi host
    • 2.5. Process placement and non-uniform memory access (NUMA)
    • 2.6. Offline compilation
    • 2.7. Using with PopVision Graph Analyser
      • 2.7.1. POPLAR_ENGINE_OPTIONS and PopVision Graph Analyser
      • 2.7.2. Launching without POPLAR_ENGINE_OPTIONS
      • 2.7.3. Launching with POPLAR_ENGINE_OPTIONS
    • 2.8. V-IPU settings
      • 2.8.1. V-IPU server address
      • 2.8.2. PopRun and IPU over fabric settings
      • 2.8.3. V-IPU partition, cluster, and allocation
    • 2.9. Storing and loading command line arguments
    • 2.10. Troubleshooting and solutions to common problems
      • 2.10.1. PopRun cannot find or create partition
      • 2.10.2. Program cannot acquire devices
  • 3. PopRun features
    • 3.1. Usage
    • 3.2. Tips and tricks
      • 3.2.1. Passing environment variables
      • 3.2.2. Character escaping
      • 3.2.3. File output forwarding
      • 3.2.4. Topology table
      • 3.2.5. Generating auto-completion file
  • 4. Poplar distributed configuration library (PopDist)
    • 4.1. Horovod installation
      • 4.1.1. PyTorch
      • 4.1.2. TensorFlow 2
      • 4.1.3. TensorFlow 1
      • 4.1.4. PopART
    • 4.2. PopDist collectives
      • 4.2.1. Running code on a subset of instances
    • 4.3. PopDist API
    • 4.4. PopDist examples
      • 4.4.1. PyTorch
      • 4.4.2. TensorFlow 1
      • 4.4.3. TensorFlow 2
    • 4.5. Conclusion
  • 5. PopDist C++ API reference
  • 6. PopDist Python API reference
    • 6.1. PopART
    • 6.2. PopTorch
    • 6.3. TensorFlow 1 and 2
  • 7. PopRun changelog
    • 7.1. v2.3 (Poplar SDK 2.3)
      • 7.1.1. New features
    • 7.2. v2.2 (Poplar SDK 2.2)
      • 7.2.1. New features
    • 7.3. v2.1 (Poplar SDK 2.1)
      • 7.3.1. New features
    • 7.4. v2.0 (Poplar SDK 2.0)
      • 7.4.1. New features
    • 7.5. v1.0 (Poplar SDK 1.4)
      • 7.5.1. New features
  • 8. PopDist changelog
    • 8.1. v2.6 (Poplar SDK 2.6)
    • 8.2. v2.3 (Poplar SDK 2.3)
    • 8.3. v2.2 (Poplar SDK 2.2)
      • 8.3.1. New features
    • 8.4. v2.1 (Poplar SDK 2.1)
      • 8.4.1. New features
    • 8.5. v2.0 (Poplar SDK 2.0)
      • 8.5.1. New features
    • 8.6. v1.0 (Poplar SDK 1.4)
      • 8.6.1. New features
  • 9. Known issues
    • 9.1. Race condition with multiple users of the same partition
  • 10. Index
  • 11. Trademarks & copyright
PopDist and PopRun: User Guide

Search help

Note: Searching from the top-level index page will search all documents. Searching from a specific document will search only that document.

  • Find an exact phrase: Wrap your search phrase in "" (double quotes) to only get results where the phrase is exactly matched. For example "PyTorch for the IPU" or "replicated tensor sharding"
  • Prefix query: Add an * (asterisk) at the end of any word to indicate a prefix query. This will return results containing all words with the specific prefix. For example tensor*
  • Fuzzy search: Use ~N (tilde followed by a number) at the end of any word for a fuzzy search. This will return results that are similar to the search word. N specifies the “edit distance” (fuzziness) of the match. For example Polibs~1
  • Words close to each other: ~N (tilde followed by a number) after a phrase (in quotes) returns results where the words are close to each other. N is the maximum number of positions allowed between matching words. For example "ipu version"~2
  • Logical operators. You can use the following logical operators in a search:
    • + signifies AND operation
    • | signifies OR operation
    • - negates a single word or phrase (returns results without that word or phrase)
    • () controls operator precedence

11. Trademarks & copyright

Graphcloud®, Graphcore®, Poplar® and PopVision® are registered trademarks of Graphcore Ltd.

Bow™, Bow-2000™, Bow Pod™, Colossus™, In-Processor-Memory™, IPU-Core™, IPU-Exchange™, IPU-Fabric™, IPU-Link™, IPU-M2000™, IPU-Machine™, IPU-POD™, IPU-Tile™, PopART™, PopDist™, PopLibs™, PopRun™, PopTorch™, Streaming Memory™ and Virtual-IPU™ are trademarks of Graphcore Ltd.

All other trademarks are the property of their respective owners.

This software is made available under the terms of the Graphcore End User License Agreement (EULA) and the Graphcore Container License Agreement. Please ensure you have read and accept the terms of the corresponding license before using the software. The Graphcore EULA applies unless indicated otherwise.

Copyright © 2021-2023 Graphcore Ltd. All rights reserved.

Previous

Revision 70615515.

Read the Docs v: latest