5.16. Model debugger

PopRT provides an accuracy tracking tool (Model Debugger) to help debug accuracy issues. Model Debugger can be used to compare the accuracy of ONNX models or TensorFlow pb models, as well as to print and modify the specified ONNX models.

5.16.1. Model Debugger tool

You can download the Model Debugger tool from GitHub: model_debugger.py .

Model Debugger provides three subcommands (compare, modify and print), each with their own parameters:

  • The compare subcommand compares the accuracy between the source model and the target mode and has the following parameters:

    • --src_model: specifies the path to the source model. Use --dst_model to specify the second model for accuracy comparison.

    • --src_backend: specifies the backend used to run the source model. Options are tensorflow (for TensorFlow), onnxruntime (for ONNX) and poprt (for PopRT). Default: onnxruntime.

    • --dst_model: specifies the target model path.

    • --dst_backend: specifies the backend used to run the target model. Options are tensorflow (for TensorFlow), onnxruntime (for ONNX) and poprt (for PopRT). Default: poprt.

    • --compare_method: specifies the comparison method. Separate multiple methods with a comma (“,”). Options are: mse (mean square error), rmse (root mean squared error), mae (mean absolute error), mape (mean absolute percentage error) and r2 (R^2 coefficient of determination between two sets of results).

    • --input_data: specifies input data for each input tensor. It can be a path to the NumPy file or a character string from zeros, ones, or rand.

    • --mark_outputs: marks the name of the tensor to be compared. Separate multiple names with a comma (“,”). If "all" is specified, then all intermediate tensors will be compared.

    • --mark_io_mapping: specifies input and output mappings to compare results of the same tensor with different names in the source model and target model.

    --dump_data: dumps the tensor data to be compared to a local file.

    --dump_dir: the directory to save the dumped data to.

  • The modify subcommand modifies the input model using the specified operation and has the following parameters:

    • --src_model or --model: specifies the path to the source model.

    • --update: uses key=value to map and update the specified weight, where key is the weight name and value can be a path to a NumPy file or a character string from zeros, ones, or rand.

    • --extract: extracts subgraphs into the model. Use --extract input_names=in_name output_names=out_name to extract the subgraphs between in_name and out_name.

    • --sort: performs topological sorting on models.

    • --output_model: the path to save the modified model to.

The print subcommand prints the input model and has the following parameters:

  • --src_model or --model: specifies the path to the source model.

  • --level: sets the printing level. Options are:

    • model to print the model.

    • graph to print the graph.

    • node to print the nodes.

    • tensor to print the tensors.

  • --name: specifies the name of the tensor or node to be printed.

5.16.2. Examples

Here are some examples

Listing 5.19 Comparing the accuracy of two ONNX models
python model_debugger.py compare --src_model model1.onnx --dst_model model2.onnx --input_data input=rand --mark_outputs all
Listing 5.20 Comparing the accuracy of an ONNX model and a TensorFlow pb model
python model_debugger.py compare --src_model model.onnx --src_backend onnxruntime --dst_model model.pb --dst_backend tensorflow --input_data x:0=rand --compare_method mse,rmse,mae --dump_data --dump_dir temp_output
Listing 5.21 Extracting the subgraphs between the specified input (Conv_output_0) and output (Mul_output_0)
python model_debugger.py modify --src_model model.onnx --extract input_names=Conv_output_0 output_names=Mul_output_0 --output_model extracted.onnx
Listing 5.22 Updating the value of the specified weight
python model_debugger.py modify --src_model model.onnx --update onnx::Conv_770=zeros
Listing 5.23 Printing the specified weight
python model_debugger.py print --src_model model.onnx --name onnx::Conv_770
Listing 5.24 Printing the graph of the model
python model_debugger.py print --src_model ./debug_tools_dev/extracted.onnx --level graph