Conversion and Inferencing using a Yolov5 model and KSNN

Which Khadas SBC do you use?

Khadas VIM3

Which system do you use? Android, Ubuntu, OOWOW or others?

Ubuntu 20.04

Which version of system do you use? Khadas official images, self built images, or others?

Khadas Official

Please describe your issue below:

I am wanting to convert a standard weighted Yolov5s (current version 6.2) model so that I can run the inferencing using the A311D processor via the KSNN api on the Khadas VIM3. Below is a description of the methods that I have tried.

The first and most obvious was to convert the model from its native format (PyTorch) to the format required for KSNN using the method described both in the documentation and the website. I had downloaded the aml_npu_sdk (using recursive) to a WSL environment and used the following to attempt to convert the model:

$ ./convert \

--model-name detector --platform pytorch \

--model ./ \

--input-size-list '3,640,640' --inputs input \

--mean-values '103.94 116.78 123.68 0.01700102' \

--quantized-dtype asymmetric_affine \

--source-files ./data/dataset/dataset0.txt \

--kboard VIM3 --print-level 1

The environment has the current version of Yolov5 and its dependencies from requirements.txt installed. The conversion attempt results in the following error:

--+ KSNN Convert tools v1.3 +--

Start import model ...

2022-09-21 00:57:53.562743: W tensorflow/stream_executor/platform/default/] Could not load dynamic library ''; dlerror: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/user/aml_npu_sdk/acuity-toolkit/bin/acuitylib:/tmp/_MEImTbDtD

2022-09-21 00:57:53.562797: I tensorflow/stream_executor/cuda/] Ignore above cudart dlerror if you do not have a GPU set up on your machine.

I Namespace(config=None, import='pytorch', input_size_list='3,640,640', inputs='input', model='', output_data='', output_model='Model.json', outputs=None, size_with_batch=None, which='import')

I Start importing pytorch...

[14650] Failed to execute script pegasus

Traceback (most recent call last):

File "", line 131, in <module>

File "", line 112, in main

File "acuitylib/app/importer/", line 294, in execute

File "acuitylib/", line 242, in load_pytorch_by_onnx_backend

File "acuitylib/onnx_ir/frontend/pytorch_frontend/", line 45, in __init__

File "torch/jit/", line 228, in load

RuntimeError: [enforce fail at] . file not found: archive/constants.pkl

There is no clear reason as to why this method doesn’t work and the documentation surrounding the use of this tool is very limited.

My second attempt (suggested from the Khadas forum) was to first convert the Yolov5s model to ONNX and then convert to the format required by KSNN. I started with a clean WSL environment and installed Yolov5 and its dependencies then downloaded the aml_npu_sdk. I converted the model to ONNX format using the following from the Yolov5 documentation:

$ python --weights --include torchscript onnx

I then converted the ONNX model to the format required by KSNN using the following as per the documentation:

$ ./convert \

--model-name detector \

--platform onnx \

--model ./yolov5s.onnx \

--mean-values '123.675 116.28 103.53 0.01700102' \

--quantized-dtype asymmetric_affine \

--source-files ./data/dataset/dataset0.txt \

--kboard VIM3 --print-level 0

I get the following output from the conversion tool:

--+ KSNN Convert tools v1.3 +--

Start import model ...

Done.import model success !!!

Start to Generate inputmeta ...

Done.Gerate inputmeta success !!!

Start quantize ...

Done.Quantize success !!!

Start export model ...

Done.Export model success !!!

All Done.

This resulted in two files:

detector.nb (8,487 KB) and libnn_detector (160 KB)

Success! Or so I thought… The next step of the process was to try and perform an inference on the Khadas VIM3 using the newly converted models. Starting with a clean environment of Ubuntu 20.04 server (image from the Khadas website), I installed KSNN as described in the documentation. I first tried an ONNX inference using the example provided which results in a successful detection of a goldfish. I then try to run an inference using my model using the following (the image is a 640x640x3 jpg):

$ python3 --model ./detector.nb --library ./ --picture ./002.jpg --level 2

The following is the last few lines of the output screen:

Run segment 1113. Type: 1, operations: [1114, 1114].

Segment 1113 ended.

Run segment 1114. Type: 1, operations: [1115, 1115].

Segment 1114 ended.

Run segment 1115. Type: 1, operations: [1116, 1116].

Segment 1115 ended.

layer_id: 0 layer name:network_binary_graph operation[0]:unkown operation type target:unkown operation target.

uid: 0

abs_op_id: 0

execution time: 309797 us

[ 1] TOTAL_READ_BANDWIDTH (MByte): 113.313650

[ 2] TOTAL_WRITE_BANDWIDTH (MByte): 202.045716

[ 3] AXI_READ_BANDWIDTH (MByte): 62.035364

[ 4] AXI_WRITE_BANDWIDTH (MByte): 48.191188

[ 5] DDR_READ_BANDWIDTH (MByte): 51.278287

[ 6] DDR_WRITE_BANDWIDTH (MByte): 153.854528

[ 7] GPUTOTALCYCLES: 242247682

[ 8] GPUIDLECYCLES: 157781362



Run the 1 time: 315.00ms or 315781.00us

vxProcessGraph execution time:

Total 315.00ms or 315851.00us

Average 315.85ms or 315851.00us

Done. inference time: 0.3375585079193115 RuntimeWarning: overflow encountered in exp

return np.exp(x)/sum(np.exp(x)) RuntimeWarning: invalid value encountered in divide

return np.exp(x)/sum(np.exp(x))


-----TOP 5-----

-1: 0.0

-1: 0.0

-1: 0.0

-1: 0.0

-1: 0.0

As can be seen from the output, no predictions are displayed and there is an overflow which can be assumed as a divide by 0. This indicates that there is some sort of error in the model conversion process, however with a lack of documentation on how to use the tools, it is very difficult to try and find a solution.

Ultimately what I would like to have is a verified guide on how to convert and inference using a Yolov5 model via KSNN. Thank you in advance for your time.

@Ingeniero You can refer to the parameters of yolov3 for your conversion parameters, and the documents in the SDK are also applicable to KSNN.

I have already tried it using the conversion parameter, --mean-values for a Darknet model (0 0 0 0.00390625). I obtain the same result.

@Ingeniero Have you checked the output of the model, are they all 0?

Yes all of the outputs are 0.


I am not sure where the problem lies. If the results are all 0, there should be a problem with the conversion parameters.