Tool to monitor NPU usage

Edge2 Ubuntu
1

Which system do you use? Android, Ubuntu, OOWOW or others?

Ubuntu

Which version of system do you use? Khadas official images, self built images, or others?

Khadas image

Please describe your issue below:

I was wondering if there was an equivalent to nvidia-smi that produces an output like this:

$ nvidia-smi
Thu Sep  7 17:39:08 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.39.01    Driver Version: 510.39.01    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA TITAN RTX    On   | 00000000:01:00.0 Off |                  N/A |
| 41%   29C    P8     7W / 280W |      1MiB / 24576MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

for the edge 2?

2

@brunobiped If you mean GPU, there is no such tool to view such parameters, however you can take a look into the sysfs of the GPU:

$ cd /sys/devices/platform/fb000000.gpu/devfreq/fb000000.gpu
$ ls
available_frequencies  cur_freq  governor  max_freq  name              power      target_freq  trans_stat
available_governors    device    load      min_freq  polling_interval  subsystem  timer        uevent

here:

available_frequencies: possible gpu frequency operation points.
available_governors: possible device power mangement governors
cur_freq: current gpu operating frequency
governor: current gpu power managing governer.
load: gpu load

regards.

3

Hello @Electr1,

thanks for the answer. I’ve tried running examples from here: https://github.com/rockchip-linux/rknpu2/tree/master/examples while monitoring the changes of the file like this:
watch -n 0.1 load.

Unfortunately, there seemed to be no changes. Any idea what I could do wrong?

Best,

Bruno

4

@brunobiped the example was for the GPU, not the NPU.

NPU does not have any monitoring/profiling interface

However, if you want to profile the execution of your model when inferencing, you can do such with the
Python rknnlite.api library, when creating the RKNN object, you need to enable the verbose as

from rknnlite.api import RKNNLite

rknn_lite = RKNNLite(verbose=True)

Here is an example of the kind of output you can expect:
Model: Mobilenet_v1.rknn

D RKNN: [11:22:44.286] ID   OpType           DataType Target InputShape                                   OutputShape            DDR Cycles     NPU Cycles     Total Cycles   Time(us)       MacUsage(%)    WorkLoad(0/1/2)-ImproveTherical        Task Number    Lut Number     RW(KB)         FullName        
D RKNN: [11:22:44.286] 1    InputOperator    UINT8    CPU    \                                            (1,3,224,224)          0              0              0              3              \              0.0%/0.0%/0.0% - Up:0.0%               0              0              147.00         InputOperator:input
D RKNN: [11:22:44.286] 2    Conv             UINT8    NPU    (1,3,224,224),(32,3,3,3),(32)                (1,32,112,112)         31861          10584          31861          217            4.88           100.0%/0.0%/0.0% - Up:0.0%             3              0              543.62         Conv:MobilenetV1/MobilenetV1/Conv2d_0/Relu6_prequant
D RKNN: [11:22:44.286] 3    Conv             INT8     NPU    (1,32,112,112),(1,32,3,3),(32)               (1,32,112,112)         45980          3528           45980          250            1.41           100.0%/0.0%/0.0% - Up:0.0%             2              0              784.53         Conv:MobilenetV1/MobilenetV1/Conv2d_1_depthwise/Relu6_prequant
D RKNN: [11:22:44.286] 4    Conv             INT8     NPU    (1,32,112,112),(64,32,1,1),(64)              (1,64,112,112)         69055          25088          69055          272            9.22           100.0%/0.0%/0.0% - Up:0.0%             2              0              1178.25        Conv:MobilenetV1/MobilenetV1/Conv2d_1_pointwise/Relu6_prequant
D RKNN: [11:22:44.286] 5    Conv             INT8     NPU    (1,64,112,112),(1,64,3,3),(64)               (1,64,56,56)           57484          1764           57484          189            0.93           100.0%/0.0%/0.0% - Up:0.0%             3              0              980.81         Conv:MobilenetV1/MobilenetV1/Conv2d_2_depthwise/Relu6_prequant
D RKNN: [11:22:44.286] 6    Conv             INT8     NPU    (1,64,56,56),(128,64,1,1),(128)              (1,128,56,56)          34960          25088          34960          163            15.39          100.0%/0.0%/0.0% - Up:0.0%             1              0              596.50         Conv:MobilenetV1/MobilenetV1/Conv2d_2_pointwise/Relu6_prequant
D RKNN: [11:22:44.286] 7    Conv             INT8     NPU    (1,128,56,56),(1,128,3,3),(128)              (1,128,56,56)          46044          3528           46044          182            1.94           100.0%/0.0%/0.0% - Up:0.0%             2              0              785.62         Conv:MobilenetV1/MobilenetV1/Conv2d_3_depthwise/Relu6_prequant
D RKNN: [11:22:44.286] 8    Conv             INT8     NPU    (1,128,56,56),(128,128,1,1),(128)            (1,128,56,56)          46916          50176          50176          173            29.00          100.0%/0.0%/0.0% - Up:0.0%             2              0              800.50         Conv:MobilenetV1/MobilenetV1/Conv2d_3_pointwise/Relu6_prequant
D RKNN: [11:22:44.286] 9    Conv             INT8     NPU    (1,128,56,56),(1,128,3,3),(128)              (1,128,28,28)          28814          882            28814          135            0.65           100.0%/0.0%/0.0% - Up:0.0%             2              0              491.62         Conv:MobilenetV1/MobilenetV1/Conv2d_4_depthwise/Relu6_prequant
D RKNN: [11:22:44.286] 10   Conv             INT8     NPU    (1,128,28,28),(256,128,1,1),(256)            (1,256,28,28)          19165          25088          25088          116            21.63          100.0%/0.0%/0.0% - Up:0.0%             1              0              327.00         Conv:MobilenetV1/MobilenetV1/Conv2d_4_pointwise/Relu6_prequant
D RKNN: [11:22:44.286] 11   Conv             INT8     NPU    (1,256,28,28),(1,256,3,3),(256)              (1,256,28,28)          23165          1764           23165          123            1.43           100.0%/0.0%/0.0% - Up:0.0%             1              0              395.25         Conv:MobilenetV1/MobilenetV1/Conv2d_5_depthwise/Relu6_prequant
D RKNN: [11:22:44.286] 12   Conv             INT8     NPU    (1,256,28,28),(256,256,1,1),(256)            (1,256,28,28)          26784          50176          50176          139            36.10          100.0%/0.0%/0.0% - Up:0.0%             1              0              457.00         Conv:MobilenetV1/MobilenetV1/Conv2d_5_pointwise/Relu6_prequant
D RKNN: [11:22:44.286] 13   Conv             INT8     NPU    (1,256,28,28),(1,256,3,3),(256)              (1,256,14,14)          14550          441            14550          99             0.45           100.0%/0.0%/0.0% - Up:0.0%             1              0              248.25         Conv:MobilenetV1/MobilenetV1/Conv2d_6_depthwise/Relu6_prequant
D RKNN: [11:22:44.286] 14   Conv             INT8     NPU    (1,256,14,14),(512,256,1,1),(512)            (1,512,14,14)          16235          25088          25088          80             31.36          100.0%/0.0%/0.0% - Up:0.0%             1              0              277.00         Conv:MobilenetV1/MobilenetV1/Conv2d_6_pointwise/Relu6_prequant
D RKNN: [11:22:44.286] 15   Conv             INT8     NPU    (1,512,14,14),(1,512,3,3),(512)              (1,512,14,14)          11869          882            11869          47             1.88           100.0%/0.0%/0.0% - Up:0.0%             1              0              202.50         Conv:MobilenetV1/MobilenetV1/Conv2d_7_depthwise/Relu6_prequant
D RKNN: [11:22:44.286] 16   Conv             INT8     NPU    (1,512,14,14),(512,512,1,1),(512)            (1,512,14,14)          26609          50176          50176          83             60.45          100.0%/0.0%/0.0% - Up:0.0%             1              0              454.00         Conv:MobilenetV1/MobilenetV1/Conv2d_7_pointwise/Relu6_prequant
D RKNN: [11:22:44.286] 17   Conv             INT8     NPU    (1,512,14,14),(1,512,3,3),(512)              (1,512,14,14)          11869          882            11869          46             1.92           100.0%/0.0%/0.0% - Up:0.0%             1              0              202.50         Conv:MobilenetV1/MobilenetV1/Conv2d_8_depthwise/Relu6_prequant
D RKNN: [11:22:44.286] 18   Conv             INT8     NPU    (1,512,14,14),(512,512,1,1),(512)            (1,512,14,14)          26609          50176          50176          83             60.45          100.0%/0.0%/0.0% - Up:0.0%             1              0              454.00         Conv:MobilenetV1/MobilenetV1/Conv2d_8_pointwise/Relu6_prequant
D RKNN: [11:22:44.286] 19   Conv             INT8     NPU    (1,512,14,14),(1,512,3,3),(512)              (1,512,14,14)          11869          882            11869          46             1.92           100.0%/0.0%/0.0% - Up:0.0%             1              0              202.50         Conv:MobilenetV1/MobilenetV1/Conv2d_9_depthwise/Relu6_prequant
D RKNN: [11:22:44.286] 20   Conv             INT8     NPU    (1,512,14,14),(512,512,1,1),(512)            (1,512,14,14)          26609          50176          50176          83             60.45          100.0%/0.0%/0.0% - Up:0.0%             1              0              454.00         Conv:MobilenetV1/MobilenetV1/Conv2d_9_pointwise/Relu6_prequant
D RKNN: [11:22:44.286] 21   Conv             INT8     NPU    (1,512,14,14),(1,512,3,3),(512)              (1,512,14,14)          11869          882            11869          46             1.92           100.0%/0.0%/0.0% - Up:0.0%             1              0              202.50         Conv:MobilenetV1/MobilenetV1/Conv2d_10_depthwise/Relu6_prequant
D RKNN: [11:22:44.286] 22   Conv             INT8     NPU    (1,512,14,14),(512,512,1,1),(512)            (1,512,14,14)          26609          50176          50176          83             60.45          100.0%/0.0%/0.0% - Up:0.0%             1              0              454.00         Conv:MobilenetV1/MobilenetV1/Conv2d_10_pointwise/Relu6_prequant
D RKNN: [11:22:44.286] 23   Conv             INT8     NPU    (1,512,14,14),(1,512,3,3),(512)              (1,512,14,14)          11869          882            11869          104            0.85           100.0%/0.0%/0.0% - Up:0.0%             1              0              202.50         Conv:MobilenetV1/MobilenetV1/Conv2d_11_depthwise/Relu6_prequant
D RKNN: [11:22:44.286] 24   Conv             INT8     NPU    (1,512,14,14),(512,512,1,1),(512)            (1,512,14,14)          26609          50176          50176          84             59.73          100.0%/0.0%/0.0% - Up:0.0%             1              0              454.00         Conv:MobilenetV1/MobilenetV1/Conv2d_11_pointwise/Relu6_prequant
D RKNN: [11:22:44.286] 25   Conv             INT8     NPU    (1,512,14,14),(1,512,3,3),(512)              (1,512,7,7)            7561           220            7561           39             0.57           100.0%/0.0%/0.0% - Up:0.0%             1              0              129.00         Conv:MobilenetV1/MobilenetV1/Conv2d_12_depthwise/Relu6_prequant
D RKNN: [11:22:44.286] 26   Conv             INT8     NPU    (1,512,7,7),(1024,512,1,1),(1024)            (1,1024,7,7)           34550          25088          34550          73             34.37          100.0%/0.0%/0.0% - Up:0.0%             1              0              589.50         Conv:MobilenetV1/MobilenetV1/Conv2d_12_pointwise/Relu6_prequant
D RKNN: [11:22:44.286] 27   Conv             INT8     NPU    (1,1024,7,7),(1,1024,3,3),(1024)             (1,1024,7,7)           6506           441            6506           61             0.72           100.0%/0.0%/0.0% - Up:0.0%             1              0              111.00         Conv:MobilenetV1/MobilenetV1/Conv2d_13_depthwise/Relu6_prequant
D RKNN: [11:22:44.286] 28   Conv             INT8     NPU    (1,1024,7,7),(1024,1024,1,1),(1024)          (1,1024,7,7)           65993          50176          65993          169            29.69          100.0%/0.0%/0.0% - Up:0.0%             1              0              1126.00        Conv:MobilenetV1/MobilenetV1/Conv2d_13_pointwise/Relu6_prequant
D RKNN: [11:22:44.286] 29   Conv             INT8     NPU    (1,1024,7,7),(1,1024,7,7),(1024)             (1,1024,1,1)           6272           49             6272           60             0.08           100.0%/0.0%/0.0% - Up:0.0%             1              0              107.00         Conv:MobilenetV1/Logits/AvgPool_1a/AvgPool
D RKNN: [11:22:44.286] 30   Conv             INT8     NPU    (1,1024,1,1),(1001,1024,1,1),(1001)          (1,1001,1,1)           59018          1001           59018          112            0.89           100.0%/0.0%/0.0% - Up:0.0%             1              0              1006.98        Conv:MobilenetV1/Logits/Conv2d_1c_1x1/BiasAdd_prequant
D RKNN: [11:22:44.286] 31   exSoftmax13      INT8     CPU    (1,1001,1,1)                                 (1,1001,1,1)           0              0              0              19             \              0.0%/0.0%/0.0% - Up:0.0%               0              0              1.96           exSoftmax13:MobilenetV1/Predictions/Reshape_1_prequant
D RKNN: [11:22:44.286] 32   Reshape          INT8     CPU    (1,1001,1,1),(2)                             (1,1001)               0              0              0              7              \              0.0%/0.0%/0.0% - Up:0.0%               0              0              1.97           Reshape:MobilenetV1/Logits/SpatialSqueeze_prequant
D RKNN: [11:22:44.286] 33   OutputOperator   INT8     CPU    (1,1001)                                     \                      0              0              0              1              \              0.0%/0.0%/0.0% - Up:0.0%               0              0              0.98           OutputOperator:MobilenetV1/Predictions/Reshape_1
D RKNN: [11:22:44.286] Total Operator Elapsed Time(us): 3387
D RKNN: [11:22:44.286] Total Memory RW Amount(MB): 0

This is an example with my own model and code,

To try something like this by taking an example from here: NPU Applications [Khadas Docs]

Just modify the source code and profile the NPU execution.

Regards.