Realtime Text Recognition with VIM4 and IMX415 MIPI Camera

you mean test.py right? I have already comented

    # usb camera
    # cap = cv.VideoCapture(int(0))
    # mipi
    pipeline = "v4l2src device=/dev/media0 io-mode=dmabuf ! queue ! video/x-raw,format=YUY2,framerate=30/1 ! queue ! videoconvert ! appsink"
    cap = cv.VideoCapture(pipeline, cv.CAP_GSTREAMER)
    print(cap.isOpened())

Hello @JietChoo ,

No, i mean the line in color_detection.py. But i remember i have comented the line.

Make sure the camera is available and camera index is right. And check it has camera permission in virtual environment.

Hi Louis,

I have tried running it with

gst-launch-1.0 -v v4l2src device=/dev/media0 io-mode=mmap ! video/x-raw,format=NV12,width=3840,height=2160,framerate=30/1 ! fpsdisplaysink video-sink=waylandsink sync=false text-overlay=false

And it’s working fine

I have added a print statement in test.py

print(cv.getBuildInformation())

And got this result

General configuration for OpenCV 4.11.0 =====================================
  Version control:               4.11.0-dirty

  Platform:
    Timestamp:                   2025-01-16T09:56:27Z
    Host:                        Linux 6.8.0-51-generic aarch64
    CMake:                       3.31.1
    CMake generator:             Unix Makefiles
    CMake build tool:            /bin/gmake
    Configuration:               Release
    Algorithm Hint:              ALGO_HINT_ACCURATE

  CPU/HW features:
    Baseline:                    NEON FP16
      requested:                 DETECT
    Dispatched code generation:  NEON_DOTPROD NEON_FP16 NEON_BF16
      requested:                 NEON_FP16 NEON_BF16 NEON_DOTPROD
      NEON_DOTPROD (1 files):    + NEON_DOTPROD
      NEON_FP16 (2 files):       + NEON_FP16
      NEON_BF16 (0 files):       + NEON_BF16

  C/C++:
    Built as dynamic libs?:      NO
    C++ standard:                11
    C++ Compiler:                /opt/rh/devtoolset-10/root/usr/bin/c++  (ver 10.2.1)
    C++ flags (Release):         -Wl,-strip-all   -fsigned-char -W -Wall -Wreturn-type -Wnon-virtual-dtor -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -Wl,-strip-all   -fsigned-char -W -Wall -Wreturn-type -Wnon-virtual-dtor -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /opt/rh/devtoolset-10/root/usr/bin/cc
    C flags (Release):           -Wl,-strip-all   -fsigned-char -W -Wall -Wreturn-type -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -Wl,-strip-all   -fsigned-char -W -Wall -Wreturn-type -Waddress -Wsequence-point -Wformat -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):      -L/ffmpeg_build/lib  -Wl,--gc-sections -Wl,--as-needed -Wl,--no-undefined  
    Linker flags (Debug):        -L/ffmpeg_build/lib  -Wl,--gc-sections -Wl,--as-needed -Wl,--no-undefined  
    ccache:                      YES
    Precompiled headers:         NO
    Extra dependencies:          /lib64/libopenblas.so Qt5::Core Qt5::Gui Qt5::Widgets Qt5::Test Qt5::Concurrent /usr/local/lib/libpng.so /usr/lib64/libz.so dl m pthread rt
    3rdparty dependencies:       libprotobuf ade ittnotify libjpeg-turbo libwebp libtiff libopenjp2 IlmImf tegra_hal

  OpenCV modules:
    To be built:                 calib3d core dnn features2d flann gapi highgui imgcodecs imgproc ml objdetect photo python3 stitching video videoio
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 java python2 ts
    Applications:                -
    Documentation:               NO
    Non-free algorithms:         NO

  GUI:                           QT5
    QT:                          YES (ver 5.15.16 )
      QT OpenGL support:         NO
    GTK+:                        NO
    VTK support:                 NO

  Media I/O: 
    ZLib:                        /usr/lib64/libz.so (ver 1.2.7)
    JPEG:                        build-libjpeg-turbo (ver 3.0.3-70)
      SIMD Support Request:      YES
      SIMD Support:              YES
    WEBP:                        build (ver decoder: 0x0209, encoder: 0x020f, demux: 0x0107)
    AVIF:                        NO
    PNG:                         /usr/local/lib/libpng.so (ver 1.6.44)
    TIFF:                        build (ver 42 - 4.6.0)
    JPEG 2000:                   build (ver 2.5.0)
    OpenEXR:                     build (ver 2.3.0)
    GIF:                         NO
    HDR:                         YES
    SUNRASTER:                   YES
    PXM:                         YES
    PFM:                         YES

  Video I/O:
    FFMPEG:                      YES
      avcodec:                   YES (59.37.100)
      avformat:                  YES (59.27.100)
      avutil:                    YES (57.28.100)
      swscale:                   YES (6.7.100)
      avresample:                NO
    GStreamer:                   NO
    v4l/v4l2:                    YES (linux/videodev2.h)

  Parallel framework:            pthreads

  Trace:                         YES (with Intel ITT)

  Other third-party libraries:
    Lapack:                      YES (/lib64/libopenblas.so)
    Eigen:                       NO
    Custom HAL:                  YES (carotene (ver 0.0.1))
    Protobuf:                    build (3.19.1)
    Flatbuffers:                 builtin/3rdparty (23.5.9)

  OpenCL:                        YES (no extra features)
    Include path:                /io/opencv/3rdparty/include/opencl/1.2
    Link libraries:              Dynamic load

  Python 3:
    Interpreter:                 /opt/python/cp39-cp39/bin/python3.9 (ver 3.9.20)
    Libraries:                   libpython3.9m.a (ver 3.9.20)
    Limited API:                 YES (ver 0x03060000)
    numpy:                       /home/ci/.local/lib/python3.9/site-packages/numpy/_core/include (ver 2.0.2)
    install path:                python/cv2/python-3

  Python (for build):            /opt/python/cp39-cp39/bin/python3.9

  Java:                          
    ant:                         NO
    Java:                        NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Install to:                    /io/_skbuild/linux-aarch64-3.9/cmake-install
-----------------------------------------------------------------

I assume under the Video I/O, the Gstreamer is off. That’s why i cannot use

pipeline = "v4l2src device=/dev/media0 io-mode=dmabuf ! queue ! video/x-raw,format=YUY2,framerate=30/1 ! queue ! videoconvert ! appsink"
cap = cv.VideoCapture(pipeline, cv.CAP_GSTREAMER)

Any way i can enable it? Or should i just factory reset my VIM4?

Hello @JietChoo ,

I feedback the camera problem to our engineer.

Have you tried the picture demo? It still detect nothing?

Hello @JietChoo ,

Your assumption is right. pip install will install OpenCV precompiled packages. The precompiled packages do not compile GStreamer. But use sudo apt install, it will compile GStreamer.

Here is the Python virtual environment package from my VIM4. In this virtual environment, OpenCV compile GStreamer. Follow the command and try again.

$ mkdir myenv
$ tar -xzf myenv.tar.gz -C myenv
$ source myenv/bin/activate
$ cd ppocr_test
$ python test.py

https://dl.khadas.com/.test/myenv.tar.gz

That is the result in another VIM4 i test.

1 Like

Hi Louis, sorry for the late reply, the link https://dl.khadas.com/.test/myenv.tar.gz does not have any tar.gz file. It responded Error 404

Hello @JietChoo ,

The link available now. https://dl.khadas.com/.test/myenv.tar.gz

Now im having this error while runing

python3 test.py
['/home/khadas/ppocr_test', '/home/khadas/myenv/lib/python3.12/site-packages/cv2/python-3.12', '/usr/lib/python312.zip', '/usr/lib/python3.12', '/usr/lib/python3.12/lib-dynload', '/home/khadas/ksnn-vim4-mosen/examples/myenv/lib/python3.12/site-packages']
Traceback (most recent call last):
  File "/home/khadas/ppocr_test/test.py", line 7, in <module>
    from ksnn.api import KSNN
  File "/home/khadas/ksnn-vim4-mosen/examples/myenv/lib/python3.12/site-packages/ksnn/api.py", line 6, in <module>
    import cv2 as cv
  File "/home/khadas/ksnn-vim4-mosen/examples/myenv/lib/python3.12/site-packages/cv2/__init__.py", line 181, in <module>
    bootstrap()
  File "/home/khadas/ksnn-vim4-mosen/examples/myenv/lib/python3.12/site-packages/cv2/__init__.py", line 153, in bootstrap
    native_module = importlib.import_module("cv2")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khadas/ksnn-vim4-mosen/examples/myenv/lib/python3.12/site-packages/cv2/__init__.py", line 181, in <module>
    bootstrap()
  File "/home/khadas/ksnn-vim4-mosen/examples/myenv/lib/python3.12/site-packages/cv2/__init__.py", line 76, in bootstrap
    raise ImportError('ERROR: recursion is detected during loading of "cv2" binary extensions. Check OpenCV installation.')
ImportError: ERROR: recursion is detected during loading of "cv2" binary extensions. Check OpenCV installation.

I already performed

$ sudo apt update
$ sudo apt install python3-pip
$ pip3 install ksnn_vim4-1.4.1-py3-none-any.whl

inside the environment you sent me

Hello @JietChoo ,

The virtual environment has installed all of them. You do not need install python and ksnn again. It will change the OpenCV. Directly run test.py in the environment.

Hi Louis,

The error is the same

['/home/khadas/ppocr_test', '/home/khadas/myenv/lib/python3.12/site-packages/cv2/python-3.12', '/usr/lib/python312.zip', '/usr/lib/python3.12', '/usr/lib/python3.12/lib-dynload', '/home/khadas/ppocr_test/myenv/lib/python3.12/site-packages']
Traceback (most recent call last):
  File "/home/khadas/ppocr_test/test.py", line 7, in <module>
    from ksnn.api import KSNN
  File "/home/khadas/ppocr_test/myenv/lib/python3.12/site-packages/ksnn/api.py", line 6, in <module>
    import cv2 as cv
  File "/home/khadas/ppocr_test/myenv/lib/python3.12/site-packages/cv2/__init__.py", line 181, in <module>
    bootstrap()
  File "/home/khadas/ppocr_test/myenv/lib/python3.12/site-packages/cv2/__init__.py", line 153, in bootstrap
    native_module = importlib.import_module("cv2")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khadas/ppocr_test/myenv/lib/python3.12/site-packages/cv2/__init__.py", line 181, in <module>
    bootstrap()
  File "/home/khadas/ppocr_test/myenv/lib/python3.12/site-packages/cv2/__init__.py", line 76, in bootstrap
    raise ImportError('ERROR: recursion is detected during loading of "cv2" binary extensions. Check OpenCV installation.')
ImportError: ERROR: recursion is detected during loading of "cv2" binary extensions. Check OpenCV installation.

It says something about the cv2 module not loading properly?

I have ran my previous python scripts, it can still run with the wrong detection.

However, i used your environment and run the test.py, it has the error above, and no windows is showing.

Hello @JietChoo ,

Sorry, i reproduce the problem and find the reason that Python virtual environment has absolute path. You need to place virtual environment in this path.

Delete all myenv file on board to avoid recursive calls. And then unzip myenv.tar.gz in /home/khadas. It is the path before virtual environment packaging.

Dear Louis, i am able to run the script in the env. However, i still cannot detect any +.

From your side, is it easily able to detect the + sign?

Hello @JietChoo ,

It is easily detect.

Video download link: https://dl.khadas.com/.test/output.mp4

Are you also using emmc ubuntu24.04 gnome on Nov 2024 firmware?

If that’s the case, let us factory reset our device and reinstall this firmware version again, and run again

Hello @JietChoo ,

I use this firmware.
https://dl.khadas.com/products/vim4/firmware/ubuntu/emmc/ubuntu-24.04/vim4-ubuntu-24.04-gnome-linux-5.15-fenix-1.7.4-250423-emmc.img.xz

You can try it.