What is Image Fusion?
Image fusion is a technique that integrates two or more images from different sources into a single image. The purpose is to enhance image details, improve contrast, and combine multi-dimensional information, enabling human eyes or machine vision systems to interpret the image more clearly. It is widely used in industrial automation, smart logistics, and robotic vision.

Common image sources include:
•    Different Spectra (e.g. Visible Light + Infrared)
•    Different Sensors (e.g. RGB Color Camera + ToF Depth Camera)
•    Images with different exposure levels, focal lengths, or shooting angles


What is Color Fusion?
Color fusion is a form of image fusion that integrates information from different image sources through color mapping or coloring, presenting the combined data in colors. This allows humans or machines to clearly identify the color and structural features of objects.

Example: Color Fusion of RGB Cameras and ToF Depth Cameras
•    ToF Camera: Generates point clouds that provide 3D depth information of objects, but the images lack color and texture details.
•    RGB Camera: Provides high-resolution RGB color information and texture details but cannot capture depth information.

By combining RGB information into a point cloud, an RGB-D image (RGB-Depth Image) with true color can be generated. This allows for simultaneously obtaining the object's position, shape, and color information, greatly enhancing the understanding and interpretation efficiency of the environment.




Steps to Achieve Color Fusion Using RGB and ToF Cameras

1.    Camera Calibration
Performs intrinsic and distortion calibration for both the ToF camera and the RGB camera, and conducts extrinsic calibration to establish their relative position and orientation. Calculates the coordinate transformation matrix to ensure that the images from both cameras are precisely aligned.

2.    Pixel Mapping
Converts the ToF depth map into 3D spatial coordinates in the ToF camera coordinate system, then uses the extrinsic matrix to transform these coordinates into the RGB camera coordinate system. Applies the RGB camera's intrinsic parameters to project these 3D points onto the RGB image plane to obtain the correct pixel correspondences.

3.    Image Overlay
Combines the aligned depth information with the RGB color information to output a colored point cloud or an RGB-D image containing (X, Y, Z) and (R, G, B) data.

Traditionally, ToF cameras and RGB cameras are two separate devices, with intrinsic and extrinsic parameters that are not fixed. Therefore, every time image fusion is performed, complicated calibration and coordinate transformation are required, consuming a significant amount of manpower and time.

To address this pain point, Delta's RGB-D ToF Camera DMV-TM integrates the RGB camera and ToF camera into a single device. The intrinsic and extrinsic parameters of both cameras are factory-calibrated and fixed. Users can quickly perform color fusion through the dedicated API or the DIAVision Machine Vision Platform plug-in, greatly improving application development efficiency.


 

Benefits of Color Fusion
•    Provides More Comprehensive Information
Overcomes the limitation of a single sensor being unable to simultaneously acquire both color and depth information.

•    Improves Model Accuracy
RGB images can fill holes and edge defects in depth maps, producing more complete and smooth 3D models.

•    Enhances System Reliability
In complex and dynamic environments, a single sensor is easily affected by specific conditions, leading to degraded performance or failure. When an RGB camera is combined with a ToF camera, the ToF camera can still obtain depth information even under low-light conditions, thereby improving overall perception stability.



Applications of Color Fusion in Industrial Automation
The RGB-D images generated by color fusion not only enhance visualization and interpretability but also expand the scope and depth of applications, such as:

•    Smart Logistics
By fusing RGB and ToF images, color, shape, and size information of packages can be obtained simultaneously. During pallet unloading, the colored point cloud helps the robot identify packages that are stuck together, enabling precise positioning and gripping of the packages.

•    Industrial Inspection
By combining color images with depth maps, it can be used to inspect surface defects, structural deformations, color differences, or pattern integrity of objects.

•    Advanced Robotic Vision Perception
Enhances the AI learning capabilities of robots, enabling them to simultaneously recognize the color and 3D position of objects, and thus perform complex tasks such as picking, sorting, and assembly. In addition, RGB-D images also improve the environmental perception of robots. For example, AMRs (Autonomous Mobile Robots) can identify different colored signal lights in a factory to ensure safety and efficiency during operations.

Image fusion and color fusion are key technologies for enhancing the perception and interpretability of machine vision systems. By integrating multiple sensors such as RGB and ToF, they can adapt to changing environments and improve the reliability and efficiency of industrial automation, logistics, and robotic applications.

Delta Industrial RGB and ToF Camera Solutions