Removal, white balance (AWB) correction, color correction (color correction), gamma correction, color space conversion (RGB to YUV), color noise removal and edge enhancement in the YUV color space, color and contrast enhancement, and in the middle of the automatic exposure control, etc., and then output YUV (or RGB) format data, and then transferred to the CPU for processing through the I/O interface. The data is then output in YUV (or RGB) format and transferred to the CPU for processing through the I/O interface.
conceptual
ISP is short for Image Signal Processor.
DSP stands for Digital Signal Processor.
ISP is generally used to process the output data of Image Sensor, such as AEC (Auto Exposure Control), AGC (Auto Gain Control), AWB (Auto White Balance), Color Correction, Lens Shading, Gamma Correction, Bad Dot Removal, Auto Black Level, Auto White Level, etc. BlackLevel, Auto White Level, and more.
DSP function is more, it can do some photo and display back (JPEG codec), video recording and playback (Video codec), H.264 codec, and many other aspects of the processing, in short, the processing of digital signals.ISP is a special class of DSP to deal with image signals.
ISP architecture solutions: divided into two forms: stand-alone (external) and integrated (internal).
CPU processors include: AP, BP, CP. where BP: baseband processor, AP: application processor, CP: multimedia gas pedal.
Main internal components of an ISP
As shown in the figure below, the ISP contains CPU, SUP IP, IF and other devices inside, in fact, it can be considered that the ISP is a SOC (system of chip), which can run various algorithmic programs to process image signals in real time.
CPU: CPU is the central processing unit, which can run various image processing algorithms such as AF, LSC, etc. and control peripheral devices. The CPU inside the modern ISP is usually of ARM Cortex-A series, such as Cortex-A5, Cortex-A7.CPU: CPU is the central processing unit, which can run various image processing algorithms such as AF, LSC, etc. and control peripheral devices. The CPU inside the modern ISP is usually of ARM Cortex-A series, such as Cortex-A5, Cortex-A7.
SUB IP: SUB IP is a generic term for a variety of functional modules that perform their own specialized processing of images. Common SUB IPs include DIS, CSC, VRA, etc.
Image Transfer Interface: There are two main types of image transfer interfaces, parallel ITU and serial CSI. CSI is the abbreviation of MIPI CSI, in view of the many advantages of MIPI CSI, in the field of cell phone cameras, the MIPI-CSI interface has been widely used for transferring image data and all kinds of customized data. An external ISP generally consists of two interfaces, MIPI-CSIS and MIPI-CSIM. Built-in ISPs generally require only the MIPI-CSIS interface.
Common Peripherals: Common peripherals refer to I2C, SPI, PWM, UART, WATCHDOG, etc. The ISP contains an I2C controller, which is used to read OTP information and control VCM, etc. For external ISP, the ISP itself is also an I2C slave device. For external ISP, the ISP itself is also an I2C slave device, and the AP can control the working mode of the ISP and get its working status through I2C.
ISP control structure
ISP includes: 1, ISP logic 2, running on its firmware
As shown in the figure, the lens will project the light signal to the sensor’s light-sensitive area, the sensor after photoelectric conversion, the Bayer format of the original image to the ISP, the ISP after algorithmic processing, the output RGB spatial domain of the image to the back-end video capture unit. In this process, the ISP runs the firmware on the ISP logic to control the lens and the sensor accordingly, thus accomplishing the functions of auto iris, auto exposure, auto white balance and so on. The firmware is driven by the interrupt of the video capture unit, and the PQ Tools tool is used to adjust the online image quality of the ISP through the network port or serial port.
ISP consists of ISP logic and Firmware running on it, the logic unit can not only complete part of the algorithm processing, but also can statistic the real-time information of the current image, Firmware by obtaining the image statistics of ISP logic, recalculate, feedback control lens, sensor and ISP logic, in order to achieve the purpose of automatically adjusting the quality of the image.
The Firmware on the ISP contains three parts
The ISP firmware consists of three parts, one is the ISP control unit and the basic algorithm library, one is the AE/AWB/AF algorithm library, and one is the sensor library. the basic idea of the firmware design is to provide the 3A algorithm library separately, and the ISP control unit schedules the basic algorithm library and the 3A algorithm library, and the sensor library registers function callbacks to the ISP basic algorithm library and the 3A algorithm library to realize the differentiated sensor adaptation. The basic idea of the firmware design is to provide a separate 3A algorithm library, and the ISP control unit schedules the basic algorithm library and 3A algorithm library, while the sensor library registers function callbacks to the ISP basic algorithm library and 3A algorithm library respectively to realize differentiated sensor adaptation.
Different sensors register control functions with the ISP algorithm library in the form of callback functions. when the ISP control unit schedules the basic algorithm library and the 3A algorithm library, it will obtain the initialization parameters through these callback functions and control the sensors, such as adjusting the exposure time, analog gain, digital gain, and controlling the lens to focus or rotate the aperture in steps, and so on.
The way the AP manipulates the ISP
CPU processors include: AP, BP, CP. BP: Baseband Processor, AP: Application Processor, CP: Multimedia Accelerator
The control method described here is the way the AP manipulates the ISP .
I2C/SPI: This is generally the practice of external ISP, SPI is generally used to download firmware, I2C is generally used for register control. In the kernel’s ISP driver, the external ISP is usually implemented as an I2C device and then packaged as V4L2-SUBDEV.
MEM MAP: This is usually done by the built-in ISP. Maps the internal register address space of the ISP to the kernel address space.
MEM SHARE: This is also the practice of the built-in ISP, where the AP allocates the memory and then passes the memory address to the ISP, both of them actually share the same memory. Therefore, AP’s operation on this shared memory will be fed back to ISP in real time.
ISP Architecture Program
The external ISP and internal ISP are mentioned several times above, which are actually ISP architectural solutions.
External ISP Architecture
External ISP architecture refers to the arrangement of a separate ISP chip outside the AP for image signal processing. The architecture of external ISP is generally shown below:
The main advantages of the external ISP architecture are:
Able to provide better image quality: Under the fierce competition in the market, external ISP manufacturers that have survived until now generally have deep attainments in this field and have accumulated rich experience in debugging image quality, and are able to provide better performance and results than the built-in ISP. Therefore, choosing a high-quality external ISP can provide professional and excellent image quality.
Ability to support richer design planning: The selection of external ISPs is basically independent of APs, so Meizu can select the most suitable devices from a wide range of products from various excellent ISP chip suppliers, thus designing more excellent products.
Product differentiation: The built-in ISP is encapsulated inside the AP and is closely linked to the AP, so if the AP is the same, then the ISP is also the same. If the AP is the same, then the ISP is also the same. Therefore, the performance of the ISP is the same for phones based on the same AP, and the conditions for tuning are also fixed, which is not conducive to product differentiation. If you choose external ISP, then the same AP can be paired with different types of ISPs, which can achieve product differentiation and provide users with richer and better quality products.
The main disadvantages of the external ISP architecture are:
High cost: External ISPs need to be purchased separately, are often expensive, and require additional fees for some special features. The use of external ISPs requires additional schematic design and LAYOUT, and the use of additional components.
Long development cycle: The design of external ISP driver requires more effort and time. When using an external ISP, the ISP driver provided by the AP supplier cannot be used, so it is necessary to design and write an additional external ISP driver. In addition, in order to perfectly match with the AP to maximize the effect, it often requires more debugging efforts. As mentioned above, the use of external ISP requires additional schematic design and LAYOUT, and the use of additional components, which also takes time to process.
Built-in ISP architecture:
Built-in ISP architecture means that the ISP IP is embedded inside the AP and the image signal processing is performed directly using the ISP inside the AP. The architecture of the built-in ISP is generally shown below:
The main advantages of the built-in ISP architecture are:
It can reduce the cost: the built-in ISP is embedded in the AP, so it does not need to be purchased additionally like external ISP, and it does not take up PCB space, so there is no need to design peripheral circuits for it separately, which saves BOM and reduces the cost. Given that most users will put the price factor in an important position when purchasing a cell phone, cost reduction can effectively lower the price of the finished product, which is conducive to capturing the market.
Enables faster time-to-market: The built-in ISP and AP are tightly integrated, eliminating the need for schematic design and LAYOUT design, thus reducing development cycles and speeding time-to-market.
Reduces development difficulty: If you use built-in ISPs, then the AP provider can provide relevant information in the early stage, and the driver developer can have enough time to familiarize with the relevant information, and there will be no software version adaptation problems, and there is no compatibility problem with the platform architecture. However, if external ISPs are used, ISP vendors are often unable to provide platform-specific code/materials, and there are usually software version compatibility issues, which requires the driver developer to spend a lot of experience and time.
The use of the built-in ISP has of course corresponding shortcomings, which are analyzed above and will not be repeated here.
In fact, given the importance of ISPs, in order to promote their APs and increase their AP competitiveness, the ISPs built into APs are now becoming more and more powerful, with performance sufficient to meet the needs of the cell phone market. Together with a series of advantages, more and more cell phones are now using built-in ISP solutions.
ISP Processing Flow
Image generation process: The optical image generated by the scene through the Lens is projected onto the sensor surface, converted to analog electrical signals through photoelectricity, de-noised and converted to digital image signals through A/D conversion, and then sent to the Digital Signal Processing chip (DSP) for processing.
Therefore, the image from the sensor side is a Bayer image, after black level compensation, lens shading correction, bad pixel correction, color interpolation, Bayer noise removal, white balance (awb) correction, color correction, gamma correction, color space conversion (RGB to YUV), color noise in the YUV color space. ), Bayer noise removal, white balance (awb) correction, color correction, gamma correction, color space conversion (RGB to YUV), color noise removal and edge enhancement, color and contrast enhancement in the YUV color space, with automatic exposure control in the middle, etc., and then output YUV (or RGB) format data. The data is then output in YUV (or RGB) format and transferred to the CPU for processing through the I/O interface.
1. Bayer (Bayer filters get colored)
When converting the actual scene into image data, it is common for the sensor to receive the red, green, and blue components separately, and then to combine the red, green, and blue components into a color image. This solution requires three filters, which is expensive and difficult to manufacture because all three filters must be aligned at every pixel.
By adding a color filter structure and a color information processing module on top of the black and white cmos image sensor, color information of the image can be obtained, and then the color information can be processed to obtain a color image with realistic colors. Usually the color image sensor surface covered by the filter is called color filter arrays (Color Filter Arrays, CFA).
Currently the most commonly used filter array is the checkerboard format, there have been many kinds of, in which the vast majority of camera products are used in the original color of the Bell template color filter array (Bayer Pattern CFA).R, G, B, respectively, the filter array unit through the red, through the green and through the blue. Since human vision is most sensitive to green, the G component in Bayer CFA is twice as much as R and B. Only one color component can be obtained at each pixel, and then the full-color image is obtained by interpolation algorithm based on the information of this color component.
2. BLC (Black level Correction)
a. Dark current
Physical devices are not ideal, and due to impurities, heat, and other causes, even if no light hits the pixel, the pixel cell generates an electrical charge, which creates a dark current. Moreover, it is difficult to distinguish between dark current and light generated charge. Public No.: OpenFPGA
b.Black Level
Black Level is used to define the signal level when the image data is 0. Due to the effect of dark current, the actual raw data from the sensor is not the black balance we need (the data is not zero). Therefore, to minimize the effect of dark current on the image signal, an effective method can be used to subtract the reference dark current signal from the acquired image signal. Generally, there are more actual pixels than effective pixels in the sensor, as shown in the figure below, the first few rows of the pixel area are used as the non-sensitive area (in fact, this part of the area is also made RGB color filter) for automatic black level correction, and its average value is used as the correction value, and then all the pixels in the following area are subtracted from the corrected value, then the black level can be corrected.
Comparison between black level correction and no black level correction, the picture without black level correction will be brighter, affecting the contrast of the image.
3.LSC(Lens Shading Correction)
Due to the physical properties of the lens itself, resulting in a gradual decrease in brightness around the image relative to the center of the brightness, as well as, due to the image of the light through the lens to the pixel on the corner of the focus angle is greater than the center of the focus angle, resulting in the corners of the loss of light. The effect on the image is that the brightness gradually decreases from the center of the image to the surrounding area, and the farther away from the center of the image, the darker the brightness. In order to compensate for the surrounding brightness, Lens Shading correction is required.
Lens Shading correction is based on a certain algorithm to calculate the brightness correction value corresponding to each pixel, so as to compensate for the attenuation of the brightness of the surrounding.
There are two types of corrections: secondary and quaternary corrections.
4.BPC(Bad Pixel Correction)
a. Bad spots
The bad points are white dots in the output image under full black environment and black dots in the output image under high light environment.
b. Bad Dot Repair Methods
Generally, the RGB signal should have a linear response with the brightness of the scene, but due to some bad pixel of Senor, the output signal is not normal, and there are white dots or black dots.
There are usually two methods of bad spot repair:
One way is to detect bad pixels and repair them automatically, and the other way is to build a chained list of bad pixels for repairing bad pixels at fixed locations, which is the OTP way.
5. Demosaic Color Interpolation (Mosaic)
As the light passes through the Bayer CFA (Color Filter Arrays) array, the monochromatic light hits the sensor and each pixel is monochromatic, resulting in an ideal Bayer image that is a dimmer mosaic.
The first thing that needs to be clarified is that demosaiced is not literally meant to remove some of the mosaic images in the movie, but is a method used in digital image processing to interpolate from incomplete color samples to generate complete color samples (because the layer pattern looks like a mosaic, so it is known as demosaiced). mosaic). On the sensor side, you usually need to use CFA filter to get the Bayer pattern, and in the later processing, you need to turn the Bayer pattern into a complete RGB444 (true color) image. Such a module is needed in the ISP to do this.
There are many algorithms in traditional ISPs that can be used to do this interpolation, including nearest-neighbor method, bilinear interpolation, cubic interpolation, and so on. Public No.: OpenFPGA
6. Bayer Denoise
When acquiring an image using a cmos sensor, light levels and sensor issues are the main factors that generate a lot of noise in the image. At the same time, when the signal passes through the ADC, it introduces some other noise. These noises will make the image blurred and lose a lot of details, so the image needs to be denoised spatial denoising traditional methods such as mean filtering, Gaussian filtering, and so on.
However, the general Gaussian filter mainly considers the spatial distance between pixels when sampling, and does not consider the degree of similarity between pixel values, so the blurring result is usually a blur of the whole image. Therefore, nonlinear denoising algorithms, such as bilateral filters, are generally used, in which not only the spatial distance relationship between pixels is considered in the sampling, but also the similarity between pixels is added, so that the original image can be kept largely divided into pieces, and thus the edges can be maintained.
7. AWB (Automatic White Balance)
The basic principle of white balance is to reduce white objects to white objects in any environment, i.e., by finding the white blocks in the image and adjusting the ratio of R/G/B as follows:
R’= R * R_Gain
G’ = G * G_Gain
B’ = B * B_Gain
R’ = G’= B’
The AWB algorithm typically consists of the following steps:
(1) Color temperature statistics: according to the image statistics of the color temperature;
(2) Calculate channel gain: Calculate the gain of the R and B channels;
(3) Perform correction of off-color: Calculate the correction of off-color images based on the given gain.
8. Color Correction
Due to the human eye visible spectral response and semiconductor sensor spectral response of the difference between the spectral response, as well as the influence of the lens, etc., the RGB value of the color will be biased, so the color must be corrected, the usual practice is to color correction through a 3×3 color change matrix. The usual practice is to use a 3×3 color change matrix for color correction.
9. Gamma Correction
The human eye’s sensitivity to external light sources is not linearly related to the input light intensity, but exponentially related. In low illumination, the human eye is easier to distinguish changes in brightness, with the increase in illumination, the human eye is not easy to distinguish changes in brightness. The camera sensitivity and input light intensity is linear, in order to facilitate the human eye to recognize the image, the camera needs to capture the image for gamma correction.
Gamma correction is a non-linear operation performed on the gray values of the input image to make the output image gray values exponentially related to the input image gray values:
out = Vin ^ gamma
This index is gamma, the horizontal coordinate is the input gray value, the vertical coordinate is the output gray value, the blue curve is the input-output relationship when the gamma value is less than 1, the red curve is the input-output relationship when the gamma value is more than 1. It can be observed that when the gamma value is less than 1 (blue curve), the overall brightness of the image is improved, and at the same time, the contrast in the low gray level is increased, which is more conducive to the differentiation of image details in the low gray level.
10. Color space conversion
YUV is a basic color space, and the human eye is much more sensitive to changes in luminance than to changes in color, so the luminance component Y is much more important to the human eye than the chromaticity components U and V. Therefore, some of the U and V components can be discarded to achieve the purpose of data compression. Therefore, some of the U and V components can be discarded appropriately to achieve the purpose of data compression.
YCbCr is actually a scaled and shifted version of YUV, where Y stands for luminance and Cr and Cb stand for chromatic aberration, the difference between the luminance values of the RGB signals, which are the red and blue components, respectively. In the YUV family, YCbCr is the most used member of the computer system, its application areas are very wide, JPEG, MPEG are used in this format. The term YUV generally refers to YCbCr. YCbCr has many sampling formats, such as 4:4:4, 4:2:2, 4:1:1, and 4:2:0.
Cb: Reflects the difference between the blue part of the RGB input signal and the brightness value of the RGB signal. Cr: Reflects the difference between the red part of the RGB input signal and the brightness value of the RGB signal.Cb: Reflects the difference between the blue part of the RGB input signal and the brightness value of the RGB signal.
Cr: Reflects the difference between the red part of the RGB input signal and the brightness value of the RGB signal.
The range of values for each component of RGB and YCbCr in the following two equations is 0-255.
RGB to Ycbcr conversion formula:
Y = 0.257*R+0.564*G+0.098*B+16
Cb = -0.148*R-0.291*G+0.439*B+128
Cr = 0.439*R-0.368*G-0.071*B+128
YCbCr转换为RGB公式：
R = 1.164*(Y-16)+1.596*(Cr-128)
G = 1.164*(Y-16)-0.392*(Cb-128)-0.813*(Cr-128)
B =1.164*(Y-16)+2.017*(Cb-128)
Color space conversion module, is to convert RGB to YUV444, and then in the YUV color space for subsequent color noise removal, edge enhancement, etc., but also for the subsequent output to facilitate the conversion of jpeg images.
11.Color Denoise
In order to suppress the color noise of the image, a low-pass filter is generally used for processing. For example, an M×N Gaussian low-pass filter is used for processing on the chrominance channel.
