About sensor_mode and angle of view of picamera

About sensor_mode

When using the Raspberry Pi camera module, you may encounter phenomena such as the angle of view narrowing or the image quality changing when shooting at a specific resolution. This is a phenomenon caused by the difference in sensor_mode.

sensor_mode is a parameter that specifies how to use the image sensor. For example, you can use only some of the pixels of the sensor, or treat 4 pixels as 1 pixel. The operation of each mode is prepared as a correspondence table for each camera module.

Correspondence table

Here, the table of v2 module is quoted as an example of how to read the table.

--# (sensor_mode) on the far left This is the sensor mode. v2 module can be specified from 1 to 7. 2 and 3 are exactly the same, but this seems to be the result of considering compatibility with the v1 module.

• Resolution The resolution of the image to be captured. If you specify a value other than the value described here, resizing will occur.

• Aspect Ratio The aspect ratio of the image.

• Framerates This is the frame rate that can be set.

• Video^[1] Indicates the sensor mode that can be set when using the video port (the x mark in the table means ✓ [^ check]). It supports from 1 to 7. The video port can be used by setting ʻuse_video_port = True` when shooting a movie or shooting a still image. It is faster than the still port, but it tends to give a grainy image.

• Image^[2] This is a sensor mode that can be set when using the still (still image) port. The still port is shot at the maximum resolution, and when Resolution is specified, it is resized to that size. Powerful denoising can be used, but it is slower.

• FoV (Field of View) Which area of the sensor to use. If Full, the entire area of the sensor is used, and if Partial, a part is used. Therefore, the angle of view (FoV) is narrower for those that are Partial.

• Binning Whether to perform pixel binning. None treats the same pixel size (dot by dot), and 2x2 treats 4 pixels as 1 pixel. In addition to reducing the resolution, pixel binning also has the effect of reducing noise.

The figure about sensor_mode and shooting range is quoted below. From the table above and the figure below, you can see that sensor_mode = 2 or 3 or 4 should be used to shoot at the maximum angle of view using the entire sensor. You can also see that the angle of view becomes narrower if you set it to any other sensor_mode or resolution (for example, 1920x1080).

As will be described later, if sensor_mode is not specified, Resolution and Framerate will automatically select sensor_mode.

The tables and images shown here are taken from the picamera documentation below. https://picamera.readthedocs.io/en/release-1.13/fov.html

[^ check]: Overseas, the x mark is used to mean a check. https://www.sociomedia.co.jp/7304

Pixel binning

It is a flow of 2x2 binning. ①: Sensor color arrangement (Bayer arrangement) ②: Take the average of 4 pixels for each color (g1 and g2 are distinguished) ③: After taking the average

Since the pixel average is taken in this way, it has a noise reduction effect. It also has the effect of speeding up shooting because the number of pixels can be reduced before image processing (demosaic, white balance adjustment, etc.) is performed.

If you want to know more about the sensor, please look for the data sheet. IMX219 (v2 module) has binning written on p.53 of the data sheet. IMX219 (v2 module) datasheet: https://www.raspberrypi.org/forums/viewtopic.php?t=177308

Table of each module

picamera is currently (2020/10) and has not been updated for more than 2 years. For that reason, there are some parts that are different from the Raspberry Pi OS (raspbian) site. Therefore, here I will quote two tables, the picamera documentation table and the Raspberry Pi OS table.

Quote source: picamera document:https://picamera.readthedocs.io/en/release-1.13/fov.html Raspberry Pi OS document: https://www.raspberrypi.org/documentation/raspbian/applications/camera.md

v1 (OV5647) picamera document

Raspberry Pi OS document

It's almost the same except for the binning part. I think the meanings of 4x4binning and 2x2 + skip are different, but what about?

v2 (IMX219)

picamera document

Raspberry Pi OS document

What should be noted is the item of sensor_mode = 7. The maximum is 90fps for picamera, but 200fps for Pi OS document. 200fps is

1For frame rates over 120fps, it is necessary to turn off automatic exposure and gain control using -ex off. Doing so should achieve the higher frame rates, but exposure time and gains will need to be set to fixed values supplied by the user.

It is necessary to turn off the automatic exposure and gain control as written, but it seems that 120fps is supported even if it is not.

But for picamera documentation

The maximum framerate of the camera depends on several factors. With overclocking, 120fps has been achieved on a V2 module but 90fps is the maximum supported framerate.

And, although 120fps may come out due to factors such as overclocking, it says that the maximum supported frame rate is 90fps.

When I actually tried the settings of sensor_mode = 7, Resolution = (320, 240), framerate = 120 using picamera, I was able to confirm that 120fps was output. By the way, even with the same sensor_mode, smaller Resolution is faster due to resizing by GPU and reduction of transfer amount.

HQ (IMX477)

Raspberry Pi OS document

HQ data is not in the picamera documentation, so you need to refer to the Raspberry Pi OS documentation.

Verification

I actually took a still image by changing sensor_mode, Resolution, and use_video_port. It's a little long, so if you want to skip it, click [here](# Flow to image acquisition).

environment

• Raspberry Pi 4 (4GB)
• Camera module V2
• Python 3.7.3
• picamera 1.13

The following test chart was printed on A4 paper and used for shooting. The background color corresponds to the shooting range of each sensor_mode shown above. In addition, the illustration of Irasutoya is used to make it easier to understand the change in the angle of view.

photograph

Brute force sensor_mode, Resolution, use_video_port

The images taken by brute force sensor_mode, Resolution, and use_video_port are shown below. Resolution is the resolution that appears in the sensor_mode table. The posted image is a screenshot of the thumbnail.

The horizontal direction (→) is the difference in sensor_mode (1 to 7), and the vertical direction (↓) is the presence or absence of Resolution and use_video_port. The file name displayed below the image is

{Resolution}_{use_video_port}_{sensor_mode}.jpg

It is.

→ sensor_mode (1~7) ↓ Resolution, use_video_port (False, True alternate)

It looks like the whole image, and when the sensor_mode is the same, almost the same range is shown. From this, it can be seen that after shooting with the angle of view of the specified sensor_mode, it is cropped and resized to the size of Resolution.

There is no change in the difference in use_video_port. If sensor_mode is specified, will the value of use_video_port be ignored? Regarding the difference in the amount of noise due to the difference in use_video_port, I think that it can not be compared because the setting at the time of shooting is appropriate, but the feeling I saw was the same.

If you look at the difference in sensor_mode, you can see that there is a difference in the shooting range. This is the difference in angle of view depending on sensor_mode. Below is the image when the resolution and sensor_mode are matched. From the image, you can see that the background color of the test chart matches the shooting range.

(1920,1080)_False_1.jpg (sensor_mode=1)

(3280, 2464)_False_2.jpg (sensor_mode=2)

(3280, 2464)_False_3.jpg (sensor_mode=3)

(1640, 1232)_False_4.jpg (sensor_mode=4)

(1640, 922)_False_5.jpg (sensor_mode=5)

(1280, 720)_False_6.jpg (sensor_mode=6)

(640, 480)_False_7.jpg (sensor_mode=7)

If sensor_mode is not specified

Next, the result of changing Resolution and sensor_mode without specifying sensor_mode is shown below. The file name is the same as before

{Resolution}_{use_video_port}_{sensor_mode}.jpg

(Since sensor_mode is not specified, it is 0). The horizontal direction (→) is the presence / absence of use_video_port, and the vertical direction (↓) is the difference in Resolution.

When use_video_port = False (left column), the "AI book illustration" is shown in the image, so you can see that all the images were taken in the maximum shooting range.

Next, if you look at when use_video_port = True (right column), the image is the same as the left except for (1920,1080). However, it is more natural to set use_video_port = True to sensor_mode, which is the closest to Resolution (for example, (640, 480) narrows the angle of view).

This behavior depends on the sensor_mode determination method of picamera, and it seems that picamera has a mechanism to determine sensor_mode in consideration of Resolution and frame rate. (Reference) https://picamera.readthedocs.io/en/release-1.13/fov.html#sensor-modes

The above result was the result of framerate = 30 (default). Next, I will post the result with frame rate = 60.

As you can see from the result of framerate = 60, the shooting range when use_video_port = True has changed. Earlier, only the yellow area was shown (1920, 1080), but now it is shown up to the blue range. This means that sensor_mode = 1 is not compatible with 60fps, so it has been changed to sensor_mode = 6.

Since the other images are also shown in the blue range, it can be seen that they were taken with sensor_mode = 6. In addition, it can be seen that (640, 480) has sensor_mode = 7 in consideration of Resolution.

If use_video_port is not specified

Finally, it is the result of changing Resolution and sensor_mode without specifying use_video_port.

When framerate = 30

When framerate = 60

From the results, we can see that it is the same as the first verification. Since it does not change even if the framerate is changed, it seems that the hierarchical relationship of the settings is sensor_mode> framerate.

Impact of binning

Using the image of 1640x922, use_video_port = False, we compared sensor_mode = 4 (2x2 binning) and sensor_mode = 2 (resized maximum resolution without binning). The left is sensor_mode = 4 and the right is sensor_mode = 2. The posted image is a screenshot of two images displayed side by side on the viewer software.

Left: sensor_mode = 4 (binning), Right: sensor_mode = 2

Left: sensor_mode = 4 (binning), Right: sensor_mode = 2

I feel that the resolution with binning is lower than that without binning. Probably, the resolution is reduced because demosaic processing is performed after binning. If you want to prioritize resolution, it seems better to shoot at the maximum resolution without binning even if you resize later.

Although not verified this time, the noise reduction effect of binning may appear when shooting in a dark environment. Also, since we have not set parameters related to exposure and white balance this time, it may be better not to compare the noise feeling in the above image.

For the time being, I will write down the settings that I wrote out when shooting these two shots. The shutter speed is slightly different and the white balance value is different, but I don't think it will affect the comparison of resolution.

Flow until image acquisition

From the above results, the flow of processing related to image size, which is performed from shooting to obtaining an image, is shown based on three types of examples. (The processing shown in this section includes my expectations, so accuracy cannot be guaranteed.)

1 1. When sensor_mode = 1, resolution = (3280, 2464)

Table: sensor_mode correspondence table of v2 module (part)

The above table is a part of the v2 module table posted earlier. sensor_mode = 1 is the mode of 1920x1080. When shooting, the FoV is Partial, so a part of the sensor (orange part) is used for shooting.

After that, it is cropped to the specified Resolution (here 3280x2464) aspect ratio (4: 3) and resized.

• It is unknown whether cropping and resizing are performed in this order.
• The exact 4: 3 is 3280x2460, so the size after cropping may be slightly different. It is also unknown how the four pixels on the short side will be treated.

2. When sensor_mode = 2, Resolution = (800, 600)

Table: sensor_mode correspondence table of v2 module (part)

This is the same process as when using the still image port without specifying sensor_mode. Since the highest image quality is used in the still image port, it is slow because such processing occurs even at 800x600.

3. 3. When sensor_mode = 7, Resolution = (800, 600)

Table: sensor_mode correspondence table of v2 module (part)

Pixel binning is performed at sensor_mode = 7. Therefore, the range of 1280x960 of the sensor is used, and it becomes 640x480 by binning. After that, it will be resized to the specified resolution.

If sensor_mode is not specified

A resolution of 800x600 and a framerate of 60fps will select the 640x480 60fps mode, even though it requires upscaling because the algorithm considers the framerate to take precedence in this case.

Quote: https://picamera.readthedocs.io/en/release-1.13/fov.html#sensor-modes

According to the picamera documentation, resizing does not always shrink depending on the frame rate you set.

In this verification, we confirmed the occurrence of upscaling due to the frame rate limitation. If you want to avoid upscaling and shoot with high image quality, you should set sensor_mode manually.

Source code

This is the source code used for this verification. I think that there is no problem because the difference in sensor_mode appears well in this code. As a point to be worried about

--Do I need to create a PiCamera instance every time I change the sensor_mode? --Is it better to specify sensor_mode in the constructor [^ sensor_mode]

There is such a place, but I have not confirmed it. Also, the sleep before shooting is 1 second, but when actually shooting something, it is better to take about 2 seconds because exposure and white balance are related. This time, the verification of the angle of view is the main, so it is appropriate.

write_config () is a function that writes out camera settings (shutter speed, etc.). It is not directly related to sensor_mode verification.

import picamera
import numpy as np
import cv2
from PIL import Image

import io
import os, sys
import datetime
import time

def write_config(f, camera):
    """
A function that exports all the settings that can be exported by the camera
    """
    f.writelines(f"timestamp*={camera.timestamp}\n")
    f.writelines(f"revision={camera.revision}\n")
    f.writelines(f"resolution={camera.resolution}\n")
    f.writelines(f"-"*30)
    f.writelines("\n")
    shutter_speed = camera.shutter_speed
    exposure_speed = camera.exposure_speed
    if(shutter_speed!=0):
        f.writelines("shutter_speed={0} (1/{1:.2f}s)\n".format(shutter_speed, 1/(shutter_speed/1000000)))
    else:
        f.writelines("shutter_speed={0}\n".format(shutter_speed))
    f.writelines("exposure_speed*={0} (1/{1:.2f}s)\n".format(exposure_speed, 1/(exposure_speed/1000000)))
    f.writelines(f"exposure_mode={camera.exposure_mode}\n")
    f.writelines(f"exposure_compensation={camera.exposure_compensation}\n")
    f.writelines(f"iso={camera.iso}\n")
    f.writelines(f"sensor_mode={camera.sensor_mode}\n")
    f.writelines(f"analog_gain*={camera.analog_gain}\n")
    f.writelines(f"digital_gain*={camera.digital_gain}\n")
    f.writelines(f"framerate={camera.framerate}\n")
    f.writelines(f"framerate_delta={camera.framerate_delta}\n")
    f.writelines(f"framerate_range={camera.framerate_range}\n")
    f.writelines(f"meter_mode={camera.meter_mode}\n")
    f.writelines(f"drc_strength={camera.drc_strength}\n")
    f.writelines(f"raw_format={camera.raw_format}\n")
    f.writelines("-"*30)
    f.writelines("\n")

    f.writelines(f"image_denoise={camera.image_denoise}\n")
    f.writelines(f"video_denoise={camera.video_denoise}\n")
    f.writelines(f"video_stabilization={camera.video_stabilization}\n")
    f.writelines("-"*30)
    f.writelines("\n")

    f.writelines(f"awb_gains=    {camera.awb_gains}\n")
    f.writelines(f"awb_mode=     {camera.awb_mode}\n")
    f.writelines(f"brightness=   {camera.brightness}\n")
    f.writelines(f"saturation=   {camera.saturation}\n")
    f.writelines(f"contrast=     {camera.contrast}\n")
    f.writelines(f"sharpness=    {camera.sharpness}\n")
    f.writelines(f"flash_mode=   {camera.flash_mode}\n")
    f.writelines(f"rotation=     {camera.rotation}\n")
    f.writelines(f"hflip=        {camera.hflip}\n")
    f.writelines(f"vflip=        {camera.vflip}\n")
    f.writelines(f"zoom=         {camera.zoom}\n")
    f.writelines("-"*30)
    f.writelines("\n")

    f.writelines(f"color_effects={camera.color_effects}\n")
    f.writelines(f"image_effect={camera.image_effect}\n")
    f.writelines(f"image_effect_params={camera.image_effect_params}\n")
    f.writelines(f"still_stats={camera.still_stats}\n")

# -----------Parameter setting used for verification--------------
RESOLUTION_LIST = ((3280, 2464),   # full sensor area #2, #3
                    (1640, 1232),  # full sensor area(binned) #4
                    (1640, 922),   # #5
                    (1280,720),    # #6
                    (1920, 1080),  # #1
                    (640, 480))    # #7
FRAMERATE = (30, 60)
USE_VIDEO_PORT = (True, False)                    
SENSOR_MODES = (1,2,3,4,5,6,7)

nowtime = datetime.datetime.now()
outputdir = nowtime.strftime("%Y%m%d-%H%M%S")
os.mkdir(outputdir)

TEST1 = True
TEST2 = True
TEST3 = True
# -----------------------------------------------------

"""
test1
Resolution, use_video_port, sensor_A test to brute force the combination of modes
"""
if(TEST1):
    foldername = os.path.join(outputdir, "test1")
    os.mkdir(foldername)

    for resolution in RESOLUTION_LIST:
        for use_vp in USE_VIDEO_PORT:
            for sensor_mode in SENSOR_MODES:
                with picamera.PiCamera() as camera:
                    camera.resolution = resolution
                    camera.sensor_mode = sensor_mode
                    camera.start_preview()
                    time.sleep(1)

                    stream = io.BytesIO()
                    camera.capture(stream, format="jpeg", use_video_port=use_vp, quality=95)
                    camera.stop_preview()
                    stream.seek(0)
                    img = Image.open(stream)
                    img = np.array(img, dtype=np.uint8)
                    img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)

                    filename = "{0}_{1}_{2}.jpg ".format(str(resolution), str(use_vp), str(sensor_mode))
                    cv2.imwrite(os.path.join(foldername, filename), img)
                    #cv2.imshow("capture", img)
                    print(filename)
                    print(img.shape)
                    print("")
                    with open(os.path.join(foldername, filename[:-4]+".txt"), "w") as f:
                        write_config(f, camera)

"""
test2 
sensor_Test when mode is not specified
"""
if(TEST2):
    for framerate in FRAMERATE:
        foldername = os.path.join(outputdir, "test2_{0}fps".format(framerate))
        os.mkdir(foldername)

        for resolution in RESOLUTION_LIST:
            for use_vp in USE_VIDEO_PORT:
                with picamera.PiCamera() as camera:
                    camera.resolution = resolution
                    camera.framerate = framerate
                    # camera.sensor_mode = sensor_mode  # sensor_Do not specify mode
                    camera.start_preview()
                    time.sleep(1)

                    stream = io.BytesIO()
                    camera.capture(stream, format="jpeg", use_video_port=use_vp, quality=95)
                    camera.stop_preview()
                    stream.seek(0)
                    img = Image.open(stream)
                    img = np.array(img, dtype=np.uint8)
                    img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)

                    filename = "{0}_{1}_{2}.jpg ".format(str(resolution), str(use_vp), str(camera.sensor_mode))
                    cv2.imwrite(os.path.join(foldername, filename), img)
                    #cv2.imshow("capture", img)
                    print(filename)
                    print(img.shape)
                    print("")
                    with open(os.path.join(foldername, filename[:-4]+".txt"), "w") as f:
                        write_config(f, camera)   

"""
test3
use_video_Test when port is not specified
"""
if(TEST3):
    for framerate in FRAMERATE:
        foldername = os.path.join(outputdir, "test3_{0}fps".format(framerate))
        os.mkdir(foldername)

        for resolution in RESOLUTION_LIST:
            for sensor_mode in SENSOR_MODES:
                with picamera.PiCamera() as camera:
                    camera.resolution = resolution
                    camera.sensor_mode = sensor_mode
                    camera.framerate = framerate
                    camera.start_preview()
                    time.sleep(1)

                    stream = io.BytesIO()
                    camera.capture(stream, format="jpeg", quality=95)  # use_video_Do not specify port
                    camera.stop_preview()
                    stream.seek(0)
                    img = Image.open(stream)
                    img = np.array(img, dtype=np.uint8)
                    img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)

                    filename = "{0}_{1}_{2}.jpg ".format(str(resolution), "None", str(sensor_mode))
                    cv2.imwrite(os.path.join(foldername, filename), img)
                    #cv2.imshow("capture", img)
                    print(filename)
                    print(img.shape)
                    print("")
                    with open(os.path.join(foldername, filename[:-4]+".txt"), "w") as f:
                        write_config(f, camera)

in conclusion

It is possible to shoot without being aware of sensor_mode, but I found that I should consider what kind of performance (high image quality, angle of view, speed) I need. Personally, I was surprised that the decrease in resolution due to pixel binning was greater than I expected.

Most of this article is taken from the picamera documentation. The picamera document describes not only how to use the library, but also the mechanism of the camera module and the flow of shooting, and it is very helpful, so we recommend that you read it.

reference

https://picamera.readthedocs.io/en/release-1.13/index.html (picaemra documentation) https://www.raspberrypi.org/documentation/raspbian/applications/camera.md (Documentation for Raspberry Pi camera application)