Developing digital twins of multi-camera metrology systems in Blender (2024)

The first measurement scenario consisted of measuring the diameter of a sphere using three Raspberry Pi V2 cameras, located on the x, y and z unity axes, at different distances from the sphere. The Raspberry Pi V2 was chosen because it is commonly available, cheap, straightforward to set up and use, with fixed optics that are characterised by the pinhole camera model. It is noted here that a consequence of the design means that the camera has a very large depth of field although best focus for this camera version is typically found in the first few metres of the image. The wide FOV (horizontally 62°, vertically 49°) makes this camera very suitable to internal confined space work hence its selection.

Spheres are widely used in metrology applications for camera calibration [37, 38], in addition the use of a sphere allows for simulated light interactions with 3D objects. A sphere is also convenient because it is geometrically speaking always observed as a two-dimensional (2D) circle no matter which orientation it is viewed from. This characteristic compensates for potential imperfections of the real and virtual experimental setup, for instance physical camera misalignment, allowing this experiment to be reproduced without adding new sources of error such as detection problems coming from such misalignment.

Whilst a sphere is straight forward in concept the simple geometry still introduces consequences for the measurement of the sphere diameter using a 2D camera. Figure 1 shows a sphere albeit in this case illustrated as a circle. Point C represents the camera's optical centre whilst points A and B are the intersection points between the extremities of the FOV and the circle. These extremities are tangential to the circle.

The segment [AB] is not equal to the real sphere's true diameter, but represents the diameter perceived by the camera because these points are the interception points between the camera's FOV, and the sphere. The angle θ will increase with increasing object distance between the sphere and the camera. Consequently, as the object distance tends to infinity then θ approaches 90°, and the segment [AB] as recorded by the camera would then approximate to the true diameter. However, it is also noted that as the object distance tends to infinity, then the spatial resolution of the camera image will degrade as a function of reduced pixel density thus introducing alternative error terms.

The outcome criterion from this real and virtual experimentation was the comparison between deviation of the measured observable sphere diameter and θ to the theoretical values. The theoretical observable sphere diameter and θ were calculated using the geometry shown in figure 1, and Pythagoras' theorem (equation (1)).

The real-world experiment was comprised of a 0.09 m diameter white sphere, two Raspberry Pi V2 pinhole cameras located on the x and y-axes (the z-axis measurement was obtained by rotating the sphere by 90° on the y-axis). The cameras were located at an object distance varying from 0.2 m to 2 m, in steps of 0.2 m from the sphere. Measurements were non-sequentially repeated three times to provide understanding of potential variance as a function of camera placement.

The experiment took place in a controlled environment (illumination intensity control, noise coming only from the cameras, etc), with two light sources, one coming from the ceiling, and one coming from underneath the sphere to avoid shadows, as shown in figure 3. The top one was a 46 W LED light unit and, the bottom one was a 25 W LED light unit. A dark background was used to create high contrast between the white room and the white sphere.

A series of MATLAB functions were developed to detect the sphere within the FOV, and return key data elements. Different functions in MATLAB exist to transform a RGB image into a binary image, and to detect a sphere in an image. This research used the im2bw(originalImage, 0.4) function for the binary transformation, and the regionprops (in the script regionprops(logical(binaryImage), originalImage, 'all')) function for the sphere detection, based on the accuracy of the regionprops function in detecting a sphere in an image, and returning key values such as the diameter of the sphere in pixels. The camera image was cropped manually (examples shown later in figure 7), by drawing a rectangle around the sphere.

The processing steps followed were:

• Crop the image around the sphere to remove undesirable background elements.
• Load the cropped image.
• Transform the RGB image to a binary image.
• Detect the sphere in the binary image.
• Measure the diameter of the sphere in pixels.
• Calculate the diameter of the sphere in metres:

The diameter in pixels found with regionprops was converted to metres using equation (1):

The real-world experimentation was then recreated in the virtual environment. The experiment again consisted of measuring the diameter and centre of a 0.09 m diameter white virtual sphere from pictures taken by the Blender simulated virtual pinhole camera located at (again) object distances varying from 0.2 m to 2 m, in steps of 0.2 m from the sphere. The whole scene was generated in Blender using a Python script as shown in figure 2. For the purposes of visualisation within this report, figure 3 does not include the black background that was used to maximise object contrast. Multiple repeats (n = 3) of the experiments were completed to determine any variance in the output.

A dedicated camera model was created in Blender and used to simulate the Raspberry Pi V2 pinhole camera. The parameters of both the real and virtual cameras were; focal length of 3.03 mm, image resolution of 1640 × 1232 pixels (using 1.12 μm square pixels), and a sensor size of 3.68 mm × 2.76 mm, as defined in the camera datasheet [39].

The Blender model sphere was modelled with 32 polygons on a black background, noting that objects created in Blender are all based on a polygon mesh. Hence, the number of polygons represent the resolution of the sphere geometry, consequently the more polygons the sphere is modelled with, then in theory the more 'spherical' the sphere will be. Polygon density was considered in a separate series of background experiments with resolutions of 100 and 1000 polygons, but any differences of results compared with 32 polygons were found to be negligible.

In Blender, illumination intensity is defined depending on the light source. For instance, sunlight is defined in terms of watts per square metre but when considering artificial lighting (spot or area lights) then the lighting is specified in watts noting that this relates to radiant flux and not to electrical energy [40]. Consequently, it is difficult to directly correlate illumination intensity between a real-world environment and the equivalent Blender model environment. In this series of experiments, two square area lights with default values initially fixed at 10 kW were placed at 1 m above and below the sphere to illuminate the whole sphere hom*ogeneously. Whilst 10 kW would be a significantly large value in the real-world scenario, within Blender this allowed for even illumination with minimal shadowing.

Three pinhole cameras were set up on the x, y and z axes to view the sphere from different points of view. The Blender modelling environment was set up to be perfect (no noise, distortion, object surface texture, etc were introduced). The same MATLAB functions and data processing developed for the real-world scenario were used.

Whilst the first series of experiments was to assess camera performance in Blender in the context of object (sphere) measurement, the second series of experiments was designed to evaluate the ability of Blender to predict the camera location relative to a camera calibration artefact, using photogrammetry techniques. This consisted of comparing the reprojection errors of the detection of the centre of five zone circles printed on a calibration board using an array of eight Raspberry Pi V2 cameras—both in the real-world environment and the Blender virtual environment.

Photogrammetry uses triangulation to rebuild a 3D scene from multiple separate images, requiring a minimum of two cameras. Background experiments were completed with a number of cameras (ranging from three to eight), to determine which combination of cameras were likely to give lower reprojection errors. According to the background results, eight were chosen.

The reprojection error [41] is a geometric error corresponding to the image distance between a projected point and a measured point. In this instance, based on the standard deviation of set of measurements, the reprojection error is used to quantify the extent to which the estimation of a 3D point recreates the real projection of the point. In the case of detection, this error makes it possible to quantify the quality of circle (centre) detection by a set of cameras and to observe the variations in detection as a function of the positions of the cameras in the scene. It is noted that quantification of reprojection errors is inconsistently reported in literature and can also be reported as a function of the root-mean-square of all the reprojection norms, mean of the reprojection errors, and, median of the reprojection errors.

With respect to equation (2), the standard deviation represents the variation or dispersion of a set of samples around the mean on the basis that the data is parametric and follows a Gaussian or Normal distribution. In this instance it is used as a measure for the reprojection error, but does not allow specific identity of error components or individual contributions

where; σ is the standard deviation of the error mean, N_e is the number of errors, and 1.96 the multiplication factor to obtain a result at a coverage factor of k ≈ 2 (95% confidence limits based upon a Normal distribution).

Circular targets have been widely used in image processing and computer vision for their invulnerability to partial occlusion and their ease of access, which increases their use in optical measurement systems [42–44]. However, the accuracy of circle centre detection is sensitive to distortion, centre deviation and other stochastic factors in the imaging process [42]. To minimise the impact of these sensitivity factors, the pattern used was comprised of five zone circles. For better detection and localisation in the image by MATLAB, an alternation between black and white zone circles was used.

There is significant debate within the computer image community about the performance characteristics of zone circle-based calibration artefacts versus checkerboard-based artefacts with performance dependent on the nature of the pixel processing algorithms used. In this instance, reliable results were obtained with the zone-circle pattern because all pixels on the periphery of the circles can be used to accurately determine the circular shape of the pattern. In addition, the zone circles enabled three estimates of the target centre to be made by considering the central black circle, the white circle that surrounds it and finally the outer back circle.

In figure 4, the zone circles cover almost the entire FOV, which facilitates detection from different viewpoints. Circle 3 positioned at 0 mm from the x-axis was used as a reference to check the measurements, and the placement of the pattern on the virtual board. The origin of the frame is placed in the centre of the artefact, as shown in figure 4, along with the zone circle order.

The zone circles were 0.1 m, 0.15 m and 0.2 m respectively in diameter on a white diffuse substrate measuring 0.5 m × 0.75 m × 0.025 m. However, due to the overall size of the zone circle artefact, it was challenging to cause the entirety of the object to be within the FOV of each camera at 0.2 m, 0.4 m, 0.6 m and 0.8 m object distance—the artefact being partially cropped within the images at these object distances. A smaller additional artefact was physically developed whereby the zone circles were 0.029 m, 0.048 m and 0.069 m respectively in diameter on a substrate measuring 0.21 m × 0.29 m × 0.005 m. Both artefacts were likewise simulated in Blender by importing the original digital artwork. The (x,y) coordinates of each zone circle for both the small and large artefacts are listed in the table 1 with reference to the origin sited at the middle of each board.

The coordinates of each individual zone circle centre were extracted from each set of camera images, before using triangulation to create a 3D reconstruction of their positions in relation to the camera positions. The reprojection error for each zone circle centre was determined by calculating the deviation between the reference circle centre coordinates and the coordinates determined within MATLAB, with the final answers expressed in terms of the standard deviation.

Eight real Raspberry Pi V2 cameras (with the same characteristics as those used for the sphere experimentation), were placed around the real zone circle calibration board. Two cameras were used at each position defining a triangular pattern of analysis (in this case illustrated in figure 5 as part of the Blender virtual equivalency). The distance between each camera on the z axis was 0.25 m. The camera-object distance varied from 0.2 m to 1.8 m (limited by physical room dimensions) incrementing in steps of 0.2 m. Multiple repeats (n = 3) of the experiments were completed to determine any variance in the output.

The calibration zone circle board was placed in the middle of the measurement volume, approximately equidistant to each camera. A dedicated MATLAB based set of algorithms were developed to complete the data processing and return key data elements. The processing steps followed were:

• Improve the contrast of each picture to allow for better detection with the function imadjust(Image, [0.6, 0.7]).
• Define the focal length and sensor size of the camera according to the datasheet [39].
• Calculate the intrinsic matrix with cameraIntrinsics(focal, principalPoint, imageSize).
• Filter the noise in the image with the function medfilt2(Image, 'indexed').
• Transform the RGB image to a grey scale image with the function rgb2gray(DenoiseImage).
• Transform the grey image to a binary image with the function imbinarize(GrayImage, 'adaptive').
• Calculate the complementary image with the function imcomplement(BinaryImage).
• Detect the centre of each circle with the functions regionprops(Image, 'basic') on the binary and complimentary images before using k-means (centers, 5) to find the centre of the cluster created.
• Check the order of the circle centre.
• Circles 1-3-4: The cross product of the vectors created between circles 1-4, 1-2 is calculated, and its sign checked. If the sign is null, the centre of the board is detected correctly; if the sign is positive circle indices 1 and 4 are correct; Circles 2-5: The angles between circles 1-4, and 1-2 is calculated, as well as the angle between 1-5, and 1-4, using simple trigonometric functions. If the circle order is correct, the angle formed between 1-4, and 1-2 is higher than the other one.
• Calculate the camera parameters (rotation matrix and translation vector) with the function extrinsicsToCameraPose(centres, reference centres, intrinsics parameters).
• Calculate the 3D coordinates of each target centre by triangulation using the point reprojected in the image frame, and the camera positions.
• Calculate the reprojection error.

The reprojection errors were calculated as based on the description given in section 2.2.1.

The results from the real single camera measurement of the 0.09 m sphere are shown in figure 6 demonstrating the deviation of the measured sphere diameter from the actual diameter. The x-axis scale represents the distance between the camera position and the sphere, the left side y-axis scale is the difference between the theoretical and measured diameter observable (deviation or residual), and the right y-axis scale is the area of the sphere within the FOV in camera pixels. The mean value of the multiple repeat results from the three camera positions on the orthogonal sphere axes are shown. The standard deviation of the trial data with respect to the diameter deviation was 0.59 mm, 0.61 mm, and 0.53 mm respectively for Trial 1–3, whilst the standard deviation across the trial repeats at each object distance ranged from 0.0002 m to 0.013 m, the latter value occurring at 0.4 m object distance and attributed to geometry error as a function of figure 1.

The data demonstrates that the three trials generate similar results, with similar data trends. The absolute deviation of the sphere diameter decreases with increasing object distance. The initial significant deviation from the true diameter is a function of the camera FOV relationship when viewing a sphere as defined in figure 1. The deviation improves with increasing object distance as noted with three sets points in the data summarised in table 2. Here it is noted that there is a discrepancy between the Trial 1 deviation data and that for Trials 2 and 3. This has been attributed to subtle differences of camera set-up between experiments (noting that the three trials were not completed contiguously) and noise elements within the experiment.

The size of the sphere defines the number of camera pixels that constitute the sphere within the FOV. As expected, as the camera object distance increases, the sphere is resolved using fewer pixels thus potentially reducing the accuracy of measurement and hence the accuracy of sphere diameter measurement at longer distances with all three camera positions showing this trend. Moreover, the camera resolution decrease causes increasing image blur as shown in figure 7, likewise affecting the quality of measurement although it is also noted that the V2 version of the Raspberry Pi camera tends to suffer from defocus issues at longer object distances as a function of the original factory build and configuration.

The results from the Blender modelled single camera measurement of the 0.09 m modelled sphere are shown in figure 8. This likewise demonstrates the deviation of the measured sphere diameter from the actual modelled diameter, and decreasing camera pixel resolution as a function of object distance using the same axis descriptions as for figure 6. Unlike the real environment experiments, the Blender analysis did not use three repeats. This is because the algorithmic nature of Blender means that repeating the same analysis multiple times with the same data input leads to negligible variance of data output. This was previously determined through background experimentation.

Initially, the sphere diameter deviation decreases up to 0.6 m object distance, but then unlike the real-world experiment the deviation negatively increases as the object distance progresses to 2 m. However, it should be noted that the left side y-axis scale factor in figure 8 is an order or magnitude smaller than the equivalent in figure 6, and if plotted on the same scale as figure 6 would approximate to a straight line. In the first instance, the deviation comes from the sphere geometry, and the diameter measurement is function of the camera FOV (see figure 1). The subsequent negative increase of diameter deviation is due to the lighting and the resolution of the camera. In addition, it was determined that the same blurring process occurred in the virtual model (similar to figure 7). It is noted here that during further background experimentation it was determined that too much light within the Blender environment exaggerated the environment of the sphere yet did not seem to saturate the virtual camera image sensor—this being a key limitation of real cameras yet not noted as being a characteristic of the Blender camera model.

Figure 9 illustrates this point. Using the Blender setup described in section 2.1.3, and the methodology given in section 2.1.1, seven light powers (5 kW, 1 kW, 0.5 kW, 0.1 kW, 75 W, 50 W and 46 W) were simulated to determine the impact of light power on the sphere diameter measurement in the virtual environment. However, to further remove the geometry error influence illustrated in figure 1, a 0.09 m diameter circle was modelled instead of a sphere—placed on a black background to achieve similar white/black high contrast associated with the previous sphere measurements.

Figure 9 shows the diameter deviation is greater for high power (5 kW, 1 kW and 0.5 kW), than for low power (0.1 kW, 75 W, 50 W and 46 W). Thus, the more powerful the light and illumination, the more the details of the circle are exaggerated, resulting in errors in the detection of the circle (and consequently sphere) exacerbated as a function of increasing object distance. This background element of work confirms that object specific illumination thresholds may exist but can only be determined on a case-by-case basis.

The initial choice of 10 kW illumination was originally and correctly justified to maintain even illumination across the entire sphere and remove shadowing (section 2.1.1) but this still generated errors when detecting the diameter of the sphere. It has been identified that in Blender, the illumination set-up and characteristics does not necessarily give the same result as the same illumination in the real world, this partly being a function of the colour transform used in Blender to generate the rendered image. By default, Blender uses the sRGB colour transform, which was originally designed to approximate the response of a cathode-ray tube monitor [45].

Table 3 also reveals that the size of the sphere is not the same between the real and the virtual image even though the camera parameters were set-up as per the manufacturer's specification. For instance, at 0.2 m for Trial 1 (table 2), the size of the sphere in the real image is 467 382 pixels compared to a virtual equivalent size of 305 561 pixels (table 3). As the object distance increases then the difference in pixel count becomes more obvious. This is related to the manner in which Blender generates images, whereby pixel count is linked to the output of the rendered scene rather than necessarily relating it directly to the number of pixels at the camera image plane.

In addition, this difference may be due to the grey scale of the real-world image versus the Blender equivalent. In Blender, the sphere is perfectly white [255, 255, 255], and the background perfectly black [0, 0, 0], whereas in the real-world case the white and black elements of the image are not perfectly white and black but contain grey scale components of varying values. In MATLAB, to detect the sphere in an image, a binary image process is employed and uses a threshold to determine the grey scale components of the image based on the Otsu method [46]. Due to the different grey scale values in the image, fewer pixels are used to detect the sphere in MATLAB.

The results from the real-world multiple camera measurement of the calibration zone circles are shown in figure 10 noting that again for the real-world investigation there were three repeats of the experimentation. The x-axis represents the distance between the cameras and the calibration artefact and the y-axis represents the reprojection error of the centre of each circle detected. Due to data density as a function of five repeat experiments for each trial, only Trials 1 and 2 are shown here to illustrate both magnitude and trend characteristics.

Two sets of data are presented for each experiment using the two separate zone circle artefacts. The smaller artefact was used for an object distance varying between 0.4 m and 0.8 m (figure 10), the larger artefact was used for object distances varying between 1.0 m and 1.8 m (figure 11).

The data (also shown in tables 4 and 5 respectively) demonstrates that the three trials generated similar results, with similar data trends for both versions of the artefacts, but not with the same magnitude of results. For instance, for zone circle 1 at 0.4 m, the reprojection error was 5.317 mm for trial 1, 7.002 mm for trial 2 and 0.272 mm for trial 3 respectively. The difference between the results comes from the script used for the detection, and the physical setup. Because the tripod mounted cameras were moved manually from 0.4 m to 1.8 m, it was found to be very challenging to maintain consistent camera rotation and positional parameters (with six translational and rotational degrees of freedom available) during repositioning at each object distance. In addition, due to the variation in camera configuration, the light on the surface of the sphere was not perceived by the camera in the same manner, which led to errors in the detection of the sphere in the image during processing in MATLAB.

With respect to figure 10 and table 4, the reprojection error increases between 0.4 m and 0.6 m, before decreasing. The reprojection error is larger for zone circles 1, 2, 4 and 5, whilst zone circle 3 has the smallest value. This difference is attributed to the fact that the zone circles are paired (1 with 4, 2 with 5) but 3 is an individual zone circle. With increasing object distance, zone circles 1 and 4 start to become more difficult to identify, likewise for 2 and 5. Moreover, zone circle centre detection might not be the true geometric centre (for each zone circle) due to optical changes of perspective and depth of field. In addition, the circles are 'ovalized', changing slightly the centre coordinates and the height of the board.

It is noticeable that Trial 2 gives a larger reprojection error than Trial 1 at 0.4 m and 0.8 m, a function of the physical setup, the camera location and orientation, and, the lighting condition. In addition, the circle detection script may be sensitive to variation, as a function of incorrect detection of the circles caused by the image contrast and the illumination environment.

With respect to larger zone circle artefact (figure 11 and table 5) the reprojection error decreases between 1.0 m and 1.6 m, albeit before slightly increasing at 1.8 m for Trial 1 and 2. For Trial 3, the reprojection error increased from 1.2 m to 1.4 m, before slightly decreasing. For zone circles 1, 2, 4 and 5 the reprojection errors were larger, whilst zone circle 3 again had the smallest value.

Similar trends for Trials 1 and 2 are observed across both sets of data shown in tables 4 and 5. However, the results for zone circle 3 are of similar values for both trials, and the results for the large artefact analysis shown in table 5 are larger than in table 4 showing that reprojection errors increase with object distance. This is to be expected due to changing pixel resolutions as a function of object distance and orientation of the camera.

The analysis of the reprojection errors determined within the Blender virtual environment is presented in a similar fashion to the real-world analysis within figure 12 (short distance) and figure 13 (long distance), supported by tables 6 and 7 respectively.

With respect to both figures and tables of data, the reprojection error for zone circles 1, 2, 4 and 5 increases with increasing object distance, whilst zone circle 3 is fairly consistent (and small) in its reprojection values. This is similar overall behaviour to that seen in the real-world experiments. Zone circle 3 may be exhibiting the best results because of its position within the coordinate framework of the artefact and analysis. In addition, texture adjustment (see below), camera distortion, and board size were all found to have varying effects on the y-axis component of the measurements, when defining the height of the array.

The data presented in tables 6 and 7 further demonstrates that the Blender environment reprojection error appears to be influenced by the size of the calibration board. At 0.4 m the reprojection error is between 0.02 mm and 3 mm, at 1.2 m, the reprojection error is between 0.05 mm and 0.7 mm. It is also clear that there is a step change in magnitude between the use of the two boards, with the smaller board causing larger reprojection errors. It is anticipated that the small board is more difficult to detect than the larger one due to its size, and consequently there is more potential for the merging of the circles within the camera images as a function of increasing distances and reducing pixel resolution. Clearly, this does not follow the real-world equivalent trends for the two sizes of artefact.

Analysis of the data and also reflecting on the requirements for establishing the virtual models in Blender, reveal that there are (as expected issues with matching the model to the real-world. For instance, in the real-world—the zone circle artefact is printed on flat laminate board which has a natural element of (diffuse) surface texture—likewise the spheres in the diameter measurements exhibit texture. These real textures can be explicitly measured and quantified in terms of 2D or 3D surface texture parameters.

However, in Blender the menu options used in the set-up of the model provide texture descriptors that are considerably more broad ranging in composition and interpretation (based on nodes, light absorption and reflection properties) that are significantly subjective and qualitative in nature, and do not appear to directly relate to the quantified metrics [40]. Texture adjustment can be achieved with the (Blender) size tool, or with the smart match tool. It was observed that iterative adjustment of the object texture caused a change of the coordinates of the centres of the zone circles, which led to incorrect detection and inflated reprojection error terms. Hence it is difficult to directly correlate the texture on the modelled zone circle board with real-world numerical values (likewise the sphere in the diameter measurement scenario).