revolution

The LiDAR sensor in the iPhone 12 Pro, released in the fall of 2020, has revolutionized the 3D scanning environment.

3D scanning, which until then required expensive and heavy scanning equipment and a high-performance PC, could now be done with a single hand and for only 100,000 yen (at the time of its release).

At the time, the impact in the 3D scanning community was tremendous, and I remember the shouts of joy that went up late at night, including the author's. The iPhone12Pro was the first of its kind.

Starting with the iPhone12Pro, scanning applications and scanning methods have been developed one after another by both companies and individuals.

On the other hand, there are those who say, "Now that we can easily scan in 3D, what should we do with it? What to do with it?

In this Tech Blog, I would like to summarize the features of each 3D scanning method that is becoming popular, as well as how to use the 3D models you create.

Popularized scans

As smartphones equipped with lightweight and high-performance cameras and LiDAR sensors hit the market, companies and individuals are developing 3D structure estimation methods and scanning applications.

Since there are various methods and features, this section describes the characteristics of each scanning method and introduces corresponding applications.

Photogrammetry

This method uses a large amount of image data to estimate the shape of an object.

The shape of an object is determined by detecting feature points in each image and matching them with each other. To put it in an easy-to-imagine way, stereograms using binocular disparity are made in all directions and angles.

Depending on the software, the key is to align the images using the feature points, so the resolution of the images is very important. The higher the resolution, the sharper and more accurate the edges of the object. On the other hand, since the position estimation is based on the feature points of each image, continuous patterns or objects that are reflected or transmitted through the image are unstable, making good shape estimation impossible. In addition, the scale must basically be adjusted by post-processing.

This technique, which can be performed even with analog images, was used in the old days to estimate the weight of the Great Buddha in Kamakura, for example.［1］

Recommended App: RealityScan

RealityScan is a photogrammetry application released by Unreal Engine. It has a support function to improve scanning accuracy and a preview function to check if the number of images taken is sufficient. It is available for free.
https://www.unrealengine.com/ja/realityscan

LiDAR Scanning

LiDAR is a method of determining the shape of an object using the LiDAR sensor mounted on the iPhone PRO series.

LiDAR (Light Detection and Ranging) is a method of measuring the distance and direction to an object using the reflection time of laser light emitted from a device. By doing this continuously, it is a method to estimate the 3D shape of an object.

It is a method with short estimation time and good accuracy because the distance to an object can be measured accurately and quickly using laser light. It is also convenient for secondary use because of its scaling.

As with photogrammetry, feature points are also used for object estimation, so reflective and transparent objects cannot be estimated well. Also, because of the limitation of the irradiation distance of the laser beam used, shape estimation can only be successfully performed at a distance of about 5 m from the device. ［4］ Recently, the shape estimation system has been used for automatic driving vehicles.

Recently, laser beams have begun to be used in self-driving cars, and are also used in cat-shaped robots that are often seen in restaurants.

Recommended App: Scaniverse

This is an application that is almost certain to be found in the smartphones of anyone who does 3D scanning (according to the author's research).

It is basically a very useful tool that can be used on both iOS and Android devices and should be downloaded immediately as it can be used on devices that do not have LiDAR. Of course, using the Pro version of iPhone with LiDAR is the most accurate and quickest way to scan.

It also supports "3DGaussianSplatting" (described later) and is becoming an all-purpose apple.

https://scaniverse.com/

3DGaussianSplatting

This method uses a concept known as 3DGaussian to smoothly represent point clouds.

3DGaussian is represented as a thin spherical object with a contour that has position, covariance, transparency, and viewpoint-dependent color information for each point. The point cloud is created using the same method as for photogrammetry described above. ［5]

By splatting them on a certain surface, an image from a certain viewpoint is approximated, like a 3D watercolor painting.

As you can see from the estimated 3D model, the areas with appropriate information are well estimated, and for areas with sparse information, you can see an elongated ellipse-like 3DGaussian.

Unlike the two methods described above, this method compares the generated 3D model with the input image data at the same angle of view. The accuracy of the 3D model is improved by optimizing the difference in the direction of minimizing the difference using the objective function and the aforementioned 3DGaussian parameters as variables.

This is useful for scanning mechanical parts, cars, etc., because it can appropriately estimate the shape of reflective and semi-transparent objects.

Recommended App: LumaAI

LumaAI is very convenient for sharing on social networking sites, as it can output rendered images without any special operations, in addition to 3D shape estimation using 3DGaussianSplatting.
In addition, since shape estimation can be performed on a website without a smartphone by inputting a group of photos or video images, it is possible to create more precise models by using a high-resolution camera.

https://lumalabs.ai/interactive-scenes

Off topic, but on June 12, 2012, the company announced the "DreamMachine," a video generation AI. The 3D models created by LumaAI are also believed to be useful in the training of these models.

Where are 3D models headed?

What should we do with the scanned 3D model? It would be a waste to keep them in your smartphone. Here are some of the ways the author recommends to enjoy them.

Enjoy a handheld

This is probably the most popular method.

If you use a normal 3D scanning application, it will output an OBJ file or FBX file at the same time as post-processing. If you use your own 3D printer, you can output the OBJ or FBX file as it is, and in no time at all, you will have a hand-held guardian dog.

We recommend that you keep a mysterious sculpture that you found on your travels with you, as it will bring back vivid memories of your travels.

Enjoy a bird's eye view

Once you get used to it to some extent, you will be able to scan large objects. In such a case, you can composite models together and enjoy an overhead view! Like Sylvanian Families.

The following is "Grandma's house" that I scanned and combined on the modeling software. I scanned and saved a house that has gone into an institution and the owner is no longer there.

It is good to see the relationship between rooms and life in the house that cannot be seen from the usual point of view.

Enjoy in virtual space

After enjoying a bird's eye view, those who want to have more fun can install the model in a virtual space!
The following is an example where the author and his friends scanned an entire old private house in Tokyo and installed it on a metaverse platform called cluster.

Although there are a few hurdles to overcome, such as model volume reduction and collision processing, the model can be viewed closely even after it has been scanned. Since events can be held inside the model, it is possible to hold meetings where the scanned model can be closely observed later!

↓Clusterlink. Anyone can freely enter, although you need to download an application.

House Theatre | Metaverse Platform cluster

Enjoy watching it behave.

Some of you who read our Tech Blog may want to see how structures here and there behave! 3D scanning makes it possible.

Since we had at hand the data of a mysterious bench that resembles a moai in Makomanai, Sapporo, let's observe the behavior of this structure.

There are many ways to do this, but here is an example using the 3D modeling tool: Rhinoceros.
(1) Scanning data often contains holes and mesh interference, so shrink-wrap is used to clean up the holes and interference. This process produces mesh data with no gaps.

(2) The created mesh is too small and dirty, so QuadRemesh is used to clean up the mesh. At the same time, SubD is created.

(3) ToNURBS is performed on the SubD to convert the object to a polysurface.

(4) Output the created polysurface as a Step file to complete the data that can be used in general-purpose analysis software.

The above is the sequence of steps. Note that even a shrink-wrapped mesh can be ToNURBSed, but as shown in the figure, a fine surface is created. Since mesh cutting of such surfaces with general-purpose analysis software often results in problems, ToNURBS is performed here after SubD conversion.

Now let's check the behavior of the moai bench that you have all been waiting for.

The following are the results of eigenvalue analysis. The moai legs are fixed to the earth and the material is concrete (E=2.1x10⁴N/mm²).

The first-order mode is like a moai nodding its head, the second-order mode is like a moai shaking its head, and the fifth-order mode is like a thin bench seat part vibrating up and down. It is interesting to note that there is also a mode in which the moai expands and contracts, although it is a higher-order mode.

Observing the moai structure based on the mode shape, it seems that the neck, which is the starting point of the primary mode, is flattened. Since the cross-sectional performance differs depending on the direction, this moai image seems to have a nodding mode as the primary mode.

Since this moai is designed for a person to sit on, let's see how much it deforms when a loaded load is applied. The boundary condition is that the bottom of the moai is fixed to the earth side, and the load is a nodal load of 100 kg in total at the center of the bench plus the bench's own weight.

The maximum analytical displacement is 0.03mm, which is 1/33333 < 1/300 for a span of 1000mm, so it can be confirmed that the displacement is within the range of no practical problem. The maximum tensile stress is 0.82 N/mm², which is 0.4 of the concrete cracking stress of Fc21 (2.1 N/mm²), indicating that there is sufficient margin.

What is possible with 3D scanning

What do you think of this author's hobby-oriented tech blog that summarizes the current state of 3D scanning and how to enjoy it?

As the technology becomes more widely available and more people come into contact with it, it will be used in more ways that were not originally envisioned and will broaden the range of expression.

If you are reading this now, what would you like to do with 3D scanning?

The possibilities are as wide as the number of things you can do with it.

Please give it a try!
Translated with www.DeepL.com/Translator (free version)

References

［1] T. Maruyasu, T. Oshima, Y. Komooka, and M. Tsuda. The three dimensional precision measurement using photogrammetric surveying : An example of the measurement of the Daibutsu statue in Kamakura (special issue on construction). The Journal of Production Research, Vol.12, No.6, June 1960, pp.271-275. Institute of Industrial Science, The University of Tokyo. https://repository.dl.itc.u-tokyo.ac.jp/records/23289

［2] Honda SENSING Elite special site: https://www.honda.co.jp/hondasensing-elite/

［3］Cat-type catering robot BellaBot: https://www.elmo.co.jp/product/robot/bellabot/

［4］iPhone15Pro: https://www.apple.com/jp/iphone-15-pro/specs/

［５］Kerbl, Bernhard, et al. "3D Gaussian Splatting for Real-Time Radiance Field Rendering." arXiv, 2023, https://arxiv.org/abs/2308.04079.