[SWIFT] Ground collapse with ARKit + SceneKit
I tried the combination of & portal to texture the virtual object to put the camera capture in place. Let's collapse the ground as a subject.
Complete image
If the captured image of the camera can be pasted on the virtual object in this way, it is possible to make the ground step-like, open the wall and use it as a door, and so on.
How to make the ground collapse
(1) Prepare a box-shaped 3D model (hereinafter, "grid") to be embedded in the ground. A small box (hereinafter referred to as "cell") is spread on the top of the grid in a size of 10x10. Make sure the cells collapse and collect at the bottom of the grid. (2) With ARKit plane recognition enabled, perform a hit test in the center of the screen at the timing of tapping the screen. (3) Obtain the transform of the plane obtained in the hit test and set it in the grid. The grid fits the hit position and posture. ④ Capture the camera image ⑤ Convert the vertices (world coordinates of the four corners) of each cell surface to screen coordinates (normalized device coordinates) ⑥ Since ④ is used as the texture of the cell, the normalized device coordinates of ⑤ are converted to uv coordinates. ⑦ Set ⑥ for the texture coordinates of each cell ⑧ Set the SCN Physics Body from the node of the center cell to the outside so that the 10x10 cell collapses from the center.
The points are described below.
(1) Prepare a box-shaped 3D model (hereinafter, "grid") to be embedded in the ground.
"⑤ Convert the vertices (world coordinates of the four corners) of each cell surface to screen coordinates (normalized device coordinates)" + "Since ⑥④ is used as the cell texture, convert the normalized device coordinates of ⑤ to uv coordinates"
How to use the captured camera image as a texture.
For the texture of each cell, set the captured image as it is as shown in the above figure. The same image is used for all cells, and the uv coordinates (red dots) of the vertices at the four corners of the cell surface are different for each cell. The method of getting the vertex coordinates of cells on the image is the same as normal rendering of a 3D model. Perform world coordinates of each vertex of the cell → view conversion → projection conversion.
//Model transformation matrix (cell surface node in world coordinate system)
let modelTransform = cellFaceNode.simdWorldTransform
//View transformation matrix. Inverse matrix of camera viewpoint (move the vertices of all world coordinates to the position with the camera as the origin)
let viewTransform = cameraNode.simdTransform.inverse
//Projection transformation matrix
let projectionTransform = simd_float4x4(camera.projectionTransform(withViewportSize: self.scnView.bounds.size))
//MVP matrix
let mvpTransform = projectionTransform * viewTransform * modelTransform
//Projection coordinate transformation
var position = matrix_multiply(mvpTransform, SIMD4<Float>(vertex.x, vertex.y, vertex.z, 1.0))
But! , This cannot be used as it is. I found the following article (helped) when I was searching for something that didn't fit.
-Convert vertex coordinates obtained by ARKit's AR Face Geometry to 2D coordinates ・ Introduction to WebGL2 3D Knowledge
The projection coordinate transformation result must be converted to ** "normalized device coordinates" ** from -1.0 to 0.0. Divide x, y by the w component of the model view projection transformation result as follows: ..
// -1.0~1.Normalized to a value of 0. Divide by w to get "normalized device coordinates"
position = position / position.w
Now that we have a coordinate system of -1.0 to 0.0, we can use it as texture coordinates by doing the following.
//Convert to uv coordinates
let texcordX = CGFloat(position.x + 1.0) / 2.0
let texcordY = CGFloat(-position.y + 1.0) / 2.0
texcoords.append(CGPoint(x: texcordX, y: texcordY))
⑦ Set ⑥ for the texture coordinates of each cell
Since it is necessary to set the uv coordinates of each vertex of the cell node arbitrarily, write it in Metal. .. .. I thought, but I could easily realize it only with SceneKit by referring to this article How to make custom geometry with SceneKit + bonus ..
//Generate cell surface geometry
let texcoordSource = SCNGeometrySource(textureCoordinates: texcoords)
let cellFaceGeometry = SCNGeometry(sources: [vertex, texcoordSource], elements: [element])
let cellFaceMaterial = SCNMaterial()
cellFaceMaterial.diffuse.contents = captureImage
cellFaceGeometry.materials = [cellFaceMaterial]
cellFaceNode.geometry = cellFaceGeometry //Replace geometry
By creating a custom geometry with SCNGeometry, you can freely specify the coordinates of the texture.
Since the cell node has already been generated when the application is started this time, the geometry of the cell node is acquired when the screen is tapped, and only the texture coordinates are replaced to create a custom geometry.
Tips
① Debug option This time, it was difficult to understand the existence and delimiter of nodes because there were transparent nodes and cells were spread out. In such a case, if you set the following debug option, a white line will be displayed at the boundary of the node, which is easy to understand and convenient.
//Debug option that can be specified for SCNSceneRenderer
scnView.debugOptions = [.showBoundingBoxes]
(2) If the node is small, the physical judgment will not work.
At first, I started to make the grid size small to 10 cm, but when the cell fell, it was not judged to collide with the node at the bottom, and the phenomenon of slipping through the bottom could not be avoided.
SceneKit has a property called continuousCollisionDetectionThreshold in SCNPhysicsBody as a countermeasure against such slip-through, and it seems that if you specify the size of the conflicting node, it will calculate so that it will not slip through, but it did not work well. Apple Documents says "
Continuous collision detection has a performance cost and works only for spherical physics shapes, but provides more accurate results. ”, So it may not have been possible with Box shapes like this time. As a countermeasure, the bottom and sides of the grid are made into a Box shape to make it thicker (although it still slips through if the grid size is reduced).
Whole source code
ViewController.siwft
import ARKit
import SceneKit
class ViewController: UIViewController {
@IBOutlet weak var scnView: ARSCNView!
private let device = MTLCreateSystemDefaultDevice()!
private let gridSize = 10 //Number of grid divisions
private let gridLength: Float = 0.8 //Grid size[m]
private let wallThickness: Float = 0.1 //Thickness of the side and bottom of the grid
private lazy var cellSize = gridLength / Float(gridSize) //Cell size
private let gridRootNode = SCNNode() //A rectangular parallelepiped buried in the ground
private let gridCellParentNode = SCNNode() //Routes of cells lined up on the surface of the earth
//Cell surface vertex coordinates
private lazy var vertices = [
SCNVector3(-cellSize/2, 0.0, -cellSize/2), //Left back
SCNVector3( cellSize/2, 0.0, -cellSize/2), //Right back
SCNVector3(-cellSize/2, 0.0, cellSize/2), //Left front
SCNVector3( cellSize/2, 0.0, cellSize/2), //Right front
]
//Cell surface vertex index
private let indices: [Int32] = [
0, 2, 1,
1, 2, 3
]
private var time = 0 //Drawing counter
private var isTouching = false //Touch detection
override func viewDidLoad() {
super.viewDidLoad()
//Setting up a grid-like box to bury in the ground
setupGridBox()
//AR Session started
self.scnView.delegate = self
let configuration = ARWorldTrackingConfiguration()
configuration.planeDetection = [.horizontal]
self.scnView.session.run(configuration, options: [.removeExistingAnchors, .resetTracking])
}
}
extension ViewController: ARSCNViewDelegate {
//
//Anchor added
//
func renderer(_: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
guard let planeAnchor = anchor as? ARPlaneAnchor else { return }
//Added planar geometry node
guard let geometory = ARSCNPlaneGeometry(device: self.device) else { return }
geometory.update(from: planeAnchor.geometry)
let planeNode = SCNNode(geometry: geometory)
planeNode.isHidden = true
DispatchQueue.main.async {
node.addChildNode(planeNode)
}
}
//
//Anchor updated
//
func renderer(_: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
guard let planeAnchor = anchor as? ARPlaneAnchor else { return }
DispatchQueue.main.async {
for childNode in node.childNodes {
//Update plane geometry
guard let planeGeometry = childNode.geometry as? ARSCNPlaneGeometry else { continue }
planeGeometry.update(from: planeAnchor.geometry)
break
}
}
}
//
//Called frame by frame
//
func renderer(_ renderer: SCNSceneRenderer, updateAtTime _: TimeInterval) {
if isTouching {
//The screen was touched
isTouching = false
DispatchQueue.main.async {
//Skip if grid is displayed
guard self.gridRootNode.isHidden else { return }
//Capture the camera image and apply it as a texture on the cell surface here
self.setupCellFaceTexture()
//Grid display
self.gridRootNode.isHidden = false
}
}
DispatchQueue.main.async {
//Do nothing if the grid is hidden
guard !self.gridRootNode.isHidden else { return }
//Ground collapse
self.hourakuAnimation()
}
}
}
extension ViewController {
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
guard let _ = touches.first else { return }
isTouching = true
}
private func setupCellFaceTexture() {
//Hit test in the center of the screen
let bounds = self.scnView.bounds
let screenCenter = CGPoint(x: bounds.midX, y: bounds.midY)
let results = self.scnView.hitTest(screenCenter, types: [.existingPlaneUsingGeometry])
guard let existingPlaneUsingGeometryResult = results.first(where: { $0.type == .existingPlaneUsingGeometry }),
let _ = existingPlaneUsingGeometryResult.anchor as? ARPlaneAnchor else {
//There is no plane in the center of the screen, so do nothing
return
}
//Capture the camera image. Easy snapshot because it does not capture continuously()use
let captureImage = self.scnView.snapshot()
//Camera acquisition
guard let cameraNode = self.scnView.pointOfView,
let camera = cameraNode.camera else { return }
//Post the transform of the hit location to the transform of the grid
self.gridRootNode.simdTransform = existingPlaneUsingGeometryResult.worldTransform
for cellNode in self.gridCellParentNode.childNodes {
guard let cellFaceNode = cellNode.childNodes.first(where: {$0.name == "face"}) else { continue }
guard let vertex = cellFaceNode.geometry?.sources.first(where: {$0.semantic == .vertex}) else { continue }
guard let element = cellFaceNode.geometry?.elements.first else { continue }
//Model transformation matrix (cell surface node in world coordinate system)
let modelTransform = cellFaceNode.simdWorldTransform
//View transformation matrix. Inverse matrix of camera viewpoint (move the vertices of all world coordinates to the position with the camera as the origin)
let viewTransform = cameraNode.simdTransform.inverse
//Projection transformation matrix
let projectionTransform = simd_float4x4(camera.projectionTransform(withViewportSize: self.scnView.bounds.size))
//MVP matrix
let mvpTransform = projectionTransform * viewTransform * modelTransform
//Convert each vertex coordinate of the cell to screen coordinates and convert it to UV coordinates
var texcoords: [CGPoint] = []
for vertex in self.vertices {
//Projection coordinate transformation
var position = matrix_multiply(mvpTransform, SIMD4<Float>(vertex.x, vertex.y, vertex.z, 1.0))
// -1.0~1.Normalized to a value of 0. Divide by w to get "normalized device coordinates"
position = position / position.w
//Convert to uv coordinates
let texcordX = CGFloat(position.x + 1.0) / 2.0
let texcordY = CGFloat(-position.y + 1.0) / 2.0
texcoords.append(CGPoint(x: texcordX, y: texcordY))
}
//Generate cell surface geometry
let texcoordSource = SCNGeometrySource(textureCoordinates: texcoords)
let cellFaceGeometry = SCNGeometry(sources: [vertex, texcoordSource], elements: [element])
let cellFaceMaterial = SCNMaterial()
cellFaceMaterial.diffuse.contents = captureImage
cellFaceGeometry.materials = [cellFaceMaterial]
cellFaceNode.geometry = cellFaceGeometry //Replace geometry
}
}
private func hourakuAnimation() {
//Stop the collapse in a certain time
guard self.time < 150 else { return }
self.time += 1
let time = Float(self.time)
let gridSize = Float(self.gridSize)
//Collapse to spread the circle from the center
let x = sin(Float.pi * 2 * time/30.0) * time / 150
let y = cos(Float.pi * 2 * time/30.0) * time / 150
let ygrid = Int((y + 1.0) / 2 * gridSize * gridSize) / self.gridSize * self.gridSize
let xgrid = Int((x + 1.0) / 2 * gridSize) + ygrid
guard 0 <= xgrid, xgrid < self.gridCellParentNode.childNodes.count else { return }
let node = self.gridCellParentNode.childNodes[xgrid]
//Skip nodes for which physics Body has already been set
guard node.physicsBody == nil else { return }
//Match the size of the physical judgment to the size of the cell
let bodyLength = CGFloat(self.cellSize) * 1.0
let box = SCNBox(width: bodyLength, height: bodyLength, length: bodyLength, chamferRadius: 0.0)
let boxShape = SCNPhysicsShape(geometry: box, options: nil)
let boxBody = SCNPhysicsBody(type: .dynamic, shape: boxShape)
boxBody.continuousCollisionDetectionThreshold = 0.001 //I have set it, but it doesn't feel like it's working
// TODO:The origin of the cell is on the center, but the origin of the geometry set in PhysicsBody is in the center, so the cell half coordinates are off.
//You can either customize the PhysicsBody geometry or bring the cell origin to the center, both of which are tedious. Fix it if you have the opportunity.
node.physicsBody = boxBody
}
private func setupGridBox() {
gridRootNode.isHidden = true
//
//Generating cell nodes
//
//Be sure to specify the position of each node. I encountered a phenomenon that the Y coordinate does not meet the expectations unless it is initialized even if it is the center coordinate.
gridCellParentNode.simdPosition = SIMD3<Float>(x: 0.0, y: 0.0, z: 0.0)
gridRootNode.addChildNode(gridCellParentNode)
gridRootNode.simdPosition = SIMD3<Float>(x: 0.0, y: 0.0, z: 0.0)
self.scnView.scene.rootNode.addChildNode(gridRootNode)
//Create a cell node of gridSize x gridSize
let cellFaceGeometry = makeCellFaceGeometry()
let cellBoxGeometry = makeCellBoxGeometry()
let cellLeftBackPos = -(gridLength / 2) + cellSize / 2
for y in 0 ..< gridSize {
for x in 0 ..< gridSize {
//Each cell node
let cellNode = SCNNode()
cellNode.simdPosition = SIMD3<Float>(x: cellLeftBackPos + (cellSize * Float(x)), y: 0, z: cellLeftBackPos + (cellSize * Float(y)))
gridCellParentNode.addChildNode(cellNode)
//Cell surface plane node generation
let cellFaceNode = SCNNode(geometry: cellFaceGeometry)
cellFaceNode.name = "face"
//Determine the coordinates so that each cell is spread out
cellFaceNode.simdPosition = SIMD3<Float>(x: 0.0, y: 0.0, z: 0.0)
cellNode.addChildNode(cellFaceNode)
//Generate a rectangular parallelepiped cell node
let cellBoxNode = SCNNode(geometry: cellBoxGeometry)
cellBoxNode.simdPosition = SIMD3<Float>(x: 0.0, y: -cellSize/2*1.001, z: 0.0)
cellNode.addChildNode(cellBoxNode)
}
}
//
//Grid side node
//
let sideOuterBox = makeGridSideOuterGeometry()
let sideInnerPlane = makeGridSideInnerGeometry()
//Grid side node generation
for i in 0..<4 {
let x = sin(Float.pi / 2 * Float(i)) * gridLength / 2.0
let z = cos(Float.pi / 2 * Float(i)) * gridLength / 2.0
//Side (outside)
let outerNode = SCNNode(geometry: sideOuterBox)
let nodePos = ((gridLength + wallThickness) / 2.0) / (gridLength / 2.0) * 1.001
outerNode.simdPosition = SIMD3<Float>(x: x * nodePos, y: -gridLength/2.0, z: z * nodePos)
outerNode.simdRotation = SIMD4<Float>(x: 0.0, y: 1.0, z: 0.0, w: -Float.pi / 2 * Float(i))
//Make a physical wall on the side
outerNode.physicsBody = SCNPhysicsBody.static()
//The outer wall is (almost) transparent and the rendering order-Set to 1 and draw before other nodes (write to Z buffer first).
//This will not draw nodes that should be drawn behind (almost) transparent, resulting in a visible background.
outerNode.renderingOrder = -1
gridRootNode.addChildNode(outerNode)
//Side (inside)
let innerNode = SCNNode(geometry: sideInnerPlane)
innerNode.simdPosition = SIMD3<Float>(x: x, y: -gridLength/2.0, z: z)
innerNode.simdRotation = SIMD4<Float>(x: 0.0, y: 1.0, z: 0.0, w: -Float.pi / 2 * Float(i))
gridRootNode.addChildNode(innerNode)
}
//
//Bottom node of grid
//
let bottomBox = makeGridButtomGeometry()
let bottomNode = SCNNode(geometry: bottomBox)
bottomNode.simdPosition = SIMD3<Float>(x: 0.0, y: -gridLength+Float(wallThickness), z: 0.0)
bottomNode.simdRotation = SIMD4<Float>(x: 1.0, y: 0.0, z: 0.0, w: -Float.pi / 2)
//Make a physical wall on the bottom
bottomNode.physicsBody = SCNPhysicsBody.static()
gridRootNode.addChildNode(bottomNode)
}
private func makeCellFaceGeometry() -> SCNGeometry {
//The geometry of the cell surface. Paste the screen-captured image on this geometry as a texture
let cellFaceVertices = SCNGeometrySource(vertices: vertices)
let cellFaceIndices = SCNGeometryElement(indices: indices, primitiveType: .triangles)
let cellFaceGeometry = SCNGeometry(sources: [cellFaceVertices], elements: [cellFaceIndices])
let cellFaceMaterial = SCNMaterial()
cellFaceMaterial.diffuse.contents = UIColor.clear
cellFaceGeometry.materials = [cellFaceMaterial]
return cellFaceGeometry
}
private func makeCellBoxGeometry() -> SCNGeometry {
//The small box part of the cell.
let cellBox = SCNBox(width: CGFloat(cellSize), height: CGFloat(cellSize), length: CGFloat(cellSize), chamferRadius: 0.0)
let cellBoxMaterial = SCNMaterial()
cellBoxMaterial.diffuse.contents = UIColor.darkGray
cellBox.materials = [cellBoxMaterial]
return cellBox
}
private func makeGridSideOuterGeometry() -> SCNGeometry {
//The geometry of the sides (outside) of the Grid. Make a thick wall so that it will not penetrate the side when the cell falls.
let sideOuterBox = SCNBox(width: CGFloat(gridLength) * 1.001, height: CGFloat(gridLength), length: CGFloat(wallThickness), chamferRadius: 0)
let sideOuterMaterial = SCNMaterial()
sideOuterMaterial.transparency = 0.001
sideOuterMaterial.diffuse.contents = UIColor.white
sideOuterMaterial.isDoubleSided = true
sideOuterBox.materials = [sideOuterMaterial]
return sideOuterBox
}
private func makeGridSideInnerGeometry() -> SCNGeometry {
//Grid side (inside) geometry
let sideInnerPlane = SCNPlane(width: CGFloat(gridLength), height: CGFloat(gridLength))
let sideInnerMaterial = SCNMaterial()
sideInnerMaterial.diffuse.contents = UIColor.gray
sideInnerMaterial.isDoubleSided = true
sideInnerPlane.materials = [sideInnerMaterial]
return sideInnerPlane
}
private func makeGridButtomGeometry() -> SCNGeometry {
//The geometry of the bottom of the Grid. Make a thick wall so that it will not penetrate the side when the cell falls.
let bottomBox = SCNBox(width: CGFloat(gridLength), height: CGFloat(gridLength), length: CGFloat(wallThickness), chamferRadius: 0)
let bottomMaterial = SCNMaterial()
bottomMaterial.diffuse.contents = UIColor.black
bottomBox.materials = [bottomMaterial]
return bottomBox
}
}
Recommended Posts