10th Aug 2020

AR with Unity !? | Image Recognition Using AR Foundation

AR, Game Development

Written By, Praharsh Khatri

Introduction to AR Foundation :

Augmented Reality can be used in Unity through AR Foundation. This interface makes Unity developer work easy. To use this package you also need some plugins which are mentioned below.

AR Foundation includes core features from ARKit, ARCore, Magic Leap, and HoloLens, as well as unique Unity features to build robust apps that are ready to ship to internal stakeholders or on any app store. This framework enables you to take advantage of all of these features in a unified workflow.

AR Foundation lets you take currently unavailable features with you when you switch between AR platforms.

The AR-related subsystems are defined in the AR Subsystems package. These APIs are in the UnityEngine.Experimental.XR namespace, and consist of a number of Subsystems.

AR Foundation uses Monobehaviours and APIs to deal with devices that support the following concepts :

  1. Device tracking: track the device’s position and orientation in physical space.
  2. Plane detection: detect horizontal and vertical surfaces.
  3. Point clouds, also known as feature points.
  4. Anchor: an arbitrary position and orientation that the device tracks.
  5. Light estimation: estimates for average color temperature and brightness in physical space.
  6. Environment probe: a means for generating a cube map to represent a particular area of the physical environment.
  7. Face tracking: detect and track human faces.
  8. 2D image tracking: detect and track 2D images.
  9. 3D object tracking: detect 3D objects.
  10. Meshing: generate triangle meshes that correspond to the physical space.
  11. Body tracking: 2D and 3D representations of humans recognized in physical space.
  12. Collaborative participants: track the position and orientation of other devices in a shared AR experience.
  13. Human segmentation and occlusion: apply distance to objects in the physical world to rendered 3D content, which achieves a realistic blending of physical and virtual objects.
  14. Raycast: queries physical surroundings for detected planes and feature points.
  15. Pass-through video: optimized rendering of mobile camera image onto the touch screen as the background for AR content.
  16. Session management: manipulation of the platform-level configuration is automatically when AR Features are enabled or disabled.

(Above concepts reference are taken from : https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@4.0/manual/index.html)

Feature Support Per Platform

 ARCoreARKitMagic LeapHoloLens
Device tracking
Plane tracking
Point clouds
Environment probes
Anchors
Face tracking
Raycast
Light estimation
Human segmentation and occlusion
Session management
Pass-through video
Collaborative participants
Meshing
3D Object tracking
2D & 3D body tracking
2D Image tracking

Prerequisites

  • Unity 2019 version or above.
  • Android SDK level 24 and above.
  • Android device which has Android 7.0 and above.

ARCore XR plugin

Currently, com.unity.xr.arcore@4.0 preview is available.
By adding this plugin it enables the use of ARCore support via unity multi-platform XR API
Supported Features are,

  1. Background Rendering
  2. Horizontal Planes
  3. Depth Data
  4. Anchors
  5. Hit Testing

For more current version information visit: https://docs.unity3d.com/Packages/com.unity.xr.arcore@4.0/manual/index.html

ARKit XR plugin

Currently, com.unity.xr.arkit@4.0 preview is available.
By adding this plugin it enables the use of ARKit support via unity multi-platform XR API
Supported Features are,

  1. Efficient Background Rendering
  2. Horizontal Planes
  3. Depth Data
  4. Anchors
  5. Hit Testing
  6. Face Tracking
  7. Environment Probes

For more current version information visit: https://docs.unity3d.com/Packages/com.unity.xr.arkit@4.0/manual/index.html

Image Recognition Using AR Foundation

Setup

To install AR Foundation you need to go window -> package manager, Press The Advance Button, and select show preview packages after that you will need to install AR Foundation version 2.2.0-preview.06, We also require to download the ARCore XR Plugin, ARKit XR Plugin the same way the version should be,

ARCore XR Plugin(For Android) : 2.2.0-preview.06
ARKit XR Plugin(For IOS) : 2.2.0-preview.06
ar-foundation
Note: make sure the SDK you are using are updated beforehand.
(The above-mentioned version are working for me that why I used these versions.)

Player Settings

In this player, the setting adds the company name and package name you want to give.
Then enable AutoGraphic API.
Then make Minimum API level to 24 and Target API level to highest.
ar-foundation-player-settings

Hierarchy

Add AR Session, AR Session Origin from pressing right-click on hierarchy go in XR tab and add them.
ar-foundation-hierarchy

Then inside AR Session origin is AR camera give it the tag of the Main Camera.

Inspector

Click on AR Session origin and add AR Tracked Image Manager component
ar-foundation-inspector

We now need a Serialized library and image prefab.

Reference Image Library

Go in project tab Press right click Create – >XR -> ReferenceImageLibrary.
This is a Serialized object, so we need to populate it.

ar-foundation-reference-image-library
After adding the data, give reference to the AR Tracked Image Manager component.

Prefab

Create a cube in the scene adjust it and drag it to the project tab and give reference to the AR Tracked Image Manager component.

Script

Create a script for recognition of image and what you want to do with it through AR Tracked Image Manager component

Add the code written in the script above.

Build

Press CTRL + B to build your app, make sure you add the scene.

Now it’s completed you can open the image which you add in the reference library and you can see the prefab you added AR Tracked Image Manager component

Video

3d Car Model : https://poly.google.com/view/7bF7UVAoYRG(Kris Tong).

Conclusion

Hence, Image Recognition is one of the features which are helpful in AR Related Applications, and hence it will be very useful in a Games.

This was just a basic setup to start working with AR Foundation in unity. There are many more features available for user-interactive gaming. Image Recognition is an example of engaging gaming experience.

This tutorial can be used to work with any game or application.

Written By,

Unity Developer at Yudiz Solutions Pvt. Ltd