Photogrammetry automation with your phone and Autodesk Forge

Autodesk Forge is our collection of Application Programming Interfaces (APIs) that we use to develop our own web services. As part of our Autodesk Forge efforts, we share those APIs (including documentation and code samples) with customers and 3^rd party developers (the Forge community) that want to leverage years’ worth of legacy and current data associated with projects.

Bastien Mazeran is a Premium Support Specialist on our Autodesk Worldwide Field Operations team. When it comes to Forge, we eat our own down food, so Bastien recently shared a project that he has been working on based on the Forge Viewer and Reality Capture APIs.

In a previous article in our series covering the Forge Platform “Get started with Forge using these learning paths”, we discussed the various learning paths available to you to learn the Forge APIs. In this article, we will continue learning by exploring how to create your first Forge mobile app.

The challenge

If you are like me, to truly learn something new you need to set up some real challenges upfront. I never wrote a mobile app before and thought it would be fun to learn how to. Next idea that came to my mind was that I hadn’t played with the Forge Reality Capture API yet, so combining the two together sounded like a great idea:

• Building a mobile app that lets you select images on your smartphone to create a photoscene that can be submitted to the Reality Capture API for processing and open the resulting 3D mesh model in the Forge Viewer.

You can watch this video to see the entire workflow.

If you are in civil engineering, architecture or manufacturing, it is likely that you need to capture existing conditions and store them in a digital form. The Forge Reality Capture API can help with that task. The Forge Reality Capture API covers the photogrammetry workflow and allows for two types of inputs: aerial and object scenes. In this article, we will cover the object scene type and use, as a dataset, images of a lion sculpture near the De Young museum in San Francisco. Those images were taken from a smartphone’s camera. On the other hand, the aerial scene type is best suited when capturing images from drones.

This sounds challenging enough? Well, let’s get started …

Mobile app development

Developing a mobile app means that we need to deal with two primary platforms: Android and iOS. This also means that different IDEs will be needed for each platform: Xcode for iOS and Android Studio.

Fortunately, there is an IDE client called Expo.io that has lots of things set up for you, so it’s quicker to get started and on the right path. Expo lets you create React Native apps that can run in any native environment containing the Expo SDK. If you are not familiar with React Native, please take a moment to review the React Native Express documentation.

With Expo, you don’t need Xcode or Android Studio. You just write JavaScript and it natively has support for both iOS and Android right out of the box. There is also no need to have a separate build process for each OS.

General Considerations

Now that we’ve chosen a development platform with Expo, let’s discuss what we expect from our mobile app.

• It needs to be secure (integrate with Autodesk Single Sign-On)
• Who will use it? What does it do exactly?
• Well-architected
• Do we need a backend?
• Which APIs does it need to manage?
• How will it perform? Remember the network…
• Is it testable?

Securing the mobile app

To fully secure our app, we need to integrate it with Autodesk Single Sign-On. To that end, we will use the 3-legged oAuth handshake. This requires our mobile app to get a callback from the Forge Authentication API. Obviously, configuring the Forge callback to point to a mobile’s phone IP address directly is a bad idea, so we need to use a ‘middleman’ that can associate a mobile device with a Forge callback. The Expo SDK has the right tool for that in the AuthSession library.

Our demo app requests an access token via an ‘Implicit Grant’ directly to the Forge service. This is interesting because with this authentication flow you don’t need to pass the Forge client secret, only the Forge client ID is needed.

The URL also contains a special callback URL, that goes back to the Expo server – our ‘middleman’. The Expo server has a ‘session cookie’ of each iPhone/Android’s IP address and transmits the access-token message. AuthSession library is waiting for this message from the Expo server and finally passes it to our React Native app and we then use it in all future ‘fetch’ requests to the Forge APIs.

Before you publish your mobile app through Expo, you will need to update your Forge App with the correct callback URL. This is found in the Expo XDE client logs window, look for a message similar to “Copy this redirect URL to the Forge app callback:  https://auth.expo.io/@ username>/<your expo app name>”.

The flow

The user’s experience should be kept simple:

• The user is responsible for selecting the images from his smartphone’s camera and once ready she can send those images to the Autodesk Cloud for processing, by a simple click to the Process Photoscene button.
• As the processing completes, the View File button becomes available to signal that the new 3D mesh model can now be viewed.

Based on the user’s experience flow, the requirements translate to:

• The app should first be able to interact with the smartphone’s camera or camera roll to select image files.
• These selected images will be the ones added to the photoscene that the Reality Capture API will use and process to generate a 3D mesh model using photogrammetry techniques.
• Finally, the 3D mesh model is loaded in the Forge Viewer.

The first and last steps can be done local, but the middle step is going to require a lot of network related activities as it needs to send image files to the Autodesk Cloud for processing and wait for the processing to complete before it can retrieve the output 3D mesh model files and prepare them for the viewer.

Going serverless

To ensure our mobile app performs well, I decided to introduce two new Node.js backend apps to perform intensive network related activities:

• The first app takes care of creating, processing and deleting the Reality Capture PhotoScene,
• While the second app is responsible for uploading the PhotoScene output files to the Autodesk Cloud and translating those files to a file format the Forge Viewer can open.

Mobile apps can easily lose internet connectivity and it is, therefore, best to offload network tasks to one or more server components.

Before I go any further, let’s quickly define some of the new concepts I will use throughout the rest of this article.

• Serverless computing: a cloud-computing execution mode in which the cloud provider dynamically manages the allocation of machine resources. Serverless computing still requires servers, but the server management and capacity planning decisions are completely hidden from the developer or operator.
• AWS Lambda Functions: event-driven, serverless computing platform provided by Amazon. It is a computing service that runs code in response to events and automatically manages the computing resources required by that code.
• AWS API Gateway: service provided by Amazon for publishing, maintaining and securing web service APIs.
• Claudia JS: a JavaScript library that makes it easy to deploy, update and destroy AWS Lambda functions.

For that task, I could have chosen a single server backend but decided to go serverless using the latest AWS technology: Lambda Functions. Going serverless is great because there are no servers to manage, and you can run code for virtually any type of backend service – all with zero administration.

To create a lambda function in AWS can be tricky, especially if you have never done this before. To help with initial setup, I decided to use ClaudiaJS, Claudia makes it easy to deploy Node.js projects to AWS lambda and AWS API Gateway. It is a JavaScript tool that automates all the error-prone deployment and configuration tasks and sets everything up the way JavaScript developers expect out of the box.

The AWS Lambda function relies on the Amazon API Gateway service. This service lets you create, publish, maintain, monitor and secure APIs at any scale. You can think of it as a “front door” for applications to access data, business logic, or functionality from your back-end service. API Gateway imposes a 30 seconds system timeout. This is the reason why I decided to have two separate AWS Lambda functions to ensure my back-end services never hit this timeout.

When you deploy your new AWS Lambda Function using ClaudiaJS, Claudia creates a new API Gateway for you. All you need to worry about is writing the server code. Another great benefit of using Claudia is that you can easily push code updates to your Lambda Function through Claudia commands. I’ve used the update command a lot to push fixes to my serverless functions, during initial development.

Serverless does have a draw-back in the sense that it is stateless by design. You cannot store session data in a Lambda Function, therefore you need to store that information elsewhere. You could decide to store it in a S3 bucket, in a database or in a memcached based service. I chose the latter with Redis Cloud for that task.

Putting it all together

Here is a diagram summarizing what we’ve used.

At this point, I am hoping you have a better understanding of the steps required to develop a new mobile app that uses Forge services.

In terms of technology stack, here is what we’ve used so far, notice how I’ve only used JavaScript for both client and server.

Client-side:

• JavaScript
• Expo.io
• Facebook React Native
• Amazon S3
• Autodesk Forge APIs (Authentication, Viewer)

Server-side:

• JavaScript
• Node.JS
• Express JS
• Redis Cloud
• AWS Lambda Functions and API Gateways
• ClaudiaJS
• Autodesk Forge APIs (Authentication, Reality Capture, Data Management, Model Derivative)

If you want to look at the source code and get the full details on how to configure this demo mobile app, please use this link to access the live version and source code.

Next steps

The next step is to tidy up the 3D mesh model by removing elements that come from the background of the photoscene to only leave the lion sculpture.

You can use Autodesk Fusion 360 to import and open the 3D mesh model .OBJ file and use the mesh palette to remove elements of the mesh that are tied to the background of the photoscene. Below is a representation of what I was able to achieve with Fusion 360.

If you want to see a great example of how to bring 3D mesh models from reality capture to Fusion 360, this YouTube video is for you to watch.

What we’ve learned!

There are a lot of technologies available today that you can pick from, but JavaScript appears to be the de-facto standard for mobile app development. JavaScript can be used to develop both the client app and the server apps, this means that as a developer you only need to learn one programming language.

Expo XDE is a great tool to quickly build mobile apps but has its own limitation. The most limiting thing about Expo is that you can't add in your own native modules without detaching and using ExpoKit. This limitation has forced me to migrate a lot of my code functionality to the Lambda functions.

For serverless, I could have used Microsoft Azure or Amazon Lambda and get similar results. The key takeaway here is that serverless frameworks are by definition stateless. This means that if your back-end web service needs to store session information, you will need to rely on a separate stateful service such as Redis. In my demo app, I have used Redis Cloud to store the ID of the photoscene along with other information that relates to it.

Regarding the Forge Reality Capture API, there are a few takeaways:

• The more pictures you add to your photoscene, the better the result will be
• The order in which the images are added to the photoscene does not matter
• The API supports textured mesh models with the .OBJ format
• The 3D mesh model is available through a time-limited HTTPS link that expires 7 days after the date of processing completion
• You should delete your photoscene via the API after the 3D mesh model has been generated

As a parting note, the Forge Viewer works great in mobile apps running on either Android or iOS devices. And of course, wishing you all happy coding with Forge and if you have Forge related questions please ask away in StackOverflow!