How To Render Static 3D Models Using OpenGL ES

Vuforia uses OpenGL ES to render 3D content. On the Android and iOS platforms, the Vuforia native SDK samples are used to show how to render simple static (non-animated) models using OpenGL ES 2.0.

Code that renders 3D models in the OpenGL rendering thread

The sample code performs the rendering of the 3D models within the OpenGL rendering thread.

Android Java

In the Android Java samples, such as VuforiaSamples-x-y-z, the relevant code is located in the renderFrame()method of the GLSurfaceView.Renderer interface. The renderFrame()method is called at every frame and can be used to update the OpenGL view. For instance, for the Image Targets sample package, this code is located in ImageTargetsRenderer.java.

Android C++

Similarly, in the Android C++ native samples, the rendering code is still triggered from the Java renderFrame() method, but the actual OpenGL code is in a native C++ function. For instance, in the ImageTargetsNative sample, the renderFrame()C++ code is located in the ImageTargets.cpp file in the /jni folder.

iOS

In the iOS ImageTargets sample, the rendering code can be found in the renderFrameQCAR function in ImageTargetsEAGLView.mm.

Within the frame-rendering function, the sample code performs the following main operations:
  • Retrieves a list of TrackableResult objects
    • The objects represent the targets that are currently active, that is, the targets being tracked by Vuforia.
  • Retrieves the Pose matrix for each target.
    • The pose represents the position and orientation of the target with respect to the camera reference frame.
  • Uses the pose matrix as a starting point to build the OpenGL model-view matrix.
    • This matrix is used for positioning and rendering the 3D geometry (mesh) associated with a given target.
    • The model-view matrix is built when you apply additional transformations to translate, rotate, and scale the 3D model with respect to the target pose (that is, with regard to the target reference frame).
  • Multiplies the model-view matrix by the camera projection matrix to create the MVP (model-view-projection) matrix that brings the 3D content to the screen.
    • Later in the code, the MVP matrix is injected as a uniform variable into the GLSL shader used for rendering.
    • Within the shader, each vertex of our 3D model is multiplied by this matrix, effectively bringing that vertex from world space to screen space (the transforms are actually object > world > eye > window).

Constructing the MVP matrix

The following code snippets show the part of the sample code that builds the MVP matrix from the trackable Pose (for each trackable).

Android/iOS – C++

for(int tIdx = 0; tIdx < state.getNumTrackableResults(); tIdx++)
{
  // Get the trackable:
  const QCAR::TrackableResult* result = state.getTrackableResult(tIdx);
  const QCAR::Trackable& trackable = result->getTrackable();
  QCAR::Matrix44F modelViewMatrix =
  QCAR::Tool::convertPose2GLMatrix(result->getPose());
  ...
  QCAR::Matrix44F modelViewProjection;
  SampleUtils::translatePoseMatrix(0.0f, 0.0f, kObjectScale,
      &modelViewMatrix.data[0]);
  SampleUtils::scalePoseMatrix(kObjectScale, kObjectScale, kObjectScale,
      &modelViewMatrix.data[0]);
  SampleUtils::multiplyMatrix(&projectionMatrix.data[0],
      &modelViewMatrix.data[0] ,
      &modelViewProjection.data[0]);
  ...
  glUniformMatrix4fv(mvpMatrixHandle, 1, GL_FALSE,
  (GLfloat*)&modelViewProjection.data[0] );
  ...
   // render your OpenGL mesh here (e.g. vertex array)

Android – Java

for (int tIdx = 0; tIdx < state.getNumTrackableResults(); tIdx++)
{
  TrackableResult result = state.getTrackableResult(tIdx);
  Trackable trackable = result.getTrackable();
  ImageTarget itarget = (ImageTarget)trackable;
  Matrix44F modelViewMatrix_Vuforia =
  Tool.convertPose2GLMatrix(result.getPose());
  float[] modelViewMatrix = modelViewMatrix_Vuforia.getData();
  float[] modelViewProjection = new float[16];
  Matrix.translateM(modelViewMatrix, 0, 0.0f, 0.0f,
      OBJECT_SCALE_FLOAT);
  Matrix.scaleM(modelViewMatrix, 0, OBJECT_SCALE_FLOAT,
      OBJECT_SCALE_FLOAT, OBJECT_SCALE_FLOAT);
  Matrix.translateM(modelViewMatrix, 0, 0.0f, 0.0f, 0.0f);
  Vec2F targetSize = itarget.getSize();
  Matrix.scaleM(modelViewMatrix, 0,
      targetSize.getData()[0], targetSize.getData()[1], 1.0f);
  Matrix.multiplyMM(modelViewProjection, 0, vuforiaAppSession
      .getProjectionMatrix().getData(), 0, modelViewMatrix, 0);
  ...
  // render your OpenGL mesh here ... 

For more details on positioning 3D content on the target, see the article How To Scale and Position Content Using OpenGL

Code that feeds model mesh vertex arrays to the rendering pipeline

The model mesh vertex arrays (vertices, normals, and texture coordinates) are fed to the OpenGL rendering pipeline. This task is achieved by doing the following:
  1. Bind the shader.
  2. Assign the vertex arrays to the attribute fields in the shader.
  3. Enable these attribute arrays.

Android/iOS – C++

glUseProgram(shaderProgramID);
glVertexAttribPointer(vertexHandle, 3, GL_FLOAT, GL_FALSE, 0,
    (const GLvoid*) &teapotVertices[0]);
glVertexAttribPointer(normalHandle, 3, GL_FLOAT, GL_FALSE, 0,
    (const GLvoid*) &teapotNormals[0]);
glVertexAttribPointer(textureCoordHandle, 2, GL_FLOAT, GL_FALSE, 0,
    (const GLvoid*) &teapotTexCoords[0]);
glEnableVertexAttribArray(vertexHandle);
glEnableVertexAttribArray(normalHandle);
glEnableVertexAttribArray(textureCoordHandle);

Android – Java

GLES20.glVertexAttribPointer(vertexHandle, 3, GLES20.GL_FLOAT,
    false, 0, mTeapot.getVertices());
GLES20.glVertexAttribPointer(normalHandle, 3, GLES20.GL_FLOAT,
    false, 0, mTeapot.getNormals());
GLES20.glVertexAttribPointer(textureCoordHandle, 2,
    GLES20.GL_FLOAT, false, 0, mTeapot.getTexCoords());
GLES20.glEnableVertexAttribArray(vertexHandle);
GLES20.glEnableVertexAttribArray(normalHandle);
GLES20.glEnableVertexAttribArray(textureCoordHandle);

Be aware that the Vuforia samples do not make use of the normal array. But you can choose to do lighting calculations in the shader using the normals. To use a model without normals, you can comment out those lines.

The sample code activates the correct texture unit, binds the desired texture; and then renders the model (mesh) using the glDrawElements call. The correct texture unit is typically 0, unless you have multiple textures.

Android/iOS – C++

glActiveTexture(GL_TEXTURE0);
glBindTexture(GL_TEXTURE_2D, thisTexture->mTextureID);
...
glDrawElements(GL_TRIANGLES, NUM_TEAPOT_OBJECT_INDEX, GL_UNSIGNED_SHORT,
    (const GLvoid*) &teapotIndices[0]);

Android – Java:
GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D,
    mTextures.get(textureIndex).mTextureID[0]);
GLES20.glUniform1i(texSampler2DHandle, 0);
// pass the model view matrix to the shader
GLES20.glUniformMatrix4fv(mvpMatrixHandle, 1, false,
    modelViewProjection, 0);
GLES20.glDrawElements(GLES20.GL_TRIANGLES,
    mTeapot.getNumObjectIndex(), GLES20.GL_UNSIGNED_SHORT,
    mTeapot.getIndices());

Android – Java

GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D,
    mTextures.get(textureIndex).mTextureID[0]);
GLES20.glUniform1i(texSampler2DHandle, 0);
// pass the model view matrix to the shader
GLES20.glUniformMatrix4fv(mvpMatrixHandle, 1, false,
    modelViewProjection, 0);
GLES20.glDrawElements(GLES20.GL_TRIANGLES,
    mTeapot.getNumObjectIndex(), GLES20.GL_UNSIGNED_SHORT,
    mTeapot.getIndices());

Note that glDrawElements takes an array of indices. This allows you to randomly index into the other model arrays when building the triangles that make up the model. However, sometimes you have a model without indices. This situation means that the model arrays are meant to be read linearly, that is, the triangles are listed in consecutive order. In that case, use the glDrawArrays method instead:
glDrawArrays(GL_TRIANGLES, 0, numVertices);

Replace the teapot model

To change out the teapot model with your own model, follow these steps:
  1. Obtain the vertex array for your model, and optionally texture coordinates and normals.
    Our samples use a simple header format to store the model data as arrays. For an example, see Teapot.h (or Teapot.java in the Java version of the Vuforia samples).
  2. When you have the arrays, you must switch out the following variables in the above code:
    • teapotVertices
    • teapotNormals
    • teapotTexCoords
    • teapotIndices
  3. Change the NUM_TEAPOT_OBJECT_INDEX, and the GL_UNSIGNED_SHORT type in the glDrawElements call if your model uses an index array with a different type. If your model does not have indices, use glDrawArrays, as described above.

Example

For example, replace the sample model with a custom mesh that represents a simple rectangular surface (aka a quad). The quad (rectangle) mesh consists of two triangles.
  1. In Android – Java, define the triangles using code similar to the following:
public class QuadMesh {
private final float[] mVertices = new float[] {
-0.5f, -0.5f, 0.0f, //bottom-left corner
0.5f, -0.5f, 0.0f, //bottom-right corner
0.5f, 0.5f, 0.0f, //top-right corner
-0.5f, 0.5f, 0.0f //top-left corner
};
private final float[] mNormals = new float[] {
0.0f, 0.0f, 1.0f, //normal at bottom-left corner
0.0f, 0.0f, 1.0f, //normal at bottom-right corner
0.0f, 0.0f, 1.0f, //normal at top-right corner
0.0f, 0.0f, 1.0f //normal at top-left corner
};
private final float[] mTexCoords = new float[] {
0.0f, 0.0f, //tex-coords at bottom-left corner
1.0f, 0.0f, //tex-coords at bottom-right corner
1.0f, 1.0f, //tex-coords at top-right corner
0.0f, 1.0f //tex-coords at top-left corner
};
private final short[] mIndices = new short[] {
0,1,2, //triangle 1
2,3,0 // triangle 2
};
public final Buffer vertices;
public final Buffer normals;
public final Buffer texCoords;
public final Buffer indices;
public final int numVertices = mVertices.length/3; //3 coords per vertex
public final int numIndices = mIndices.length;
public QuadMesh() {
// init vertex buffers
vertices = fillBuffer(mVertices);
normals = fillBuffer(mNormals);
texCoords = fillBuffer(mTexCoords);
indices = fillBuffer(mIndices);
}
private Buffer fillBuffer(float[] array) {
final int sizeOfFloat = 4;//1 float = 4 bytes
ByteBuffer bb = ByteBuffer.allocateDirect(sizeOfFloat * array.length);
bb.order(ByteOrder.LITTLE_ENDIAN);
for (float f : array) {
bb.putFloat(f);
}
bb.rewind();
return bb;
}
private Buffer fillBuffer(short[] array) {
final int sizeOfShort = 2;//1 short = 2 bytes
ByteBuffer bb = ByteBuffer.allocateDirect(sizeOfShort * array.length);
bb.order(ByteOrder.LITTLE_ENDIAN);
for (short s : array) {
bb.putShort(s);
}
bb.rewind();
return bb;
}
}
  1. In C++ (Android or iOS), create a file called QuadMesh.h, and fill the file with the following vertex arrays definitions:
static const float quadVertices[] =
{
-0.5f, -0.5f, 0.0f, //bottom-left corner
0.5f, -0.5f, 0.0f, //bottom-right corner
0.5f, 0.5f, 0.0f, //top-right corner
-0.5f, 0.5f, 0.0f //top-left corner
};
static const float quadTexcoords[] ={
0.0f, 0.0f, //tex-coords at bottom-left corner
1.0f, 0.0f, //tex-coords at bottom-right corner
1.0f, 1.0f, //tex-coords at top-right corner
0.0f, 1.0f //tex-coords at top-left corner
};
static const float quadNormals[] =
{
0.0f, 0.0f, 1.0f, //normal at bottom-left corner
0.0f, 0.0f, 1.0f, //normal at bottom-right corner
0.0f, 0.0f, 1.0f, //normal at top-right corner
0.0f, 0.0f, 1.0f //normal at top-left corner
};
static const unsigned short quadIndices[] =
{
0, 1, 2, // triangle 1
2, 3, 0 // triangle 2
};
  1. Using either Java or C++, pass the quad vertices, texture coordinates and normals as parameters to the glVertexAttribPointer()functions as previously explained, while passing the quad indices to the glDrawElements() function.
  2. Finally, change the texture.