SceneAppController

In the previous chapter, we saw how to create a basic application using the bg2 engine. We learned how to connect the main components of the application:

• Canvas: connects the DOM’s Canvas element to the rendering backend.

• AppController: the class that handles application events, initialization, scene redrawing, etc.

• MainLoop: the class that connects the two previous elements and manages the entire lifecycle.

The previous example is very low-level and is designed this way so we can learn how the bg2 engine works internally. However, in general, we won’t be writing an AppController from scratch ourselves; instead, we’ll use a high-level implementation called SceneAppController.

1. Base Project

In the previous example, we started from a predefined project that was already configured to use bg2 engine. In this case, we will start by setting up a base project from scratch.

To get started, you will need to have Node v20 or higher installed and a text editor. Visual Studio Code works very well with TypeScript, but you can use any editor.

1.1. Building and Packaging

We will use Vite for this. The bg2 engine code is plain TypeScript, and you can use any build system as long as you adapt it to the compilation requirements, but Vite is becoming a standard and is one of the most used and easiest to configure systems, so we will focus on this tool during this course.

There are two requirements we need to meet to work with the TypeScript version of bg2 engine:

• The build system must be able to include text files with the .glsl extension, which are the files used to define shaders. This is because the TypeScript version of the graphics engine comes with the plain code, uncompiled, and since import is used internally to add the shader code, we need to configure this feature even if we don’t want to create any shaders manually.

• We need to copy the WebAssembly library for loading 3D models. bg2 engine has its own binary format for loading 3D models implemented in C. It is the same system used for the native version of the graphics engine, so the model import is identical in the C++ and TypeScript APIs. It is necessary to copy the WebAssembly library so it can be accessed at runtime.

What we will do in this section is create a Vite project from scratch that meets these requirements.

1.2. Creating the Vite Project

We start by creating a vanilla Vite project (for now we won’t use any framework, but later in the course we will see how to integrate with React). We will use TypeScript as the language. In the code repository, I have named this project basic_project:

$ npm create vite@latest
> name: `basic_project`
> Framework: Vanilla
> Language: TypeScript

Once the project is created, we will install the vite-plugin-static-copy dependency, which we will use to copy the WebAssembly file, and we will also install bg2e-js:

$ cd basic_project
$ npm install -D vite-plugin-static-copy
$ npm install bg2e-js

Now we will create the Vite configuration file in the root of the project:

vite.config.js

import { defineConfig } from 'vite';
import { viteStaticCopy } from 'vite-plugin-static-copy';

// Path to bg2io package
const bg2ioPath = './node_modules/bg2io/';

// Path to copy the bg2io WebAssembly resources
const bg2ioDst = 'bg2e'

export default defineConfig({
  plugins: [
    viteStaticCopy({
      targets: [
        {
          src: `${bg2ioPath}/bg2io.js`,
          dest: bg2ioDst
        },
        {
          src: `${bg2ioPath}/bg2io.wasm`,
          dest: bg2ioDst
        }
      ]
    })
  ],
  optimizeDeps: {
    esbuildOptions: {
      loader: {
        ".glsl": "text",
      },
    },
  },
  assetsInclude: ["**/*.glsl"]
});

In the file above, we are using viteStaticCopy to copy the bg2io.js and bg2io.wasm files to the bg2e directory in the final distribution folder. It is important to specify where we are going to copy these resources, because we will need to indicate this later when configuring the scene-loading plugin.

Additionally, we are configuring the glsl files to be interpreted as plain text upon import. We do this in the optimizeDeps > esbuildOptions > loader section and also in the assetsInclude section. We have to do this twice because Vite uses two different build systems depending on whether we are compiling for development or production.

With this, we can run the first test: execute npm run build, and verify that a dist directory is generated with, at least, the following contents:

- dist
|- assets > .js and .css files
|- bg2e
|   |- bg2io.js
|   |- bg2io.wasm
|- index.html

1.3. Base Code

First, modify the index.html file. We will include a canvas where we will draw the scene. In the template code generated by Vite, replace the div element with a canvas. We have also added styles to make the canvas occupy the entire page:

<!doctype html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <link rel="icon" type="image/svg+xml" href="/vite.svg" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>basic_project</title>
<style>
      html, body { margin: 0; padding: 0; }
      canvas {
        width: 100vw;
        height: 100vh;
        display: block;
      }
    </style>
  </head>
  <body>
    <canvas id="app"></canvas>
    <script type="module" src="/src/main.ts"></script>
  </body>
</html>

Now delete all the code in the src/main.ts file and replace it with the base code we will use:

import Canvas from "bg2e-js/ts/app/Canvas.ts";
import MainLoop, { FrameUpdate } from "bg2e-js/ts/app/MainLoop.ts";
import SceneAppController from "bg2e-js/ts/render/SceneAppController.ts";
import WebGLRenderer from "bg2e-js/ts/render/webgl/Renderer.ts";

class MyAppController extends SceneAppController {
}

window.onload = async () => {
  const canvasElem = document.getElementById("app") as HTMLCanvasElement;
  if (!canvasElem) {
    console.error("Canvas element not found");
    return;
  }

  const canvas = new Canvas(canvasElem, new WebGLRenderer());
  const appController = new MyAppController();
  const mainLoop = new MainLoop(canvas, appController);
  mainLoop.updateMode = FrameUpdate.MANUAL;
  await mainLoop.run();
}

2. The Scene

The previous code does not work because SceneAppController expects us to configure a base scene that has at least one light source. In this chapter, we will see how to open a scene created with the bg2 Composer scene editor. Later, we will learn how to create and modify scenes manually.

You can get a sample scene from the bg2e-js GitHub repository:

bg2 engine TypeScript on GitHub

Copy the samples/resources/test-scene folder with all its contents into the public directory of your Vite project. The public directory is the folder where we can place static resources that we want to be available at runtime. The structure of the public directory should look like this:

- public
  |- test-scene
     |- country_field_sun.jpg
     |- Plane.bg2
     |- Sphere.bg2
     |- test-scene.vitscnj

2.1. Preparing to Load a Scene

To load a scene, we first need to configure the loader plugin. By default, bg2 engine does not load any plugins, so it can only load native browser resources, which are basically textures and plain text files. In addition to this, the scene component system is modular and can be extended and customized with our own components. Therefore, we also need to configure which components we want to support.

To do all of the above, we need to import the following elements:

import Loader, { registerLoaderPlugin  } from "bg2e-js/ts/db/Loader.ts";
import VitscnjLoaderPlugin from "bg2e-js/ts/db/VitscnjLoaderPlugin.ts";

In the SceneAppController class, we need to implement the loadScene() function. This function is called when the entire rendering system has been initialized, and it must return a valid scene that contains at least one camera and one light. We will add this function and also use it to register the loader plugin and scene components.

class MyAppController extends SceneAppController {
  async loadScene() {

    registerLoaderPlugin(new VitscnjLoaderPlugin({
      bg2ioPath: 'bg2e/'
    }));

    // TODO: Return a valid scene
  }
}

The registerLoaderPlugin function is used to enable loader plugins. In this case, we have registered the default scene loader plugin. Notice how we are passing the path where we have deployed the WebAssembly model loading library to the plugin. If you change the location of this library, this is where you need to indicate the path. We can call registerLoaderPlugin as many times as we want with different loader plugins, but in this example, we will only register this plugin.

2.2. Loading the Scene

The path of the scene you placed in the public folder will be /test-scene/test-scene.vitsnj. Before continuing, you can take a look at the scene file. You will see that it is a JSON structure formed by nodes and components. The scene contains a camera, a light source, and two objects. The camera has a component that allows it to be controlled with the mouse, so you can move it around and see the scene from different angles.

To load a scene, we will use the Loader class that we imported earlier. The loader has several loading functions, each used for a specific purpose. In the case of scenes, we will use the loadNode function. This function loads a node from a file, and since a scene is a node that contains other nodes, we can use this function to load the entire scene. The loadNode function is asynchronous and returns a promise that resolves to the root node of the loaded scene. We will implement the loadScene() function to load the scene and return its root node:

async loadScene() {
  registerLoaderPlugin(new VitscnjLoaderPlugin({
    bg2ioPath: 'bg2e/'
  }));

  const loader = new Loader();
  const sceneRoot = await loader.loadNode("/test-scene/test-scene.vitscnj");

  return sceneRoot;
}

Now we see how the scene is loaded, although it seems that something is not working well: the image looks distorted. We will fix this in the next section.

3. Modifying a Scene

We already have a loaded scene, although it looks distorted if the window is not perfectly square. This is because the scene we loaded contains a basic camera. Given a rendering viewport, we need to specify a series of parameters in the camera to configure the projection. The projection can be configured automatically, but to do this, the camera must have a projection method assigned.

We will configure an optical projection method, where we will specify the size of the camera’s virtual sensor and the characteristics of the optics to determine the projection, and with this ready, you will see how the scene’s aspect ratio adjusts automatically.

To do this, we will need to modify the scene we have loaded.

There are many ways to modify the scene, depending on what we want to do. The more advanced ways require knowledge about the structure of a scene in bg2 engine, but since the main camera is a somewhat special element, there is an easy way to get a reference to it for making modifications.

3.1. Camera Component

We have previously mentioned that bg2 engine scenes are formed by a tree structure of nodes, which in turn are composed of components. Nodes are used to organize the elements of the scene, while components determine the functionality of each node. A node, by default, does nothing, it only serves to group other nodes. If we want a node to have behavior, we will add different components.

A camera is a node that contains a Camera component and a Transform component. Additionally, the main camera must contain a camera handling component if we want it to respond to user actions.

The Camera component has a static function that serves to get the main camera of a scene. We will use this function to get the camera of the scene we have loaded, and then we will make a couple of modifications to that camera.

...
import Camera, { OpticalProjectionStrategy } from "bg2e-js/ts/scene/Camera.ts";
...
async loadScene() {
    ...
    const sceneRoot = await loader.loadNode(...);

    const mainCamera = Camera.GetMain(sceneRoot);
    if (mainCamera) {
      const strategy = new OpticalProjectionStrategy();
      strategy.focalLength = 55;
      strategy.frameSize = 35;
      mainCamera.projectionStrategy = strategy;
    }

    return sceneRoot;
}

We have created an optical projection strategy. In computer graphics, a projection matrix is used to convert coordinates in 3D space into a 2D projection. The projection matrix is a 4x4 matrix such that if we have a vector of the form [x, y, z, w] that defines the position of a point in 3D space, when we multiply that vector by the matrix, we obtain a projection of the point onto a 2D plane. The w coordinate of the vector is included because with 3D coordinates it is necessary to work in affine space: that is, we work in a coordinate space that has an extra w component, which in our vectors will generally have the value 1.0. This allows us to use matrices to perform all kinds of 3D transformations: translations, rotations, scalings, and projections.

By default, the camera only stores its projection matrix. There are methods for constructing various types of projection matrices: orthographic, perspective, warped, etc. In the math package of the bg2 engine, we have the Mat4 class, which contains many utilities for constructing these matrices.

However, we generally won’t be working at such a low level. We’ll use a series of abstractions so we don’t have to generate and multiply matrices manually. In the case of the camera, projection strategies are used to automatically generate the projection matrix from values that are easier to interpret.

In this example, we have created an OpticalProjectionStrategy, which generates a perspective projection matrix based on physical measurements. Here, we are specifying the camera sensor size (35mm) and the focal length (55mm).

After configuring the projection, we assign it to the camera using the mainCamera.projectionStrategy setter. From this point on, the camera’s projection will automatically adjust whenever any scene parameter it depends on is modified. For this reason, the scene now appears properly proportioned, and if we resize the window, the view is regenerated with the correct aspect ratio.

3.2. Camera Handling

The position and orientation of the camera is set using a Transform component. This type of component is used to modify the position and orientation of a scene node. By default, a scene node is always at position (0, 0, 0). The Transform component adds a transformation matrix that affects the node it belongs to, as well as all child nodes of that node.

To define the position of the camera, therefore, you would need to modify the transformation matrix of the Transform component that belongs to the camera node.

A camera handler is a component that captures mouse events to automatically modify the Transform component’s matrix. For this to work, the camera node must contain three components:

• The Camera component

• A Transform component

• A camera handler component.

In the scene we loaded, there was already a predefined OrbitCameraComponent, so the scene already supported camera handling after loading. What we are going to do now is remove that camera handler and add another one that works the same way, but adds motion interpolation between each input event to achieve smoother movement. This will also give us an introduction to component handling.

Accessing the camera node: we can access the camera’s scene node through the Camera component, using the node getter, which is defined for all scene components.

Getting a component from a node: the component(componentId: string) function of the Node class allows you to get a component from a node by its identifier. The component identifier is a string that usually matches the class name. To know the identifier of a component, we can consult its documentation. We will see more about this when we learn to create our own components.

Removing a component: the removeComponent(component: string | Component) function allows you to remove a component from a node. You can specify either the component instance or its identifier.

Adding a component: the addComponent(component: Component) function adds a component to a node.

It is important to understand that a node can only have one component of each type. For example, if we add a Transform component to a node that already has one, the existing component will be removed before adding the new one.

Let’s modify the loadScene() function to replace the OrbitCameraController component with SmoothOrbitCameraController. In this case, we have two components that do the same thing, so it is important to remove the component we don’t want first.

...
import OrbitCameraController from "bg2e-js/ts/scene/OrbitCameraController.ts";
import SmoothOrbitCameraController from "bg2e-js/ts/scene/SmoothOrbitCameraController.ts";
...

...
if (mainCamera) {
  const strategy = ...
  ...

  const cameraNode = mainCamera.node!;
  const controller = cameraNode.component("OrbitCameraController") as OrbitCameraController;
  if (controller) {
    cameraNode.removeComponent(controller);
  }
}

With the code above, we’re removing the previous camera controller, so you’ll notice that you can no longer control the camera. In reality, the code above can be used to retrieve a component either because we want to make sure it exists, or because we want to modify one of its properties. But if we just want to remove it, we can use the removeComponent() function by passing the component ID. If we ask to remove a component that doesn’t exist, the function won’t fail or throw an error:

const cameraNode = mainCamera.node!;
cameraNode.removeComponent("OrbitCameraController");

Once we’ve removed the component we don’t want, we create the new one and add it:

const smoothController = new SmoothOrbitCameraController();
cameraNode.addComponent(smoothController);

Now we see a couple of things that are not right. The first is that if we move the mouse over the scene, the camera suddenly changes position. This happens because we have configured the scene with on-demand rendering. The SmoothOrbitCameraController component adds interpolation to the camera movement, so as long as there are no user inputs, the camera will remain fixed, even if the controller is configured with a different initial position.

Secondly, the camera is very close to the object. This is because we created the camera controller with its default values.

To solve the first problem, we just need to call the postRedisplay() function of mainLoop. All application controllers have a reference to the main loop, so we can access it directly from the loadScene() function we are implementing:

this.mainLoop.postRedisplay();

To solve the second problem, we just need to configure the camera controller with values that better suit what we want.

smoothController.distance = 15;
smoothController.rotation.x = 20;
smoothController.rotation.y = 45;

Another option would be to copy the configuration values from the OrbitCameraController.

• Get the OrbitCameraController before clearing it.

• Assign all the configuration values from the old controller to the new one. The SmoothOrbitCameraController class extends OrbitCameraController, so it shares the same configuration attributes.

But we can improve the second step: instead of copying the controller’s properties one by one, we can use the assign() function to do it for us. All components must implement an assign() function that is used to copy configuration values from another component. It is generally used to copy between components of the same type, but in certain cases we can also use it for different components. In this case, since the SmoothOrbitCameraController component extends OrbitCameraController and does not define any new properties, we can use this function to copy the configuration values in a single step:

const cameraNode = mainCamera.node!;
const orbitController = cameraNode.component("OrbitCameraController") as OrbitCameraController;
const smoothController = new SmoothOrbitCameraController();
if (orbitController) {
  cameraNode.removeComponent("OrbitCameraController");
  smoothController.assign(orbitController as SmoothOrbitCameraController);
}
else {
  smoothController.distance = 15;
  smoothController.rotation.x = 20;
  smoothController.rotation.y = 45;
}
cameraNode.addComponent(smoothController);

this.mainLoop.postRedisplay();

Now there’s just one thing left: when we move the camera, releasing the mouse button causes the animation to stop abruptly, and it only continues to play if we keep generating input events (for example, by moving the mouse). This happens because when we set rendering to on-demand mode, the window doesn’t update unless we explicitly tell it to by calling postRedisplay(). By default, the postRedisplay() function adds 10 frames to the rendering queue each time we call it. However, if we don’t make the call explicitly, the scene will only be redrawn once per input event: when we stop moving the mouse, the animation stops because the scene isn’t being redrawn.

To fix this, the SceneAppController class has a property that we can configure to queue 60 window update requests each time an input event occurs. It’s important to note that the requested updates are not cumulative: that is, if two events occur, we won’t have 120 frames to update; we’ll only have the 60 frames from the last event.

You can change this property in the initialization function itself, although in reality the value of this property can be modified at any time, not just during initialization:

async loadScene() {
  ...
  this.updateOnInputEvents = true;
  this.updateInputEventsFrameCount = 120;
  return sceneRoot;
}

Si se diera el caso de que con los 60 fotogramas de actualización no es suficiente o es demasiado, es posible modificar el número de fotogramas que queremos actualizar después del último evento de entrada con la propiedad updateInputEventsFrameCount:

async loadScene() {
  ...
  this.updateOnInputEvents = true;
  this.updateInputEventsFrameCount = 120;
  return sceneRoot;
}

4. Other functions of SceneAppController

The SceneAppController class has some functions that can be useful. Here we will list them, although we will not explain them in detail.

• async registerLoaders(): Promise<void>: This function can be used to register loaders and components, in the same way we did in the loadScene() function, only that using this function we have the logic of registering plugins and components separate from the logic of loading the scene.

• async loadEnvironment() : Promise<Environment | null>: We can return a configured environment in this function that will be used only if the scene loaded with loadScene does not include an environment.

• async loadDone(): Promise<void>: This function is called when the engine loading is finished. It is important to note that the code in this function is not equivalent to placing the same code at the end of loadScene(): when this function is called, extra initialization tasks have been performed.

In addition to the accessors updateOnInputEvents and updateInputEventsFrameCount, we have other attributes that are used to control and configure the element selection system in the scene:

• selectionHighlightEnabled (read/write): enables or disables the highlighting of selected elements in the scene. This value can be modified throughout the application’s lifecycle.

• selectionManagerEnabled (read/write): enables or disables the selection system. This value can be modified throughout the application’s lifecycle.

• selectionManager (read): returns the SelectionManager object, which we can use to configure the behavior of the selection system with canvas clicks, register selection callbacks, etc.

• selectionHighlight (read): returns the SelectionHighlight object, which we can use to configure the visual appearance of element selection.

The scene element selection system will be covered later, but it basically depends on its configuration through SceneAppController and the activation of selectable elements in the scene.