Graphics hardware acceleration

Graphics rendering is generally divided into hardware-accelerated rendering and software rendering. In hardware-accelerated rendering, the rendering is performed by a dedicated graphics accelerator (Graphics Processing Unit, GPU), while in software rendering the rendering is performed by the device main processor (Central Processing Unit, CPU). Switching between GPU and CPU rendering involves synchronizing pixel data between these two processing units, which in practice means pixel data transfer.

Since Symbian^3, transferring data from the GPU to CPU or the other way around degrades performance, since the processing units do not share the same memory. Avoid data transfers between the GPU and CPU if possible.

Figure: Graphics memory architecture

Graphics rendering for Java APIs in Symbian^3

Symbian^3 onwards rendering for Java APIs is performed in the following ways:

  • LCDUI

  • eSWT

  • Scalable 2D Vector Graphics

  • Mobile 3D Graphics

  • Nokia UI API

LCDUI

LCDUI Graphics implementation is based on software rendering (CPU). All graphics handled in the rendering commands (draw, fill, and copy methods of Graphics) are rasterized to a bitmap in CPU memory. This applies to all targets of Graphics, namely Canvas, GameCanvas, Image, and CustomItem. At the end of a Canvas, GameCanvas, or CustomItem paint cycle, the bitmap is transferred from CPU to GPU.

It is faster for a MIDlet to use the drawRGB method once and pass a larger RGB data array than for it to call the method several times, passing a smaller data array.

eSWT

GC (org.eclipse.swt.graphics.GC) drawing to eSWT Controls is hardware-accelerated. GC drawing to eSWT Images uses software rendering.

In case an eSWT application needs to intensively draw Images to Controls, it is recommended that it uses immutable Images, since they provide better performance than mutable Images. Image pixel data is stored in the CPU memory, and is transferred and cached to the GPU memory when Image is drawn for the first time. Pixel data transfer is avoided on redraw if Image has not been modified. Modifying the Image forces pixel data transfer to GPU again on redraw.

Scalable 2D Vector Graphics

Scalable 2D Vector Graphics API (M2G) uses software rendering. For all targets (Graphics of Canvas, GameCanvas, Image, and CustomItem) M2G content is rasterized to a bitmap in the CPU memory.

Mobile 3D Graphics

The Mobile 3D Graphics API (M3G) is fully on hardware accelerated and uses hardware-accelerated rendering surfaces instead of bitmaps. Due to the fact that Graphics and M3G use different types of rendering targets, the LCDUI Canvas implementation supports a hardware-accelerated rendering surface when M3G is used for rendering on Canvas (see "2D and 3D mixed" in the following figure). The main motivation is to minimize the need for pixel data transfer between CPU and GPU, and to allow a pure M3G use case to achieve optimal performance. The rendering target type, hardware-accelerated surface or bitmap, is automatically selected based on the APIs (Graphics and/or M3G) used by the MIDlet when painting on Canvas. The following figure illustrates these scenarios.

Figure: Rendering scenarios

It is visible from the figure that best performance is achieved using only Graphics or M3G. Using only M3G gives full hardware acceleration benefits and is the most optimal use case. For more information, see the following examples on how a game scene background can be rendered efficiently with using only M3G.

Clearing background with M3G

When a MIDlet renders M3G content, use Background for the following purposes to ensure maximum performance:

  • Clearing the Canvas area

  • Drawing a 2D image as a background for M3G content

Avoid using Graphics methods for the use cases, since they result in decreased performance. The following code snippet shows the wrong and correct way for clearing the Canvas area. 

Graphics3D myG3D = Graphics3D.getInstance();
//
// Wrong: using Graphics methods drops performance 
//  
protected void paint(Graphics g) { 
    // Clear Canvas with Graphics.fillRect(), 
    // poor performance in 2D + M3G mixing case 
    g.setColor(0x00000000); 
    g.fillRect(0, 0, getWidth(), getHeight()); 

    myG3D.bindTarget(g); 
    // ... render the scene ... 
    myG3D.releaseTarget(); 
}
 
// 
// Ok: M3G Background used for clearing Canvas 
// 
Background myBackground = new Background(); 
myBackground.setColor(0x00000000); 
myBackground.setColorClearEnabled(true); 

protected void paint(Graphics g) { 
    myG3D.bindTarget(g); 
    // Clear Canvas with Background object, 
    // no performance penalty 
    myG3D.clear(myBackground); 
    // ... render the scene ... 
    myG3D.releaseTarget(); 
}

The following code snippet shows the wrong and correct way for drawing a 2D background image. 

Graphics3D myG3D = Graphics3D.getInstance();
//
// Wrong: using Graphics methods drops performance 
//  
Image myImage = Image.createImage("/image.png"); 
protected void paint(Graphics g) { 
    g.drawImage(myImage, 0, 0, Graphics.TOP | Graphics.LEFT); 
    
    myG3D.bindTarget(g); 
    // ... render the scene ... 
    myG3D.releaseTarget(); 
}

// 
// Ok: M3G Background used 
// 
Image myImage = Image.createImage("/image.png"); 
Image2D myImage2D = new Image2D ( Image2D.RGB, myImage); 
Background myBackground = new Background(); 
myBackground.setImage(myImage2D); 

protected void paint(Graphics g) { 
    myG3D.bindTarget(g); 
    myG3D.clear(myBackground); 
    // ... render the scene ... 
    myG3D.releaseTarget(); 
}

Optimizing Graphics and M3G mixing

3D Java games commonly mix M3G and Graphics rendering. A typical use case is to render a game scene with M3G and use Graphics content on top of the scene. For example, the content drawn with Graphics can include game score and status information in upper part of the Canvas and Command labels at the bottom of Canvas. For this kind of a use case the Canvas (and GameCanvas) implementation optimizes the quantity of pixels drawn with Graphics that need to be transferred from a bitmap to a hardware-accelerated surface.

Canvas implementation internally uses two rectangular areas to identify which parts of the Canvas surface need to be updated with Graphics drawn content. The first rectangle is dynamically resized so that it is the smallest rectangular area that covers all Graphics rendering in upper half of Canvas. The second rectangle is dynamically resized so that it is the smallest rectangular area that covers all Graphics rendering in lower half of Canvas.

The red boxes in the following figures show how Canvas resizes its rectangles in mixed M3G and Graphics rendering. In the Figure: Update rectangles the MIDlet first renders M3G to the entire Canvas area and then uses Graphics.drawString() to add text on top. Iin Figure: Upper rectangle expanded , the upper rectangle is expanded to cover text in the middle part of the Canvas. This causes a considerably larger area to be converted to a hardware-accelerated surface. The best performance in mixing can be achieved by rendering with Graphics only to the smallest possible areas at the top and/or bottom of the Canvas.

Figure: Update rectangles

Figure: Upper rectangle expanded

In cases where an Image needs to be drawn frequently to Canvas, the best performance is achieved by using immutable Image. This allows the implementation to cache image data in graphics memory, enabling faster redraws.

Rendering M3G to image targets

The M3G API is fully hardware-accelerated and always renders to hardware-accelerated surfaces in the GPU memory, whereas LCDUI Images reside in CPU memory. When Image is used as M3G rendering target, the CPU memory needs to be synchronized with the GPU each time M3G rendering starts and ends, that is, when Graphics3D.bindTarget and Graphics3D.releaseTarget are called. This makes Image as a M3G rendering target slower than Canvas. Because of this, using mutable Image for example as a game MIDlet back buffer is not recommended.

MMAPI video and M3G content on Canvas

If there is MMAPI video and M3G content on Canvas at the same time, Canvas is not able to use its hardware accelerated surface for M3G rendering. In this case Canvas equals to Image as a M3G rendering target. This means that the CPU memory needs to be synchronized with the GPU memory every time M3G rendering starts and ends. This might cause the overall user interface performance to be poor.

Nokia UI API

The Nokia UI API uses software rendering. DirectGraphics.getPixels is slow if used for Canvas with M3G content, as synchronization is needed between CPU and GPU memory.

Graphics memory constraints

Due to the limited amount of graphics memory in devices, MIDlets have to stop rendering when switching to the background, that is, when another application moves to the foreground. This allows the implementation to free all used GPU resources. If a MIDlet fails to stop rendering when moving to the background and graphics memory is running too low, the system terminates the MIDlet as a last resort to free resources, enabling it to serve the foreground application. This is especially important with MIDlets using M3G, because in general 3 D graphics require a considerable amount of graphics memory.

If a MIDlet has a separate thread for triggering Canvas repaints or performing GameCanvas rendering, the MIDlet needs to stop the thread when Canvas.hideNotify() is called.

Summary

  • Use only Graphics or M3G, if possible.

  • If mixing Graphics and M3G is needed, limit the number of switches between Graphics and M3G within a paint cycle.

  • Use M3G for clearing the scene background.

  • Use immutable Images instead of mutable Images.

  • Stop rendering when a MIDlet is in the background.

© Nokia 2010.