LuaAV Graphics Tutorial

An overview of visual synthesis in LuaAV.

Window(title, x, y, w, h)

Windowis the main visual surface for drawing and displaying graphics in LuaAV. Window is also the main generator of user interaction events such as mouse movement and keyboard presses. Window itself is highly configurable with a wide range of options for how it appears on the screen (border, fullscreen, floating, ...), and it always contains an OpenGL context. In LuaAV, OpenGL is the primary graphics rendering technology.

Windows can be created with a list of arguments specifiying its title and rectangle on the screen or with a table supplying key-value pairs for settable window properties.

Window("Canvas", 0, 0, 512, 512)

or equivalently

	title = "Canvas",
	origin = {0, 0},
	dim = {512, 512},

The events Window generates can be handled in a script simply by defining a Lua function on a Window. For example, when the Window gets resized, it will check if a function named 'resize' exists and call it if it does.

win = Window("Canvas", 0, 0, 512, 512)

-- define a resize event handler
function win:resize()
	local dim = self.dim
	print("new dim: ", dim[1], dim[2])
The complete list of callbacks can be found in the Window documentation

Drawing to a Window

All drawing to a Window is done through OpenGL. LuaAV provides both high-level and low-level bindings to OpenGL, which are call contained in the opengl module and its sub-modules. The low-level OpenGL bindings make available nearly all of the functions specified in the OpenGL reference pages. Using these functions in Lua is practically identical to their C equivalents, so it's generally straightforward to translate sample code found online into Lua OpenGL code. The main difference is that all of the bindings are in a Lua table as are the OpenGL enums, so a C call like glBegin(GL_LINES) becomes gl.Begin(GL.LINES) where '.' replaces '_' for enums and gets inserted after the 'gl' for functions. To draw a point at (0, 0, 0) for example looks like:

local gl = require("opengl")
local GL = gl

function win:draw()
		gl.Vertex(0, 0, 0)
Notce how 'gl' is aliased to 'GL' so that when using enums looks almost identical to the C code equivalent.

High-level OpenGL

While the low-level OpenGL bindings provide access to the majority of the OpenGL specification, there are a few important areas of functionality missing, notable those dealing textures and extensions such as the GLSL shading language. These areas form logical groupings that can easily be brought together. Furthermore, some functions dealing with textures and buffers of geometry are too low-level to deal with directly in a scripting language such as Lua. Iterating over lage blocks of memory would impose a huge performance cost in Lua, so instead we let the bindings do the hard work and Lua direct the traffic.

The high-level interfaces are exposed as sub-modules to the OpenGL module. They all require a valid OpenGL context to operate properly. When used with Window, the lifetime of the OpenGL resources they use will automatically be synchronized with the lifetime of the Window's OpenGL context. See the opengl module for a list of sub-modules.

As an example, the following code loads an image file and displays it on a quad:

local gl = require("opengl")
local GL = gl
local Texture = require("opengl.Texture")
local sketch = require("opengl.sketch")

local image = require("image")
local Image = image.Image

local ctx = "Load Image"
win = Window(ctx, 0, 0, 512, 512)

local image = Image(LuaAV.findfile("LuaAV.96.png"))
local tex = Texture(ctx)

function win:draw()
	gl.Color(1, 1, 1, 1)
	gl.Scale(0.5, 0.5, 0.5)

There are a few things to note in the above code:

In addition to opengl and opengl.Texture modules, the code uses opengl.sketch for displaying the texture data.

The All Important Lattice

As already discussed, dealing directly with low-level memory is prohibitive within Lua, so instead we let the bindings to C/C++ do the heavy lifting. Still, the memory has to be passed around within the script somehow. In LuaAV, all large blocks of memory (images, video frames, geometry buffers, audio samples, etc.) are represented by the Lattice. The Lattice is essentially a block of memory tied to a description of how the memory is formatted. In the previous, section, the call in the example code to img:matrix() returned a lattice representing the image as an RGBA buffer of pixels while tex:frommatrix() takes the lattice as an argument and prepares it for loading onto the GPU.

When dealing with video data from a video file or a live stream from a camera, each video frame can be accessed by calling video:matrix() to get the Lattice representing the frame data.

local lattice = require("lattice")
local Lattice = lattice.Lattice

local lat = Lattice{
	components = 4,         -- The number of components (samples per-cell)
	type = lattice.Float32, -- The data type
	dim = {512, 512},       -- The dimensions
	align = 4,              -- Optional byte-alignment

The above code will create a 4-channel, float buffer 512x512 in size with 4-byte alignment. See the lattice documentation for more details