Image filter (Webpack)
In this example we're going to build a web app that apply a filter on an image. We'll create two versions: one that processes the image in the UI thread and another which offloads the task to web workers.
This example makes use of WebAssembly threads. As support in Zig 0.14.0 is still immature, you'll need to patch the standard library before continuing.
First, we'll create the basic skeleton:
mkdir filter
cd filter
npm init -y
npm install --save-dev react react-dom\
webpack webpack-cli webpack-dev-server css-loader style-loader file-loader html-webpack-plugin\
@babel/core @babel/preset-env @babel/preset-react babel-loader http-server\
zigar-loader
mkdir src zig imgCreate index.css in src:
:root {
font-family: Inter, system-ui, Avenir, Helvetica, Arial, sans-serif;
line-height: 1.5;
font-weight: 400;
color-scheme: light dark;
color: rgba(255, 255, 255, 0.87);
background-color: #242424;
font-synthesis: none;
text-rendering: optimizeLegibility;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
}
body {
margin: 0;
display: flex;
flex-direction: column;
place-items: center;
min-width: 320px;
min-height: 100vh;
}And App.css:
#root {
flex: 1 1 100%;
width: 100%;
}
.App {
display: flex;
position: relative;
flex-direction: column;
width: 100%;
height: 100%;
}
.App .nav {
position: fixed;
width: 100%;
color: #000000;
background-color: #999999;
font-weight: bold;
flex: 0 0 auto;
padding: 2px 2px 1px 2px;
}
.App .nav .button {
padding: 2px;
cursor: pointer;
}
.App .nav .button:hover {
color: #ffffff;
background-color: #000000;
padding: 2px 10px 2px 10px;
}
.App .contents {
display: flex;
width: 100%;
margin-top: 2em;
}
.App .contents .pane {
flex: 1 1 50%;
padding: 5px 5px 5px 5px;
}
.App .contents .pane CANVAS {
border: 1px dotted rgba(255, 255, 255, 0.10);
max-width: 100%;
max-height: 90vh;
}
.App .contents .pane .controls INPUT {
vertical-align: middle;
width: 50%;
}
@media screen and (max-width: 600px) {
.App .contents {
flex-direction: column;
}
.App .contents .pane {
padding: 1px 2px 1px 2px;
}
.App .contents .pane .controls {
padding-left: 4px;
}
}
.hidden {
position: absolute;
visibility: hidden;
z-index: -1;
}
.align-left {
text-align: left;
}
.align-right {
text-align: right;
}Create the React component App.jsx:
import { useCallback, useEffect, useRef, useState } from 'react';
import SampleImage from '../img/sample.png';
import './App.css';
function App() {
const srcCanvasRef = useRef();
const dstCanvasRef = useRef();
const fileInputRef = useRef();
const [ bitmap, setBitmap ] = useState();
const [ intensity, setIntensity ] = useState(0.3);
const onOpenClick = useCallback(() => {
fileInputRef.current.click();
}, []);
const onFileChange = useCallback(async (evt) => {
const [ file ] = evt.target.files;
if (file) {
const bitmap = await createImageBitmap(file);
setBitmap(bitmap);
}
}, []);
const onRangeChange = useCallback((evt) => {
setIntensity(evt.target.value);
}, [])
useEffect(() => {
// load initial sample image
(async () => {
const img = new Image();
img.src = SampleImage;
await img.decode();
const bitmap = await createImageBitmap(img);
setBitmap(bitmap);
})();
}, []);
useEffect(() => {
// update bitmap after user has selected a different one
if (bitmap) {
const srcCanvas = srcCanvasRef.current;
srcCanvas.width = bitmap.width;
srcCanvas.height = bitmap.height;
const ctx = srcCanvas.getContext('2d', { willReadFrequently: true });
ctx.drawImage(bitmap, 0, 0);
}
}, [ bitmap ]);
useEffect(() => {
// update the result when the bitmap or intensity parameter changes
if (bitmap) {
const srcCanvas = srcCanvasRef.current;
const dstCanvas = dstCanvasRef.current;
const srcCTX = srcCanvas.getContext('2d', { willReadFrequently: true });
const { width, height } = srcCanvas;
const srcImageData = srcCTX.getImageData(0, 0, width, height);
dstCanvas.width = width;
dstCanvas.height = height;
const dstCTX = dstCanvas.getContext('2d');
dstCTX.putImageData(srcImageData, 0, 0);
}
}, [ bitmap, intensity ]);
return (
<div className="App">
<div className="nav">
<span className="button" onClick={onOpenClick}>Open</span>
<input ref={fileInputRef} type="file" className="hidden" accept="image/*" onChange={onFileChange}/>
</div>
<div className="contents">
<div className="pane align-right">
<canvas ref={srcCanvasRef}></canvas>
</div>
<div className="pane align-left">
<canvas ref={dstCanvasRef}></canvas>
<div className="controls">
Intensity: <input type="range" min={0} max={1} step={0.0001} value={intensity} onChange={onRangeChange}/>
</div>
</div>
</div>
</div>
)
}
export default AppBasically, we have two
HTML canvases in our app. We
load the initial image with the first useEffect hook, placing the resulting bitmap into the
state variable bitmap:
useEffect(() => {
// load initial sample image
(async () => {
const img = new Image();
img.src = SampleImage;
await img.decode();
const bitmap = await createImageBitmap(img);
setBitmap(bitmap);
})();
}, []);The async iife is necessary here, as useEffect would be unhappy receiving a promise.
The second useEffect hook, activated when bitmap changes, draws the bitmap on the first canvas:
useEffect(() => {
// update bitmap after user has selected a different one
if (bitmap) {
const srcCanvas = srcCanvasRef.current;
srcCanvas.width = bitmap.width;
srcCanvas.height = bitmap.height;
const ctx = srcCanvas.getContext('2d', { willReadFrequently: true });
ctx.drawImage(bitmap, 0, 0);
}
}, [ bitmap ]);The third useEffect hook then obtains an
ImageData object from the first
canvas and draws it on the second canvas:
useEffect(() => {
// update the result when the bitmap or intensity parameter changes
if (bitmap) {
const srcCanvas = srcCanvasRef.current;
const dstCanvas = dstCanvasRef.current;
const srcCTX = srcCanvas.getContext('2d', { willReadFrequently: true });
const { width, height } = srcCanvas;
const srcImageData = srcCTX.getImageData(0, 0, width, height);
const dstImageData = srcImageData;
dstCanvas.width = width;
dstCanvas.height = height;
const dstCTX = dstCanvas.getContext('2d');
dstCTX.putImageData(dstImageData, 0, 0);
}
}, [ bitmap, intensity ]);Then index.js:
import React from 'react'
import ReactDOM from 'react-dom/client'
import App from './App.jsx'
import './index.css'
ReactDOM.createRoot(document.getElementById('root')).render(
<React.StrictMode>
<App />
</React.StrictMode>,
)And index.html:
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Webpack + React + Zigar</title>
</head>
<body>
<div id="root"></div>
</body>
</html>We still need webpack.config.js:
const path = require('path');
const htmlWebpackPlugin = require('html-webpack-plugin');
module.exports = {
entry: './src/index.js',
output: {
path: path.join(__dirname, '/dist'),
filename: 'bundle.js',
},
plugins: [
new htmlWebpackPlugin({
template: 'src/index.html',
}),
],
devServer: {
port: 3030,
},
module: {
rules: [
{
test: /\.jsx?$/,
exclude: /node_modules/,
use: 'babel-loader',
},
{
test: /\.css$/,
use: [ 'style-loader', 'css-loader' ],
},
{
test: /\.zig$/,
exclude: /node_modules/,
use: 'zigar-loader',
},
],
},
};And .babelrc:
{
"presets": [
"@babel/preset-env",
[ "@babel/preset-react", { "runtime": "automatic" } ]
]
}Add some commands to package.json as we have done before:
"scripts": {
"dev": "webpack serve --mode development",
"build": "webpack --mode production",
"preview": "http-server ./dist"
},Almost done! As a final step, download the following image into img as sample.png (or choose
an image of your own):
Now we can see if our code works:
npm run devYou should see the following in the browser:
Nothing will happen when you move the slider, as we haven't yet implemented the filtering functionality. We'll proceed with doing so now that we see that the basic code for our app is working.
First, download
sepia.zig
into the zig sub-directory.
The code in question was translated from a Pixel Bender filter using pb2zig. Consult the intro page for an explanation of how it works.
At the top of App.jsx, insert an import statement for the function createOutput():
import { createOutput } from '../zig/sepia.zig';In our useEffect hook we make use of it:
useEffect(() => {
// update the result when the bitmap or intensity parameter changes
if (bitmap) {
const srcCanvas = srcCanvasRef.current;
const dstCanvas = dstCanvasRef.current;
const srcCTX = srcCanvas.getContext('2d', { willReadFrequently: true });
const { width, height } = srcCanvas;
const srcImageData = srcCTX.getImageData(0, 0, width, height);
const input = { src: srcImageData };
const params = { intensity };
const output = createOutput(width, height, input, params);
const dstImageData = new ImageData(output.dst.data.clampedArray, width, height);
dstCanvas.width = width;
dstCanvas.height = height;
const dstCTX = dstCanvas.getContext('2d');
dstCTX.putImageData(dstImageData, 0, 0);
}
}, [ bitmap, intensity ]);createOutput() has the follow declaration:
pub fn createOutput(
allocator: std.mem.Allocator,
width: u32,
height: u32,
input: Input,
params: Parameters,
) !Outputallocator is automatically provided by Zigar. We get width and height from the source canvas.
params contains a single f32: intensity. We initialize it using our state variable
of the same name, which changes when we move the slider.
Input is a parameterized type:
pub const Input = KernelInput(u8, kernel);Which expands to:
pub const Input = struct {
src: Image(u8, 4, false);
};Then further to:
pub const Input = struct {
src: struct {
pub const Pixel = @Vector(4, u8);
pub const FPixel = @Vector(4, f32);
pub const channels = 4;
data: []const Pixel,
width: u32,
height: u32,
colorSpace: ColorSpace = .srgb,
offset: usize = 0,
};
};Image was purposely defined in a way so that it is compatible with the browser's
ImageData. Its
data field is []const @Vector(4, u8), a slice pointer that accepts a Uint8ClampedArray
as target without casting. We can therefore simply pass { src: srcImageData } to createOutput
as input.
Like Input, Output is a parameterized type. It too can potentially contain multiple images. In
this case (and most cases), there's only one:
pub const Output = struct {
dst: {
pub const Pixel = @Vector(4, u8);
pub const FPixel = @Vector(4, f32);
pub const channels = 4;
data: []Pixel,
width: u32,
height: u32,
colorSpace: ColorSpace = .srgb,
offset: usize = 0,
},
};dst.data points to memory allocated from allocator. Array objects holding numbers in Zigar have
the property typedArray, which provides a matching
TypedArray
view of their data. When it is a Uint8Array, the object will also have the property
clampedArray, which yields a Uint8ClampedArray. We use that to construct an ImageData object:
const dstImageData = new ImageData(output.dst.data.clampedArray, width, height);Now our app does what it's supoosed to:
Modern CPUs typically have more than one core. We can take advantage of the additional computational power by performing data processing in multiple threads. Doing so also means the main thread of the browser won't get blocked, helping to keep the UI responsive.
Multithreading is not enabled by default for WebAssembly. To enable it, add the multithreaded
option in webpack.config.js:
{
test: /\.zig$/,
exclude: /node_modules/,
use: 'zigar-loader',
options: { multithreaded: true },
},Then replace the import statement in App.jsx:
const { createOutput } = await import(`../zig/${filter}.zig`);with the following:
const { createOutputAsync, startThreadPool, stopThreadPool } = await import(`../zig/${filter}.zig`);In the useEffect hook, change the function being called, after wrapping everything in an async
iife first:
(async () => {
// ...
const output = await createOutputAsync(width, height, input, params);
// ...
})();Then add an additional useEffect hook:
useEffect(() => {
startThreadPool(navigator.hardwareConcurrency);
return () => stopThreadPool();
}, []);After saving the file, you'll notice the app no longer works. In the development console you'll find the following message:
Multithreading requires the use of shared memory, a feature available on the browser only when the document is in a secure context. Two HTTP headers must be set.
During development, we can ask Webpack Dev Server to provide them:
devServer: {
port: 3030,
headers: {
'Cross-Origin-Opener-Policy': 'same-origin',
'Cross-Origin-Embedder-Policy': 'require-corp',
}
},You must be able to do the same at the web server when the app is eventually deployed in order to make use of multithreading.
After saving the change, the app will start to work again. The slider won't work very well though
when you drag it. The problem here is that mouse movement could generate very many calls to
createOutputAsync(), far quicker than the computer is able to handle them. We need additional
logic that ensures only the most recent settings received from the UI get worked on. Any unfinished
work triggered by prior changes should simply be abandoned.
Add the following class to the bottom of App.jsx:
class AsyncTaskManager {
currentTask = null;
async call(cb) {
const controller = (cb?.length > 0) ? new AbortController : null;
const promise = this.perform(cb, controller?.signal);
const thisTask = this.currentTask = { controller, promise };
try {
return await thisTask.promise;
} finally {
if (thisTask === this.currentTask) this.currentTask = null;
}
}
async perform(cb, signal) {
if (this.currentTask) {
this.currentTask.controller?.abort();
await this.currentTask.promise?.catch(() => {});
// throw error now if the task was aborted before the function is called
if (signal?.aborted) throw new Error('Aborted');
}
return cb?.(signal);
}
}
const atm = new AsyncTaskManager();The above above creates an
AbortController and
passes its signal to the callback function. It expects a promise as return value. If
this promise fails to resolve before call() is invoked again, then we abort it and
await its rejection.
Then in the useEffect hook, change the call to createOutputAsync:
try {
// ...
const output = await atm.call(signal => createOutputAsync(width, height, input, params, { signal }));
// ...
} catch (err) {
if (err.message != 'Aborted') {
console.error(err);
}
}As an error will get thrown when a call is interrupted, we need to wrap everything in a try/catch.
In the second useEffect hook, we change the cleanup function such that it waits for the
cancellation of any ongoing operation before shutting down the thread pool:
useEffect(() => {
atm.call(() => startThreadPool(navigator.hardwareConcurrency));
return () => atm.call(() => stopThreadPoolAsync());
}, []);With this mechanism in place preventing excessive calls, the app should work correctly.
Now, let us our Zig code. We'll start with startThreadPool():
pub fn startThreadPool(count: u32) !void {
try work_queue.init(.{
.allocator = internal_allocator,
.stack_size = 65536,
.n_jobs = count,
});
}work_queue is a struct containing a thread pool and non-blocking queue. It has the following
declaration:
var work_queue: WorkQueue(thread_ns) = .{};The queue stores requests for function invocation and runs them in separate threads. thread_ns
contains public functions that can be used. For this example we only have one:
const thread_ns = struct {
pub fn processSlice(signal: AbortSignal, width: u32, start: u32, count: u32, input: Input, output: Output, params: Parameters) !Output {
var instance = kernel.create(input, output, params);
if (@hasDecl(@TypeOf(instance), "evaluateDependents")) {
instance.evaluateDependents();
}
const end = start + count;
instance.outputCoord[1] = start;
while (instance.outputCoord[1] < end) : (instance.outputCoord[1] += 1) {
instance.outputCoord[0] = 0;
while (instance.outputCoord[0] < width) : (instance.outputCoord[0] += 1) {
instance.evaluatePixel();
if (signal.on()) return error.Aborted;
}
}
return output;
}
};The logic is pretty straight forward. We initialize an instance of the kernel then loop
through all coordinate pairs, running evaluatePixel() for each of them. After each iteration
we check the abort signal to see if termination has been requested.
createOutputAsync() pushes multiple processSlice call requests into the work queue to
process an image in parellel. Let us first look at its arguments:
pub fn createOutputAsync(allocator: Allocator, promise: Promise, signal: AbortSignal, width: u32, height: u32, input: Input, params: Parameters) !void {Allocator, Promise, and AbortSignal are special
parameters that Zigar provides automatically. On the JavaScript side, the function has only four
required arguments. It will also accept a fifth argument: options, which may contain an alternate
allocator, a callback function, and an
abort signal.
The function starts out by allocating memory for the output struct:
var output: Output = undefined;
// allocate memory for output image
const fields = std.meta.fields(Output);
var allocated: usize = 0;
errdefer inline for (fields, 0..) |field, i| {
if (i < allocated) {
allocator.free(@field(output, field.name).data);
}
};
inline for (fields) |field| {
const ImageT = @TypeOf(@field(output, field.name));
const data = try allocator.alloc(ImageT.Pixel, width * height);
@field(output, field.name) = .{
.data = data,
.width = width,
.height = height,
};
allocated += 1;
}Then it divides the image into multiple slices. It divides the given Promise struct as well:
// add work units to queue
const workers: u32 = @intCast(@max(1, work_queue.thread_count));
const scanlines: u32 = height / workers;
const slices: u32 = if (scanlines > 0) workers else 1;
const multipart_promise = try promise.partition(internal_allocator, slices);partition() creates a new promise
that fulfills the original promise when its resolve() method has been called a certain number of
times. It is used as the output argument for work_queue.push():
var slice_num: u32 = 0;
while (slice_num < slices) : (slice_num += 1) {
const start = scanlines * slice_num;
const count = if (slice_num < slices - 1) scanlines else height - (scanlines * slice_num);
try work_queue.push(thread_ns.processSlice, .{ signal, width, start, count, input, output, params }, multipart_promise);
}
}
The first argument to push() is the function to be invoked. The second is a tuple containing
arguments. The third is the output argument. The return value of processSlice(), either the
Output struct or error.Aborted, will be fed to this promise's resolve() method. When the
last slice has been processed, the promise on the JavaScript side becomes fulfilled.
Let us look at one last function: stopThreadPoolAsync:
pub fn stopThreadPoolAsync(promise: zigar.function.Promise(void)) void {
work_queue.deinitAsync(promise);
}Shutdown of the work queue can only happen asynchronously, since blocking the main thread can lead to a deadlock. In any event, it's prohibited to do so in the web browser.
Just run the following commands:
npm run build
npm run previewWithout the overhead of Zig runtime safety, the app should be much snappier. It should be noted
that nothing stops you from adding optimize: 'ReleaseSmall' to the plugin options so you would
get full performance from WASM code even during development.
You can find the complete source code for this example here.
A major advantage of using Zig for a task like image processing is that the same code can be deployed both on the browser and on the server. After a user has made some changes to an image on the frontend, the backend can apply the exact same effect using the same code. Consult the Node version of this example to learn how to do it.
The image filter employed for this example is very rudimentary. Check out pb2zig's project page to see more advanced code.
That's it for now. I hope this tutorial is enough to get you started with using Zigar.