| title | author | category | excerpt | retired | status | ||
|---|---|---|---|---|---|---|---|
GPUImage |
Mattt |
Open Source |
GPUImage is a BSD-licensed iOS library that lets you apply GPU-accelerated filters and other effects to images, live camera video, and movies. If you're not careful, you may well end up creating a camera app by the end of the article. |
true |
|
Here at NSHipster, we're all about diving into the darker corners of Objective-C to learn something new about the systems we interact with every day. Often, this means sifting through Apple frameworks or language features (it is, after all, a lot of what it means to work in Objective-C). However, on occasion, it's nice to take a look to the burgeoning landscape of third-party libraries and frameworks (and there are some truly remarkable ones) for a glimpse of what's new and great outside of Cupertino.
This week, we'll be taking a look at one of the most impressive open source projects you'll find: GPUImage. Buckle up, NSHipsters—if you're not careful, you may well end up creating a camera app by the end of the article.
GPUImage is a BSD-licensed iOS library written by Brad Larson that lets you apply GPU-accelerated filters and other effects to images, live camera video, and movies.
Every iPhone ships with two processors: a CPU, or Central Processing Unit and a GPU, or Graphics Processing Unit. Each processor has its own strengths, and modern chip architecture (like in the iPhone's A4) integrate the CPU and GPU onto the same physical die.
When you write C or Objective-C code in Xcode, you're generating instructions that will be handled almost exclusively by the CPU. The GPU, by contrast, is a specialized chip that is especially well-suited for computation that can be split out into many small, independent operations, such as graphics rendering. The kinds of instructions understood by the GPU are quite different from that of the CPU, and as such, we write this code in a different language: OpenGL (or specifically, OpenGL ES on the iPhone & iPad).
Check out Jeff LaMarche's GLProgram OpenGL ES 2.0 book for a great introduction to OpenGL ES and the rendering pipeline.
Comparing the performance of GPU-based rendering to CPU rendering for something like video, the differences are staggering:
| Calculation | GPU FPS | CPU FPS | Δ |
|---|---|---|---|
| Thresholding ⨉ 1 | 60.00 | 4.21 | 14.3⨉ |
| Thresholding ⨉ 2 | 33.63 | 2.36 | 14.3⨉ |
| Thresholding ⨉ 100 | 1.45 | 0.05 | 28.7⨉ |
Let me put it this way:
Instagram : GPUImage :: Disposable Camera : NASA Space Optics Manufacturing Center
To put it another way, within GPUImage's APIs lay thousands of camera apps, just waiting for the right combination of filters and a little spark of imagination.
Here's a table of the 125 (!) filters that come with GPUImage:
| Color Adjustments | Image Processing | Blending Modes | Visual Effects |
|---|---|---|---|
|
|
|
|
Seriously, the Filter Showcase Example App that comes bundled in the repository could easily retail on the AppStore for $3.99, as-is. Add Twitter integration and a few sound effects, and you could bump that up to a respectable$6.99.
{% asset gpuimage-pipeline.svg %}
GPUImage is, at its core, an Objective-C abstraction around a rendering pipeline. Source images from the camera, network, or disk are loaded and manipulated according to a chain of filters, and finally outputted either a view, graphics context, or data stream.
For example, images from the video camera could have a Color Levels filter applied to simulate different types of color blindness and displayed in a live view.
GPUImageVideoCamera *videoCamera = [[GPUImageVideoCamera alloc]
initWithSessionPreset:AVCaptureSessionPreset640x480
cameraPosition:AVCaptureDevicePositionBack];
videoCamera.outputImageOrientation = UIInterfaceOrientationPortrait;
GPUImageFilter *filter = [[GPUImageLevelsFilter alloc] initWithFragmentShaderFromFile:@"CustomShader"];
[filter setRedMin:0.299 gamma:1.0 max:1.0 minOut:0.0 maxOut:1.0];
[filter setGreenMin:0.587 gamma:1.0 max:1.0 minOut:0.0 maxOut:1.0];
[filter setBlueMin:0.114 gamma:1.0 max:1.0 minOut:0.0 maxOut:1.0];
[videoCamera addTarget:filter];
GPUImageView *filteredVideoView = [[GPUImageView alloc] initWithFrame:self.view.bounds];
[filter addTarget:filteredVideoView];
[self.view addSubview:filteredVideoView];
[videoCamera startCameraCapture];Or, combining various color blending modes, image effects, and adjustments, you could transform still images into something worthy of sharing with your hipster friends (example taken from FilterKit, which is built on GPUImage):
GPUImageFilterGroup *filter = [[GPUImageFilterGroup alloc] init];
GPUImageSaturationFilter *saturationFilter = [[GPUImageSaturationFilter alloc] init];
[saturationFilter setSaturation:0.5];
GPUImageMonochromeFilter *monochromeFilter = [[GPUImageMonochromeFilter alloc] init];
[monochromeFilter setColor:(GPUVector4){0.0f, 0.0f, 1.0f, 1.0f}];
[monochromeFilter setIntensity:0.2];
GPUImageVignetteFilter *vignetteFilter = [[GPUImageVignetteFilter alloc] init];
[vignetteFilter setVignetteEnd:0.7];
GPUImageExposureFilter *exposureFilter = [[GPUImageExposureFilter alloc] init];
[exposureFilter setExposure:0.3];
[filter addGPUFilter:exposureFilter];
[filter addGPUFilter:monochromeFilter];
[filter addGPUFilter:saturationFilter];
[filter addGPUFilter:vignetteFilter];Looking through all of what GPUImage can do, one can't help but get excited. Easy enough to get started immediately (without needing to know anything about OpenGL) yet performant enough to power whatever you dream up. And not just that, but it also comes with a dizzying number of building blocks—all of the color adjustments, blending modes, and visual effects you could ever want (or never knew you needed).
GPUImage is a rare treat for the open source community, and we as Mac & iOS developers are lucky to have it at our disposal. Use it to make something great, and show others the world in a whole new way.