Custom Shaders in Hexels
Adding Your Own Blending and Post-Processing Effects
By Ken Kopecky, Lead Hexels Kengineer
Note: This is a re-post of my Shaders tutorial from the Marmoset website. I’ve posted it here because we’re currently in the middle of transitioning to a new website, and this tutorial hasn’t made its way over yet.
One of the coolest new features in Hexels 2.5 is the use of fragment shaders for layer blending and post effects (PFX). Shaders allow these effects to be calculated in real-time for Hexels’ vector graphics. This tutorial gives an overview of shaders and how they’re used in Hexels, and then shows a couple simple examples of adding custom shaders, allowing Hexels to perform new tricks. A basic knowledge of GLSL 1.2 (OpenGL Shading Language) is highly recommended, but anyone with some programming experience should be able to do something with this tutorial. If you need a refresher on GLSL, try:
http://www.lighthouse3d.com/tutorials/glsl-12-tutorial/
Or this quick-reference guide:
http://www-evasion.imag.fr/Membres/Sebastien.Barbier/Enseignement/glsl_quickref.pdf
Please note that as we improve Hexels, this information may change, and any shaders you make may need to be updated for future versions, but we’ll try to keep as much backwards compatibility as possible..
What is a Shader?
A fragment shader is a short program that runs on the graphics card (GPU) to determine the color of an individual pixel. Shaders run on the GPU far faster than compiled code runs on the CPU, which means that post effects can be re-calculated every time Hexels draws without causing a noticeable slowdown. Another important feature of shaders is that they can be loaded and reloaded while the application is running. For developers, this is useful because code changes can be seen immediately without having to recompile and re-run. For Hexels users, this allows people to create their own blending modes and PFX, potentially doing things with Hexels that we at Marmoset never dreamed of.
How Does Hexels Use Shaders?
- Hexels incorporates shaders into every stage of its drawing pipeline:
- Drawing shapes and calculating low-level effects like halftone and texturing
- Layer masking
- Post-processing effects applied to each layer
- Clipping
- Layer compositing (blending)
- Calculation of glow
- Final compositing of glow onto the Hexels image
- Static (baked) color adjustment
Currently, three of these stages–PFX, layer compositing (blend), and baked color adjustment–support selection of shaders, including custom shaders supplied by the user.
Post-processing (PFX) shaders will be used for our two examples. PFX are similar to filters in Photoshop in that they modify an existing image according to a set of rules. Common PFX in Hexels include blur and color adjustment. PFX are only able to read textures from a single layer.
Blend shaders are like blending modes in Photoshop; they combine a layer with the image below it. Thus, they read the color of the incoming layer and the color of the already-blended layer stack, and output a result that is a combination of those two colors.
Both PFX and blend shaders are applied in real-time to layers, and any input values they have can be changed and even animated. Baked shaders, on the other hand, are applied once and the effect is permanent (that is, destructive, unless you hit Undo, of course). These are available in the Edit-> Adjust Colors menu in Hexels. Rather than operating on each pixel of the image, baked shaders operate just once on each Hexel, changing its stored color. Baked shaders must be flagged as Atomic. This is explained further in Technical Details.
A Simple Example
To get started, let’s see what it takes to add a very basic PFX shader to Hexels. This shader will simply invert the color.
- Launch Hexels
- Under the Shaders menu, select “Open User Shaders Folder…”
- Open the “Post Effects” folder inside the folder from Step 2.
- Open a text editor or IDE (preferably something with syntax
- highlighting and line numbers, such as Notepad++ on Windows or TextWrangler on Mac)
- Create a file with the following code and save it as “Invert.frag” inside the Post Effects folder.
void main()
{
//sample the incoming pixel color
vec4 incoming = texture2D(_Layer, gl_TexCoord[0].xy);
//ensure the incoming color is between 0-1, so we don't get negative colors!
incoming = clamp(incoming, 0.0, 1.0);
//basic color inversion here of the r, g, and b channels
//the built-in _EffectStrength variable is used to adjust the “amount” of a shader
//currently, _EffectStrength is used only with the opacity of an adjustment layer
//use it lessen the effect of a shader in a tasteful way
incoming.rgb = mix(incoming.rgb, vec3(1.0, 1.0, 1.0) - incoming.rgb, _EffectStrength);
//set the outgoing color
gl_FragColor = incoming;
}
- To view your newly created shader:
Close Hexels and launch it again. This will re-scan the add-ons directory for shaders.
- Draw something, then go to the Layer Properties window for the current layer, and click on Post Effects. You should see “Invert” listed when you click the + button. Go ahead and add it to the layer.
The above picture shows the invert effect applied to a masterpiece of Trixels. The left side is the original, and the right side is after the effect is applied. Hopefully you saw a change in your image, too. If not, either you made a mistake in the above steps, or this tutorial is out of date. Fortunately, we tried to make it easy to get feedback on your custom shaders. Start by going to your Shaders menu and selecting “Reload Shaders”, or just hitting ctrl(cmd on Mac)+shift+R. This causes Hexels to re-read all of its shader files, recompile them, and then write all the compilation logs to a file in your log directory.
Note: When editing your shader, you’ll need to reload shaders every time you change your shader file.
To see the contents of the shader compile logs, select “Show Shader Log…” under the Shaders menu to open your log directory, and open ShaderCompile.txt. Shader errors should be clearly marked in this file with easy-to-spot things like “***”, “___”, and “ERROR”. Look for the filename of your shader, and then look immediately below it for any errors that popped up. Please note that the file provides line numbers, but these line numbers are off by 1000, due to the way Hexels assembles shaders internally (i.e. an error on line 1025 means line 25 in your shader). So just ignore that extra 1.
The Shader menu also has an option to show shader errors on the main Hexels canvas if you don’t feel like digging through a text file. If no errors are found, selecting this option won’t display anything.
Fixing syntax and other shader errors are beyond the scope of this this tutorial, but our Technical Details section will provide information regarding shader uniform variables provided by Hexels.
Adding Adjustment Controls
Hopefully, the simple PFX shader in the last section worked for you (If not, please contact us and let us know what we can do to improve this tutorial). While the above is a clear example of what you can do, it’s not very exciting. Most post-effects have settings with sliders, buttons and widgets that you can play with to make them do different things. Fortunately, Hexels provides hooks to let you add UI controls to your shaders. In fact, nearly every PFX in Hexels (as of this writing, the only exception is Curves) uses a UI that’s generated based on text in the shader file itself. In this example, we’ll modify the Invert shader we just made to allow control for each channel. At the top of the Invert.frag file, add these three lines:
UI_Uniform_Float(Red, 0, 1, 1); //this text will show up as a tooltip
UI_Uniform_Float(Green, 0, 1, 1); //so will this
UI_Uniform_Float(Blue, 0, 1, 1); //and this
These lines tell Hexels to create sliders in the shader’s user interface (UI) pane. Inside the parentheses are the parameter name, and its minimum, maximum, and default values. Comments following the declaration will be used as tooltips in the UI. Save your file and reload shaders in Hexels (ctrl/cmd + shift + R). When you go back to Layer Properties and click on the Invert post-effect, you should see something like this:
The values for “Red”, “Green”, and “Blue” are now accessible in Hexels. If you’ve set your Invert post-effect to be animated (by clicking the key icon next to it in the Layers window), these three parameters will even animate if they are set differently at different frames. In the shader itself, Red, Green, and Blue are float uniforms and can be used in code accordingly:
UI_Uniform_Float(Red, 0, 1, 1); //this text will show up as a tooltip
UI_Uniform_Float(Green, 0, 1, 1); //so will this
UI_Uniform_Float(Blue, 0, 1, 1); //and this
void main()
{
//sample the incoming pixel color
vec4 incoming = texture2D(_Layer, gl_TexCoord[0].xy);
//ensure the incoming color is between 0-1, so we don't get negative colors!
incoming = clamp(incoming, 0.0, 1.0);
//use the values of our Red, Green, and Blue parameters to affect how much
//inverting is actually happening to each channel
vec3 neg = mix(incoming.rgb, vec3(1.0, 1.0, 1.0)-incoming.rgb, vec3(Red, Green, Blue));
incoming.rgb = mix(incoming.rgb, neg, _EffectStrength);
//set the outgoing color
gl_FragColor = incoming;
}
Other UI items are available, including checkboxes, int sliders, and color pickers (color pickers coming soon…). Syntax for those can be found in the “Technical Details” section of this tutorial.
Technical Details
Now that you know the basics of custom shaders in Hexels, this section will cover some of the more nitty gritty things you need to know to actually write them. The most important thing to know is that all of the shaders covered in this document have a common shader file that is inserted above your code. This file is located inside Hexels’ shader directory. Select “Open Built-In Shaders Folder” under the Shaders menu and look for Header.frag in the Post Effects folder.
Protip: While you’re in there, be sure to take a look at the shader code we shipped with Hexels to get an idea of the syntax.
Header.frag includes all of the uniforms and textures passed to PFX, blend, and bake shaders, as well as some #defines for interfacing with the UI. This file is heavily commented and will be up-to-date for your current version of Hexels. I won’t go over everything that’s in there, but here are some highlights:
UI_Uniform_(stuff)
As we saw above in the second example, these statements are what connect shader variables to Hexels. They help Hexels create UI elements like sliders and checkboxes so you can edit and animate the values in your shader. There are four basic types that are currently usable:
UI_Uniform_Float(name, min, max, default); //comments here are converted to tooltips
This creates a basic floating point value slider in the UI. You’ll probably use it more than any of the other types. Underscores in the name are converted to spaces so they look nice in the shader properties window.
UI_Uniform_Angle(name, min, max, default); //...
This is the same as a float, except its value always moves the “short way” around a circle (in degrees). So if you’re animating this value from 10 to 350, it will pass through zero and sweep just 20 degrees, rather than sweeping a full 340 degrees like a UI_Uniform_Float would.
UI_Uniform_Int(name, min, max, default); //…
The int slider takes the same arguments as the float slider, but integers are passed to the shader instead of floats.
UI_Uniform_Checkbox(name, default); //…
Instead of a slider with arbitrary values, this one is a checkbox and its value are just 1 and 0.
More uniform types (such as colors) are supported internally but are not yet accessible via the UI.
EffectStrength
This one is important, and should be used in every PFX shader. The “_EffectStrength” variable is used when adjusting the opacity of a blend shader to smoothly increase or decrease the “amount” of an effect. In our blur shader, for example, as _EffectStrength is reduced, so is the blur radius. In our color adjustment shader, it fades the output color back to the initial color. Try to use it in an elegant way in your shader, if possible. It’s usually just a simple multiply.
Multi-Pass Shaders
Several shaders in Hexels need to run several times to achieve the desired effect. Our blur shader, for example, saves time by running two (relatively) fast linear blurs in perpendicular directions. You can add this sort of thing to your own shader by using
UI_Uniform_Const_int(Render_Passes, c);
Where c is the number render passes you want your shader to use. Try to keep this number low or Hexels will grind to a halt. The number of the current render pass is stored in a uniform called (you guessed it) _RenderPass.
Atomic (Baked) Shaders
Because baked shaders aren’t aware of the spatial relationship of each Hexel, they should only be used for PFX that don’t change the “shape” of the drawing, like color adjustment, rather than blur. Any PFX shader containing #define ATOMIC somewhere (including inside comments) will be made available as a baked shader in the Color Adjustment window. A wonderful exception to this is if you’re editing an image cel, any post-effect is available for baking. You can even select a portion of the image and modify that.
Blend Shaders
Although the above examples are for PFX shaders, blend shaders are extremely simple to make. The only difference is they have two texture inputs instead of one. Check out the blend shader code supplied with Hexels 2 (“Open Built-In Shaders Folder…” under the Shaders menu) for plenty of examples.
Conclusion
I hope you have found this tutorial to be both helpful and inspiring. We are very excited to see what sorts of special effects people add to Hexels, and we worked hard to make it as easy as possible while still giving you, the user, a great deal of control. If you have any questions, comments or suggestions for how we can improve the custom shader system, please let us know by sending an email to hexels@marmoset.co
