Branching and looping for p5.strands

[davepagurek]

Resolves #7868

Changes

• Finished implementation of if/else GLSL generation
• Added transpilation of JS branching into GLSL branching
• Added tests for ifs and nested ifs
• Added tests for swizzle assignments in blocks
• Looping support
• Transpilation of JS loops into strands loops
• looping tests
• Bug fix: handle self-update statements like i++ again
• Bug fix: handle >= operator properly
• Added a contributor doc explaining the key parts of the p5.strands architecture

I refactored a bit in order to handle nested blocks more easily: there's now a SCOPE_START and SCOPE_END block for { and } respectively, and I added an ASSIGNMENT statement that is currently only used to assign to phi variables in each branch.

Details

The main challenge here is that we have to fully replace control flow (if/for) in users' code into function calls so that these structures can generate nodes in our program graph. If we don't, they run in javascript, and are invisible to GLSL generation. For example, if you had a loop that runs 10 times that adds 1 each time, it would output the add 1 line 10 times rather than outputting a for loop.

However, once we have a function call instead of real control flow, we also need a way to make sure that when the users' javascript subsequently references nodes that were updated in the control flow, they properly reference the modified value after the if or for and not the original value. For that, we make the function calls return updated values, and we generate JS code that assigns these updated values back to the original JS variables.

If statements

You write an if statement like this:

const testShader = baseMaterialShader().modify(() => {
  const condition = uniformFloat(() => 1.0);
  getPixelInputs(inputs => {
    let color = 0.5; // initial gray
    if (condition > 0.5) {
      color = 1.0; // set to white in if branch
    }
    inputs.color = [color, color, color, 1.0];
    return inputs;
  });
});

It gets transpiled into this function call. Each branch callback gets a copy of the node so that it can modify it without affecting the original, and each branch callback then returns an object with modified versions of those variables. The whole if/else structure returns an object with nodes representing the output of the branch, and we then assign back values from that to the original variables.

() => {
  const condition = uniformFloat('condition', () => 1);
  getPixelInputs(inputs => {
    let color = float(0.5);
    {
      color = __p5.strandsNode(color);
      const __block_2 = __p5.strandsIf(
        // Condition
        __p5.strandsNode(condition).greaterThan(0.5),
        // If branch
        () => {
          let __copy_color_0 = color.copy();
          __copy_color_0 = myp5.float(1);
          return { color: __copy_color_0 };
        }
      // Else branch
      ).Else(() => {
        let __copy_color_1 = color.copy();
        return { color: __copy_color_1 };
      });
      color = __block_2.color;
    }
    inputs.color = __p5.strandsNode([
      color,
      color,
      color,
      1
    ]);
    return inputs;
  });
};

This then gets compiled to the following GLSL:

(Inputs inputs) {
  float T0 = float(0.5000);
  float T1;
  if (condition > float(0.5000))
  {
    float T2 = float(1.0000);
    T1 = T2;
  }
  else
  {
    T1 = T0;
  }
  inputs.normal = inputs.normal;
  inputs.texCoord = inputs.texCoord;
  inputs.ambientLight = inputs.ambientLight;
  inputs.ambientMaterial = inputs.ambientMaterial;
  inputs.specularMaterial = inputs.specularMaterial;
  inputs.emissiveMaterial = inputs.emissiveMaterial;
  inputs.color = vec4(T1, T1, T1, 1.0000);
  inputs.shininess = inputs.shininess;
  inputs.metalness = inputs.metalness;
  return inputs;
}

For loops

You write a loop like this:

const testShader = baseMaterialShader().modify(() => {
  getPixelInputs(inputs => {
    let color = float(0.0);

    for (let i = 0; i < 3; i++) {
      color = color + 0.1;
    }

    inputs.color = [color, color, color, 1.0];
    return inputs;
  });
});

This gets transpiled into this format, where we use a strands function call, and have a callback function for each part of the for loop. The loop body is structured more like a reduce, taking in current state + loop iteration and returning next state. At the end of the for loop, the properties of the final state are assigned back to their original variables.

() => {
 getPixelInputs(inputs => {
    let color = float(0);
    {
      const __block_1 = __p5.strandsFor(
        // Initial iterator value
        () => { return 0; },
        // Loop condition
        loopVar => (__p5.strandsNode(loopVar).lessThan(3)),
        // Loop update
        loopVar => {
          return loopVar = __p5.strandsNode(loopVar).add(1);
        },

        // Loop body, taking in current state values and returning updated state
        (loopVar, vars) => {
          let __copy_color_0 = vars.color.copy();
          __copy_color_0 = __p5.strandsNode(__copy_color_0).add(0.1);
          return { color: __copy_color_0 };
        },
        
        // Initial state
        { color: __p5.strandsNode(color) }
      );
      color = __block_1.color;
    }
    inputs.color = __p5.strandsNode([
        color,
        color,
        color,
        1
    ]);
    return inputs;
  });
};

This then gets turned into the following GLSL:

(Inputs inputs) {
  float T0 = float(0.0000);
  float T1;
  float T2 = float(0.0000);
  for (
  T1 = T2;
  (T1 < float(3.0000));
  T1 = (T1 + float(1.0000))
  )
  {
    T0 = (T0 + float(0.1000));
  }
  inputs.normal = inputs.normal;
  inputs.texCoord = inputs.texCoord;
  inputs.ambientLight = inputs.ambientLight;
  inputs.ambientMaterial = inputs.ambientMaterial;
  inputs.specularMaterial = inputs.specularMaterial;
  inputs.emissiveMaterial = inputs.emissiveMaterial;
  inputs.color = vec4(T0, T0, T0, 1.0000);
  inputs.shininess = inputs.shininess;
  inputs.metalness = inputs.metalness;
  return inputs;
}

PR Checklist

• npm run lint passes
• Inline reference is included / updated
• Unit tests are included / updated

[perminder-17]

Just wondering that we don't need to have !=, !== cases right? Like if user uses any of the other operations maybe (**) which are not in the case, can we throw an error message saying Unsupported operator?

[davepagurek]

good catch, that's probably what will happen!

[davepagurek]

** might have to be handled differently, because unlike the rest, which get turned into a method in js but then back into an operator in GLSL, ** becomes pow and needs to stay pow in GLSL, which will need some special casing in the code. I'll just leave a TODO for that one for the future.

[ksen0]

As I was testing this, I found that the error on using ** is not really usable - though I saw the TODO and I understand that this is not intended final behavior.

[lukeplowden]

dynamicNode has been changed to strandsNode since the refactor

[davepagurek]

ah yes good catch! updated

[lukeplowden]

I can't get the black to show. Maybe it has something to do with the codegen, rather than the graph though? Take a look at the outputted code. I changed the colours from this test example to make it clearer for me. I guesss that the nested if statement doesn't know that it should be updating T0 and creates its own output variable instead?

let testShader;

async function setup() {
  createCanvas(windowWidth, windowHeight, WEBGL);
  testShader = baseMaterialShader().modify(() => {
    const value = uniformFloat(() => 0.5); // middle value
    getPixelInputs(inputs => {
      let color = vec3(0.0);
      if (value > 0.8) {
        color = vec3(1.0, 0, 0); // white for high values
      } else if (value > 0.3) {
        color = vec3(0, 1, 0); // gray for medium values
      } else {
        color = vec3(0, 0, 1); // black for low values
      }
      inputs.color = [color, 1.0];
      return inputs;
    });
  });
}

function draw() {
  background(0);
  shader(testShader);
  testShader.setUniform('value', mouseX/width)
  sphere(100)
}

(Inputs inputs) {
  vec3 T0;
  if (value > float(0.8000))
  {
    vec3 T1 = vec3(1.0000, 0.0000, 0.0000);
    T0 = T1;
  }
  else
  {
    T0 = vec3(0.0000, 1.0000, 0.0000);
    vec3 T2;
    if (value > float(0.3000))
    {
      vec3 T3 = vec3(0.0000, 1.0000, 0.0000);
      T2 = T3;
    }
    else
    {
      vec3 T4 = vec3(0.0000, 0.0000, 1.0000);
      T2 = T4;
    }
  }
  inputs.normal = inputs.normal;
  inputs.texCoord = inputs.texCoord;
  inputs.ambientLight = inputs.ambientLight;
  inputs.ambientMaterial = inputs.ambientMaterial;
  inputs.specularMaterial = inputs.specularMaterial;
  inputs.emissiveMaterial = inputs.emissiveMaterial;
  inputs.color = vec4(T0, 1.0000);
  inputs.shininess = inputs.shininess;
  inputs.metalness = inputs.metalness;
  return inputs;
}

[davepagurek]

Did some digging tonight, the problem seems to be that we add assignment to the phi node in the branch body block, but there may also be sub-blocks that happen after. To accommodate, I've made it add a new block just for phi assignments that happens right at the end of a branch to ensure it never ends up accidentally generated abrove sub-branches.

[davepagurek]

this is probably also an issue for for loops so I need to take a look at that too still

[davepagurek]

ok I think I fixed the issues in for loops -- I also had to update local variable detection there which I had only done for if statements before and added some more tests.

Live here: https://editor.p5js.org/davepagurek/sketches/jeOynvqNG

There are some other issues I'm running into though, I'll make an update to fix those

[lukeplowden]

Is it possible that the assignments arent being wrapped in strandsNode calls because of the pass order? It's causing an error if you do

  if (condition > 0.5) {
    col = 1.0;
  }

Instead of col = float(1) on the second line above.

[davepagurek]

good catch! just added a test for this and it failed, so I pushed an update to wrap all assignments in branches in strandsNode. then it works!

[davepagurek]

There was an additional bug where texture was being used as a constructor for a sampler2D variable, so I changed that to just be sampler2D(), and now I can make a super basic blurlike effect with for loops! https://editor.p5js.org/davepagurek/sketches/41N90RSgk

[ksen0]

This is really exciting to see!

Just a small note on debugging with branches and loops, which create a lot more opportunity to use variables inside strands. Variables that exist in global scope can still be referenced inside strands, though they will not work. From ex below, I can write instead of if (value < 0.3) { rather if (mouseX/windowWidth < 0.3) {. This produces no errors, but also doesn't work. I am not sure if there's any way to do anything differently, and I think this is outside the scope of this PR, but maybe something to consider on the FES x. p5.strands topic.

let testShader;
let pixelate;
let state_;


async function setup() {
  state_ = 0;
  createCanvas(windowWidth, windowHeight, WEBGL);
  testShader = baseMaterialShader().modify(() => {
    const value = uniformFloat(() => mouseX/windowWidth);
    const state = uniformFloat(() => state_);

    let time = uniformFloat(() => frameCount/5);
    getPixelInputs((inputs, canvasContent) => {
      let color = inputs.color;
      if (value < 0.3) {
        color = vec3(1.0, state*0.1, 0);
      } else if (value < 0.8) {
        color = vec3(state*0.1, 1, state*0.1);
      } else {
        color = vec3(state*0.1, 0, 1);
      }
      inputs.color = [color, 1.0];
      return inputs;
    });
  });
}

function draw() {
  background(0);
  shader(testShader);
  testShader.setUniform('value', mouseX/width)
  sphere(100);
}

function mousePressed(){
  state_ += 1;
  console.log(state_)
}

[ksen0]

This whole documentation is wonderful! Very minor and not necessary for this iteration: DAG illustration / connecting it to code block below not only in text

[davepagurek]

+1, I'll see if I can make an SVG to put in. Minor note, in the past I've used Memaid code blocks to do inline diagrams which is helpful for writing and also pretty easily editable, but we never got around to making the p5.js website render it correctly, so it currently shows up as the Mermaid source code there. At some point I'd love to get that working so I can keep using that format, but for now probably svg is still the way to go.

[ksen0]

Great point, as we just discussed on Discord, I'll do a bit more research on it and open processing/p5.js-website#168 for work. So please feel free to add only the mermaid code here if you prefer, though of course an SVG would be more helpful sooner

[ksen0]

As I was testing this, I found that the error on using ** is not really usable - though I saw the TODO and I understand that this is not intended final behavior.

[ksen0]

I think it would be really good to more directly express the intended audience. There's parts of this that would be useful to anyone encountering strands, and parts only useful to people who want to contribute to strands (though this is the contribution section on the site, the intro reads like a general motivation for strands usage). Ie, it's important all users of p5.strands are aware this is a "different mode" of javascript; and that there is also GLSL and OpenGL. However, I don't think terminology of transpiler is generally necessary unless people are interested. Maybe at the end of the intro, direct readers to the tutorials on strands and specify that the rest of this document is for contribution?

[davepagurek]

I think in the future there are two things I'd like to do to address usage of outside variables:

• Add p5.strands overrides for more p5 properties and functions so that you can actually just reference mouseX, width, etc, and it adds uniforms for you under the hood when it sees you using them

• For non-p5 things referenced from the outside, I looked into it a bit and it seems like there's just not a good way to get it to Just Work™, so like you mention, the next step is probably to try to detect it and give a helpful message. We do support this pattern where you can pass in outside variables in an object and then get them again in arguments to the p5.strands callback, which is how you have to reference p5 itself in strands in instance mode:

  let val = 10
  const myShader = baseMaterialShader().modify(() => {
    getWorldInputs((inputs) => {
      inputs.position.x = val
      return inputs
    })
  }, { val })

  ...so maybe we can try to detect usages of outside variables and suggest adding them in this pattern? It could be that in our transpilation step we can keep track of all variable declarations, and check if we see a reference to something that isn't in those declarations and also isn't a builtin, and then output a snippet for you to update your code with.

I could make follow-up issues for both of those if that makes sense!

[ksen0]

Those follow-ups sound great! Just some thoughts below.

...so maybe we can try to detect usages of outside variables and suggest adding them in this pattern? It could be that in our transpilation step we can keep track of all variable declarations, and check if we see a reference to something that isn't in those declarations and also isn't a builtin, and then output a snippet for you to update your code with.

Ah, that would be nice. Not plausible to go 1 step further and somehow infer that { val } inclusion behind the scenes? Also to what extent is const condition = uniformFloat(() => val); different from the modify({ val }) pattern in making val available?

[davepagurek]

So all the limitations come from the fact that we're not running the user's callback directly, just parsing it to a string, and then making our own function with it. By reconstructing a function like that, it loses all scope that the original function had. So while this works:

let val = 10;
let cb = () => console.log(val);
cb();

...this doesn't, because the runtime creation of the function makes a new, empty scope for it, and this is essentially what we're doing after transpilation:

let val = 10;
let cb = new Function('console.log(val)');
cb(); // `val` is not in scope

The object thing that we pass in is a little hack to take in a value from outside and pass it in, so that we still get access:

let val = 10;
let cb = new Function('val', 'console.log(val)');
cb(val); // `val` is now just a parameter to the function, so it's in scope

So that's not just a change inside of the callback's source code, we also need to change something outside of the callback in order to pass in val. That's the reason why we aren't able to do that automatically: we'd have to rewrite not just the callback, but the code that passed in the callback, which would create the same problem for any variables it references. So basically we could only do it if we're able to parse and rewrite the entire source code of the user's sketch, which is a lot heavier and probably will introduce problems in other environments/build systems/etc where p5 could be embedded.

The difference between just passing in val and creating a uniform is that the former captures the value of val at the time the shader is created, while the latter is dynamic. So this would still make a red circle:

let color = [1, 0, 0, 1];
let colorShader = baseMaterialShader().modify(() => {
  getFinalColor(() => {
    return color;
  });
}, { color });

color = [0, 0, 1, 1]; // Set to blue, but the original red is already captured

shader(colorShader);
sphere(30);

...whereas a uniform can be updated live:

function setup() {
  createCanvas(400, 400, WEBGL);
  
  let color = [1, 0, 0, 1];
  let colorShader = baseMaterialShader().modify(function() {
    const colorUniform = uniformVector4()
    getFinalColor(() => {
      return colorUniform;
    });
  });
  
  color = [0, 0, 1, 1];

  shader(colorShader);
  colorShader.setUniform('colorUniform', color)
  sphere(30);
}