Parallelization is "free" on hardware

[rachitnigam]

I was just chatting with @andrewb1999 about the interplay between unsharing and auto-parallelization---unsharing a bunch of components may expose extra parallelization in the design---and realized that you probably don't need a cost model when deciding to parallelize a program or not. In other words, parallelization is "free" in hardware because everything is already parallel anyways.

Consider the standard dilemma in software: parallelization is good but often comes with substantial coordination costs if you're not careful. Furthermore, it is limited by the amount of hardware support for parallelism in a normal processor. In contrast, when we generate circuits, we pay a cost when we sequentialize a computation---we have to build the FSM that tracks which state is active when. If we parallelize something, we're getting rid of that cost because all we need is some mechanism to wait for all the threads to join.

In other words, the cost of "spawning a thread" in hardware is much lower which means we should be aggressively parallelizing when we can.

[cgyurgyik]

and realized that you probably don't need a cost model when deciding to parallelize a program or not.

This is assuming your hardware constraints are already known, and not part of the cost model?

[rachitnigam]

Yeah, to clarify: if you’ve already decided to unshare two components and it exposes parallelism, there is no reason not to run them in parallel AFAICT

[cgyurgyik]

While not necessarily modeled in Calyx, power consumption would play a role (2 active versus 1 active/1 idle).

[rachitnigam]

Not necessarily. In smaller process technologies, static power becomes the dominant aspect and is a function of the number of transistors in the design instead of their switching behavior: https://csci.williams.edu/people/faculty/mark-hopkins/

Basically, you need to turn off substantial parts of a circuit to see benefits in the power consumption.

On FPGAs, the trade off is even more stark because you’re paying for the power usage of the FPGA fabric all the time

[rachitnigam]

Actually, I’m wrong about the FPGA stuff but I have an even simpler argument: if you don’t parallelize, you run twice as long so the power benefits of not paralyzing need to be quite big to dominate

[cgyurgyik]

I'm imagining a scenario where your chip has limited power, limited heat dissipation, or whatever, and this needs to be taken into account. You could handle this at the software level or the hardware level, but I think how much you parallelize things would play a role in such a cost model.

[sampsyo]

Yeah, that's an interesting way of framing it! I might restrict the statement to something like: "resource-preserving parallelism is free," in the sense that you might as well parallelize if you already have the hardware resources to parallelize, you might as well do it. But of course, many good parallelism strategies really do entail duplicating hardware—that's the default notion with HLS loop unrolling, for instance.