First Timeline Semaphore use #158

MoritzRoth · 2023-03-30T18:15:43Z

The first step towards full timeline semaphore support. Related Issue: #45

Changes:

Adds and implements context_vulkan::record_and_submit_with_timeline_semaphore
Replaces fences in model_loader example with timeline semaphores.

johannesugb

Cool stuff! Please mind the comments and update according to the requested changes in the Auto-Vk PR!

johannesugb · 2023-04-03T18:00:36Z

auto_vk_toolkit/include/configure_and_compose.hpp

@@ -186,6 +186,7 @@ namespace avk
 			.setShaderUniformTexelBufferArrayDynamicIndexing(VK_TRUE)
 			.setShaderStorageTexelBufferArrayDynamicIndexing(VK_TRUE)
 			.setDescriptorIndexing(VK_TRUE)
+			.setTimelineSemaphore(VK_TRUE)


This should probably not be enabled by default but instead, enabled through the std::function<void(vk::PhysicalDeviceVulkan12Features&)> parameter that can be passed to configure_and_compose.

johannesugb · 2023-04-03T18:05:02Z

auto_vk_toolkit/include/context_vulkan.hpp

+		/** \brief Records and submits commands to a queue together with a newly created timeline semaphore, which is signalled upon completion.
+		* \param aRecordedCommandsAndSyncInstructions	List of commands to record and submit.
+		* \param aQueue									The queue to submit the commands to.
+		* \param aSrcSignalStage						Defines in which stage it is safe to signal the created timeline semaphore.
+		* \param aSignalValue							The value to SIGNAL the timeline semaphore to.
+		* \param aInitialValue				(optional)	The value to INITIALIZE the newly created timeline semaphore with.
+		* \param aUsageFlags				(optional)	CommandBuffer usage flags.
+		* \return The newly created timeline semaphore.
+		*/


Suggested change

/** \brief Records and submits commands to a queue together with a newly created timeline semaphore, which is signalled upon completion.

* \param aRecordedCommandsAndSyncInstructions List of commands to record and submit.

* \param aQueue The queue to submit the commands to.

* \param aSrcSignalStage Defines in which stage it is safe to signal the created timeline semaphore.

* \param aSignalValue The value to SIGNAL the timeline semaphore to.

* \param aInitialValue (optional) The value to INITIALIZE the newly created timeline semaphore with.

* \param aUsageFlags (optional) CommandBuffer usage flags.

* \return The newly created timeline semaphore.

*/

/** @brief Records and submits commands to a queue together with a newly created timeline semaphore, which is signaled upon completion.

* @param aRecordedCommandsAndSyncInstructions List of commands to record and submit.

* @param aQueue The queue to submit the commands to.

* @param aSrcSignalStage Defines in which stage(s) after which is safe to signal the created timeline semaphore.

* @param aSignalValue The value to signal the timeline semaphore to.

* @param aInitialValue (optional) The value to initialize the newly created timeline semaphore with.

* @param aUsageFlags (optional) CommandBuffer usage flags.

* @return The newly created timeline semaphore.

*/

Please use @ instead of \, and please try to horizontally align all the @.
Also, please use spaces to align the comments!

Please mind the comment fixes!

And "signaled" is written in a wrong way multiple times, I'm afraid (also before this PR).

johannesugb · 2023-04-03T18:22:47Z

auto_vk_toolkit/src/context_vulkan.cpp

+			.submit();
+
+		aSignalInfo.mSignalSemaphore.get().handle_lifetime_of(std::move(cmdBfr));
+		// TODO we probably need another way to handle the lifetime of resources since timeline semaphores aren't single use ==> live longer


That should actually be okay, because the (timeline) semaphore takes care of the ownership of the command buffer.
Or what was your argument in this case exactly?

What I was talking about here was that since timeline semaphores are no longer single-shot synchronization primitives (1 signal, 1 wait) they might have a lifetime close to the whole application runtime. If a command buffer now gets enqueued and executed early on in the semaphore's lifespan, won't it be kept alive until the semaphore is destroyed, way longer than it needs to exist?

Oh, yeah. Now I get the point. You're completely right.

Hmm, the right way to do this would probably be to turn std::vector<any_owning_resource_t> into std::vector<std::tuple<any_owning_resource_t, uint64_t>>, where the uint64_t item represents the timeline semaphore's value after which it is safe to delete the resource.

A similar approach is implemented for command buffers, see the following code:

// prepare command buffer for re-recording void command_buffer_t::prepare_for_reuse() { if (mPostExecutionHandler.has_value()) { // Clear post-execution handler mPostExecutionHandler.reset(); } if (mCustomDeleter.has_value() && *mCustomDeleter) { // If there is a custom deleter => call it now (*mCustomDeleter)(); mCustomDeleter.reset(); } mLifetimeHandledResources.clear(); }

Just for timeline semaphores, it would not be a mLifetimeHandledResources.clear();, but a clear depending on the uint64_t value.

With the most recent changes, I added an optional value parameter to semaphore_t::handle_lifetime_of. Here I just pass the same value the semaphore will be signaled to, once the work has finished.

I suspect this is wrong in cases where the passed aSignalInfo uses a stage other than all_commands. Do you think we should add another param for a cmdBuffer deletion value, or just let the caller setup the lifetime handling themselves?

johannesugb · 2023-04-03T18:26:21Z

examples/model_loader/source/model_loader.cpp

 				std::move(posFillCmd),
 				std::move(tcoFillCmd),
 				std::move(nrmFillCmd),
 				std::move(idxFillCmd)
 				// ^ No need for any synchronization in-between, because the commands do not depend on each other.
-			}, *mQueue);
+				}, * mQueue, { vk::PipelineStageFlagBits2::eAllTransfer }, 1);


Suggested change

}, * mQueue, { vk::PipelineStageFlagBits2::eAllTransfer }, 1);

}, *mQueue, avk::stage::all_transfer, 1);

Better in terms of conciseness and formatting.

johannesugb · 2023-04-03T18:27:25Z

examples/model_loader/source/model_loader.cpp

 			std::move(materialCommands),
 			mMaterialBuffer->fill(gpuMaterials.data(), 0)
-		}, *mQueue);
-		matFence->wait_until_signalled();
+		}, *mQueue, {vk::PipelineStageFlagBits2::eAllCommands}, 1);


Suggested change

}, *mQueue, {vk::PipelineStageFlagBits2::eAllCommands}, 1);

}, *mQueue, avk::stage::all_commands, 1);

johannesugb

Let's review again after the requested changes in Auto-Vk!

johannesugb · 2023-08-01T08:54:20Z

auto_vk_toolkit/include/context_vulkan.hpp

+		*   @param aRecordedCommandsAndSyncInstructions List of commands to record and submit.
+		*   @param aQueue                               The queue to submit the commands to.
+		*   @param aSrcSignalStage                      Defines in which stage it is safe to signal the created timeline semaphore.
+		*   @param aSignalValue                         The value to SIGNAL the timeline semaphore to.
+		*   @param aInitialValue                        (optional) The value to INITIALIZE the newly created timeline semaphore with.
+		*   @param aUsageFlags                          (optional) CommandBuffer usage flags.
+		*   @return The newly created timeline semaphore.
+		*/


Suggested change

* @param aRecordedCommandsAndSyncInstructions List of commands to record and submit.

* @param aQueue The queue to submit the commands to.

* @param aSrcSignalStage Defines in which stage it is safe to signal the created timeline semaphore.

* @param aSignalValue The value to SIGNAL the timeline semaphore to.

* @param aInitialValue (optional) The value to INITIALIZE the newly created timeline semaphore with.

* @param aUsageFlags (optional) CommandBuffer usage flags.

* @return The newly created timeline semaphore.

*/

* @param aRecordedCommandsAndSyncInstructions List of commands to record and submit.

* @param aQueue The queue to submit the commands to.

* @param aSrcSignalStage Defines in which stage it is safe to signal the created timeline semaphore.

* @param aSignalValue The value to SIGNAL the timeline semaphore to.

* @param aInitialValue (optional) The value to INITIALIZE the newly created timeline semaphore with.

* @param aUsageFlags (optional) CommandBuffer usage flags.

* @return The newly created timeline semaphore.

*/

Please add spaces before the *, so that all * are vertically aligned!

johannesugb · 2023-08-01T08:58:40Z

auto_vk_toolkit/src/context_vulkan.cpp

+			.submit();
+
+		aSignalInfo.mSignalSemaphore.get().handle_lifetime_of(std::move(cmdBfr));
+		// TODO we probably need another way to handle the lifetime of resources since timeline semaphores aren't single use ==> live longer


Oh, yeah. Now I get the point. You're completely right.

Hmm, the right way to do this would probably be to turn std::vector<any_owning_resource_t> into std::vector<std::tuple<any_owning_resource_t, uint64_t>>, where the uint64_t item represents the timeline semaphore's value after which it is safe to delete the resource.

A similar approach is implemented for command buffers, see the following code:

// prepare command buffer for re-recording void command_buffer_t::prepare_for_reuse() { if (mPostExecutionHandler.has_value()) { // Clear post-execution handler mPostExecutionHandler.reset(); } if (mCustomDeleter.has_value() && *mCustomDeleter) { // If there is a custom deleter => call it now (*mCustomDeleter)(); mCustomDeleter.reset(); } mLifetimeHandledResources.clear(); }

Just for timeline semaphores, it would not be a mLifetimeHandledResources.clear();, but a clear depending on the uint64_t value.

add context_vulkan::record_and_submit_with_timeline_semaphore replace fences in model_loader.cpp with timeline semaphores

comment formatting use more concise options for PipelineStageFlagBits2

… sem >> value

…r prone atm)

…ng directory, s.t. assets and shaders can be found when run from within VS.

add particle + metadata structs add compute shader skeleton add compute shader queue and setup pipeline (still unused atm) create particle buffer, intermediate buffer and vertex buffers in init()

…ctly with the particle buffer i intend to use add a temporary uniform mHostParticleBuffer and use it to draw prticles with mRenderPipeline adjust a_triangle.vert to use the particleBuffer

compute shader still runs on tha same queue directly before each frame

…ll handle_lifetime_of, with the value the semaphore will be signaled to at the end of execution

…ate thread add slider modifying physics refresh rate

keep workgroup dimensions at 1 for now

MoritzRoth self-assigned this Mar 30, 2023

johannesugb requested changes Apr 3, 2023

View reviewed changes

MoritzRoth force-pushed the timeline-semaphores branch from 69721d4 to fe3a09c Compare July 5, 2023 18:28

MoritzRoth changed the base branch from master to development July 5, 2023 18:57

MoritzRoth force-pushed the timeline-semaphores branch from fe3a09c to 2d00de1 Compare July 22, 2023 12:07

MoritzRoth requested a review from johannesugb July 25, 2023 15:54

johannesugb requested changes Aug 1, 2023

View reviewed changes

MoritzRoth and others added 7 commits February 27, 2024 14:10

enable timeline semaphore feature

84b7467

add context_vulkan::record_and_submit_with_timeline_semaphore replace fences in model_loader.cpp with timeline semaphores

add documentation

c23dd36

typo fix

d2f9424

adjust for auto_vk changes to semaphore_signal_info creation

1278e6b

stop enabling timeline semaphores by default

9231193

comment formatting use more concise options for PipelineStageFlagBits2

Adapted to changes in Auto-Vk, which now require the syntax: stage >>…

5a5441f

… sem >> value

adapt to operator changes for timeline semaphores in auto_vk

db06a39

MoritzRoth force-pushed the timeline-semaphores branch from 2230be1 to db06a39 Compare February 27, 2024 13:30

MoritzRoth and others added 10 commits February 27, 2024 18:06

create particle example setup (duplicate of hello_world example, erro…

3feb870

…r prone atm)

Fixed the .user file for the particles project, which fixes the worki…

4bbb1c8

…ng directory, s.t. assets and shaders can be found when run from within VS.

start compute shader setup

af6b349

add particle + metadata structs add compute shader skeleton add compute shader queue and setup pipeline (still unused atm) create particle buffer, intermediate buffer and vertex buffers in init()

intermediate changes to ensure that the graphics pipeline works corre…

158b9ed

…ctly with the particle buffer i intend to use add a temporary uniform mHostParticleBuffer and use it to draw prticles with mRenderPipeline adjust a_triangle.vert to use the particleBuffer

rename a_triangle.vert/.frag to particle.vert/.frag

e7f32ee

iterate further to use particle buffer filled by compute shader

a7c0d1b

compute shader still runs on tha same queue directly before each frame

also use metadata uniform buffer

440b921

when recording and submitting a cmdBuffer with timeline semaphore, ca…

2d3e3e6

…ll handle_lifetime_of, with the value the semaphore will be signaled to at the end of execution

move compute shader execution to separate queue and dispatch in separ…

acb073a

…ate thread add slider modifying physics refresh rate

start implementing the particle compute shader

fca360b

keep workgroup dimensions at 1 for now

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

First Timeline Semaphore use #158

First Timeline Semaphore use #158

MoritzRoth commented Mar 30, 2023

johannesugb left a comment

johannesugb Apr 3, 2023

MoritzRoth Jul 22, 2023

johannesugb Apr 3, 2023

MoritzRoth Jul 22, 2023

johannesugb Apr 3, 2023

MoritzRoth Jul 22, 2023

johannesugb Aug 1, 2023

MoritzRoth Mar 3, 2024

johannesugb Apr 3, 2023

MoritzRoth Jul 22, 2023

johannesugb Apr 3, 2023

MoritzRoth Jul 22, 2023

johannesugb left a comment

johannesugb Aug 1, 2023

johannesugb Aug 1, 2023

johannesugb Aug 1, 2023

	}, * mQueue, { vk::PipelineStageFlagBits2::eAllTransfer }, 1);
	}, *mQueue, avk::stage::all_transfer, 1);

	}, *mQueue, {vk::PipelineStageFlagBits2::eAllCommands}, 1);
	}, *mQueue, avk::stage::all_commands, 1);

First Timeline Semaphore use #158

Are you sure you want to change the base?

First Timeline Semaphore use #158

Conversation

MoritzRoth commented Mar 30, 2023

johannesugb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johannesugb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment