RPS Tutorial Part 4 – RPS SDK multithreading guide

Extending our sample app

So, we just mentioned that a call to rpsCmdSetCommandBuffer should be made at the end of the callback. This was enough to paint a general picture of how we could accomplish multi-threaded recording in the callback.

We will now modify our sample app towards a complete implementation.

For our DX12 MT use-case, we desire that all the command lists have the same pipeline state setup – the state that the preamble setup for us.

How can we get RPS to do this for all of our worker thread command lists?

This is accomplished with rpsCmdBeginRenderPass. This call does exactly the same thing as the node preamble (which was explained in the 3rd tutorial part). The important bit is that we get to call this on our own time, and that the difference between the explicit invocation the implicit one is that the flags bound at node bind time do not apply to rpsCmdBeginRenderPass. This is important, as we want to set RPS_CMD_CALLBACK_CUSTOM_ALL to skip the automatic preamble, but we do not want this behavior to apply for the explicit preamble.

Let’s make these changes in the app.

First, we set the custom all flag in order to skip the default preamble – we only want to set up graphics state (bind the render target, etc) on the worker thread command buffers,

virtual void OnInit(ID3D12GraphicsCommandList*         pInitCmdList,
                        std::vector<ComPtr<ID3D12Object>>& tempResources) override
{
    // ...
    // Bind nodes.
    AssertIfRpsFailed(rpsProgramBindNode(rpsRenderGraphGetMainEntry(m_rpsRenderGraph),
                                        "GeometryPass",
                                        &RpsMultithreading::GeometryPass,
                                        this,
                                        RPS_CMD_CALLBACK_CUSTOM_ALL));
    // ...
}

Next, we want to make sure to call rpsCmdBeginRenderPass on each command buffer. However, we aren’t quite ready to do so. Consider the signature of the function,

RpsResult rpsCmdBeginRenderPass(const RpsCmdCallbackContext* pContext, RpsRuntimeRenderPassFlags flags);

The first parameter is the callback context. Recall that this context contains a handle to the command buffer that we passed in when kicking off the render graph recording. If we were to call this function with the context that our callback gets called with, the graphics state would be setup on the main command buffer, not the worker thread one.

How can we get this function to record to an arbitrary command buffer?

Introducing another API call: rpsCmdCloneContext. You provide this call with the context that the callback receives, and it clones that context. At the same time, you ensure to provide a new handle to a command buffer. Every context maps to a command buffer.

And so, now we can actually make the call to rpsCmdBeginRenderPass.

void GeometryPass(const RpsCmdCallbackContext* pContext,
                  rps::UnusedArg               u0,
                  float                        oneOverAspectRatio,
                  float                        timeInSeconds)
{
    //...
    for (uint32_t i = 0; i < renderJobs; i++)
    {
        // auto hNewCmdBuf = AcquireNewCommandBuffer( ... );
        const RpsCmdCallbackContext* pLocalContext = {};
        {
            std::lock_guard<std::mutex> lock(m_cmdListsMutex);

            AssertIfRpsFailed(rpsCmdCloneContext(pContext, hNewCmdBuf, &pLocalContext));
        }
        // ...
        auto job = [=]() {
            RpsCmdRenderPassBeginInfo beginInfo = {};
            RpsRuntimeRenderPassFlags rpFlags   = RPS_RUNTIME_RENDER_PASS_FLAG_NONE;
            rpFlags |= (i != 0) ? RPS_RUNTIME_RENDER_PASS_RESUMING : 0;
            rpFlags |= (i != (renderJobs - 1)) ? RPS_RUNTIME_RENDER_PASS_SUSPENDING : 0;
            beginInfo.flags = rpFlags;
            AssertIfRpsFailed(rpsCmdBeginRenderPass(pLocalContext, &beginInfo));
            // render ...
            AssertIfRpsFailed(rpsCmdEndRenderPass(pLocalContext));
        };

        // launch job.
    }
    // ... 
}

There’s two additional ideas in the snippet above that have not been explained yet. Firstly, a lock is used when calling rpsCmdCloneContext. The cloned context is a data structure managed by the RPS SDK and therefore an allocation occurs for this call. Undefined behavior will occur if we are trying to call clone context from two threads at once.

The second idea is the mysterious render pass resuming and suspending flags that are passed along with the call to rpsCmdBeginRenderPass. These flags are passed to indicate that a render pass is suspending, and will resume with another render pass. If a render pass is resuming, any clears that would have otherwise automatically occurred will not. Likewise, if a render pass is suspending, any resolves will not automatically occur.

And that’s it, we’re pretty much done! We just need to make the call to rpsCmdSetCommandBuffer to override the buffer to be used for subsequent node callback recording.

void GeometryPass(const RpsCmdCallbackContext* pContext,
                  rps::UnusedArg               u0,
                  float                        oneOverAspectRatio,
                  float                        timeInSeconds)
{
    //...
    for (uint32_t i = 0; i < renderJobs; i++)
    {
        // ...
    }

    // auto hNewCmdBuf = AcquireNewCommandBuffer( ... );
    AssertIfRpsFailed(rpsCmdSetCommandBuffer(pContext, hNewCmdBuf));
}

Finally, recall that RPS does not track historical command buffers. Thus, for our sample app, a special helper function AcquireNewCommandBuffer has been created. This is used to construct a linked list structure of command buffers so that at queue submission time we know how to submit them in the correct order.