Skip to content

FidelityFX Frame Interpolation 3.1.5

Table of contents

Introduction

FidelityFX Frame Interpolation is a technique that analytically generates an intermediate frame from two consecutive source images, interpolating the motion of pixels between the start & end images.

The frame generation context computes the interpolated image. Once this is accomplished, the interpolated and real back buffers still need to be used, i.e. usually sent to the swapchain. On the topic of how to handle presentation and pacing of the back buffers, please refer to the frame interpolation swapchain documentaion.

Shading language requirements

  • HLSL
    • CS_6_2
    • CS_6_6†

CS_6_6 is used on some hardware which supports 64-wide wavefronts.

Integration

FidelityFX Frame Interpolation should be integrated using the FidelityFX API. This document describes the API constructs specific to FidelityFX Frame Interpolation.

Memory Usage

Figures are given to the nearest MB, taken on Radeon RX 9070 XTX using DirectX 12, and are subject to change. Does not include frame generation swapchain overheads.

Output resolutionMemory usage(MB)
3840x2160457
2560x1440214
1920x1080124

An application can get amount of GPU local memory required by Frame Interpolation after context creation by calling ffxQuery with the valid context and ffxQueryDescFrameGenerationGetGPUMemoryUsage.

An application can get GPU local memory required by default Frame Interpolation version before context creation by calling ffxQuery with NULL context and filling out ffxQueryDescFrameGenerationGetGPUMemoryUsageV2. To get the memory requirement info for a different Frame Interpolation version, additionally link ffxOverrideVersion.

See code examples how to call Query.

Create frame generation context

In order to use frame generation first call ffxCreateContext with a description for frame generation and a backend description.

The ffxCreateContextDescFrameGeneration structure contains configuration data:

  • A set of initialization flags
  • The maximum resolution the rendering will be performed at
  • The resolution of the resources that will get interpolated
  • The format of the resources that will get interpolated

The initialization flags are provided though the FfxApiCreateContextFramegenerationFlags enumeration:

FlagNote
FFX_FRAMEGENERATION_ENABLE_ASYNC_WORKLOAD_SUPPORTA bit indicating whether the context supports async. dispatch for frame-generation workloads.
FFX_FRAMEGENERATION_ENABLE_DISPLAY_RESOLUTION_MOTION_VECTORSA bit indicating if the motion vectors are rendered at display resolution.
FFX_FRAMEGENERATION_ENABLE_MOTION_VECTORS_JITTER_CANCELLATIONA bit indicating that the motion vectors have the jittering pattern applied to them.
FFX_FRAMEINTERPOLATION_ENABLE_DEPTH_INVERTEDA bit indicating that the input depth buffer data provided is inverted [1..0].A bit indicating the depth buffer is inverted.
FFX_FRAMEINTERPOLATION_ENABLE_DEPTH_INFINITE A bit indicating that the input depth buffer data provided is using an infinite far plane.
FFX_FRAMEGENERATION_ENABLE_HIGH_DYNAMIC_RANGEA bit indicating if the input color data provided to all inputs is using a high-dynamic range.
FFX_FRAMEGENERATION_ENABLE_DEBUG_CHECKINGA bit indicating that the runtime should check some API values and report issues.

Example using the C++ helpers:

ffx::Context frameGenContext;
ffx::CreateBackendDX12Desc backendDesc{};
backendDesc.device = GetDevice()->DX12Device();
ffx::CreateContextDescFrameGeneration createFg{};
createFg.displaySize = {resInfo.DisplayWidth, resInfo.DisplayHeight};
createFg.maxRenderSize = {resInfo.DisplayWidth, resInfo.DisplayHeight};
createFg.flags = FFX_FRAMEGENERATION_ENABLE_HIGH_DYNAMIC_RANGE;
if (m_EnableAsyncCompute)
createFg.flags |= FFX_FRAMEGENERATION_ENABLE_ASYNC_WORKLOAD_SUPPORT;
createFg.backBufferFormat = SDKWrapper::GetFfxSurfaceFormat(GetFramework()->GetSwapChain()->GetSwapChainFormat());
ffx::ReturnCode retCode = ffx::CreateContext(frameGenContext, nullptr, createFg, backendDesc);

Configure frame generation

Configure frame generation by filling out the ffxConfigureDescFrameGeneration structure with the required arguments and calling ffxConfigure.

This must be called once per frame. The frame ID must increment by exactly 1 each frame. Any other difference between consecutive frames will reset frame generation logic.

ffxConfigureDescFrameGeneration memberNote
swapChainThe swapchain to use with frame generation.
presentCallbackA UI composition callback to call when finalizing the frame image.
presentCallbackUserContextA pointer to be passed to the UI composition callback.
frameGenerationCallbackThe frame generation callback to use to generate a frame.
frameGenerationCallbackUserContextA pointer to be passed to the frame generation callback.
frameGenerationEnabledSets the state of frame generation. Set to false to disable frame generation.
allowAsyncWorkloadsSets the state of async workloads. Set to true to enable generation work on async compute.
HUDLessColorThe hudless back buffer image to use for UI extraction from backbuffer resource. May be empty.
flagsZero or combination of flags from FfxApiDispatchFrameGenerationFlags.
onlyPresentGeneratedSet to true to only present generated frames.
generationRectThe area of the backbuffer that should be used for generation in case only a part of the screen is used e.g. due to movie bars
frameIDIdentifier used to select internal resources when async support is enabled. Must increment by exactly one (1) for each frame. Any non-exactly-one difference will reset the frame generation logic.
// Update frame generation config
FfxApiResource hudLessResource = SDKWrapper::ffxGetResourceApi(m_pHudLessTexture[m_curUiTextureIndex]->GetResource(), FFX_API_RESOURCE_STATE_COMPUTE_READ);
m_FrameGenerationConfig.frameGenerationEnabled = m_FrameInterpolation;
m_FrameGenerationConfig.flags = 0;
m_FrameGenerationConfig.flags |= m_DrawFrameGenerationDebugTearLines ? FFX_FRAMEGENERATION_FLAG_DRAW_DEBUG_TEAR_LINES : 0;
m_FrameGenerationConfig.flags |= m_DrawFrameGenerationDebugResetIndicators ? FFX_FRAMEGENERATION_FLAG_DRAW_DEBUG_RESET_INDICATORS : 0;
m_FrameGenerationConfig.flags |= m_DrawFrameGenerationDebugView ? FFX_FRAMEGENERATION_FLAG_DRAW_DEBUG_VIEW : 0;
m_FrameGenerationConfig.HUDLessColor = (s_uiRenderMode == 3) ? hudLessResource : FfxApiResource({});
m_FrameGenerationConfig.allowAsyncWorkloads = m_AllowAsyncCompute && m_EnableAsyncCompute;
// assume symmetric letterbox
m_FrameGenerationConfig.generationRect.left = (resInfo.DisplayWidth - resInfo.UpscaleWidth) / 2;
m_FrameGenerationConfig.generationRect.top = (resInfo.DisplayHeight - resInfo.UpscaleHeight) / 2;
m_FrameGenerationConfig.generationRect.width = resInfo.UpscaleWidth;
m_FrameGenerationConfig.generationRect.height = resInfo.UpscaleHeight;
// For sample purposes only. Most applications will use one or the other.
if (m_UseCallback)
{
m_FrameGenerationConfig.frameGenerationCallback = [](ffxDispatchDescFrameGeneration* params, void* pUserCtx) -> ffxReturnCode_t
{
return ffxDispatch(reinterpret_cast<ffxContext*>(pUserCtx), &params->header);
};
m_FrameGenerationConfig.frameGenerationCallbackUserContext = &m_FrameGenContext;
}
else
{
m_FrameGenerationConfig.frameGenerationCallback = nullptr;
m_FrameGenerationConfig.frameGenerationCallbackUserContext = nullptr;
}
m_FrameGenerationConfig.onlyPresentGenerated = m_PresentInterpolatedOnly;
m_FrameGenerationConfig.frameID = m_FrameID;
m_FrameGenerationConfig.swapChain = GetSwapChain()->GetImpl()->DX12SwapChain();
ffx::ReturnCode retCode = ffx::Configure(m_FrameGenContext, m_FrameGenerationConfig);
CauldronAssert(ASSERT_CRITICAL, !!retCode, L"Configuring FSR FG failed: %d", (uint32_t)retCode);

If using the frame generation callback, the swapchain will call the callback with appropriate parameters. Otherwise, the application is responsible for calling the frame generation dispatch and setting parameters itself. In that case, the frame ID must be equal to the frame ID used in configuration. The command list and output texture can be queried from the frame generation context using ffxQuery. See the sample code for an example.

The user context pointers will only be passed into the respective callback functions. FSR code will not attempt to dereference them.

When allowAsyncWorkloads is set to false the main graphics queue will be used to execute the Optical Flow and Frame Generation workloads. It is strongly advised to profile, if significant performance benefits can be gained from asynchronous compute usage. Not using asynchronous compute will result in a lower memory overhead.

Note that UI composition and presents will always get executed on an async queue, so they can be paced and injected into the middle of the workloads generating the next frame.

FSR3 non async workflow

When allowAsyncWorkloads is set to true, the Optical Flow and Frame Generation workloads will run on an asynchronous compute queue and overlap with workloads of the next frame on the main game graphics queue. This can improve performance depending on the GPU and workloads.

FSR3 non async workflow

UI Composition

For frame interpolation the user interface will require some special treatment, otherwise very noticeable artifacts will be generated which can impact readability of the interface.

To prohibit those artifacts frame-generation supports various options to handle the UI:

The preferred method is to use the presentCallback. The function provided in this parameter will get called once for every frame presented and allows the application to schedule the GPU workload required to render the UI. By using this function the application can reduce UI input latency and render effects that do not work well with frame generation (e.g. film grain).

The UI composition callback function will be called for every frame (real or generated) to allow rendering the UI separately for each presented frame, so the UI can get rendered at presentation rate to achieve smooth UI animations.

ffxReturnCode_t FSR3RenderModule::UiCompositionCallback(ffxCallbackDescFrameGenerationPresent* params, void* userCtx)
{
ID3D12GraphicsCommandList2* pDxCmdList = reinterpret_cast<ID3D12GraphicsCommandList2*>(params->commandList);
ID3D12Resource* pRtResource = reinterpret_cast<ID3D12Resource*>(params->outputSwapChainBuffer.resource);
ID3D12Resource* pBbResource = reinterpret_cast<ID3D12Resource*>(params->currentBackBuffer.resource);
// Use pDxCmdList to copy pBbResource and render UI into the outputSwapChainBuffer.
// The backbuffer is provided as SRV so postprocessing (e.g. adding a blur effect behind the UI) can easily be applied
return FFX_API_RETURN_OK;
}

FSR3 non async workflow

If frame generation is disabled presentCallback will still get called on present. FSR3 non async workflow

The second option to handle the UI is to render the UI into a dedicated surface that will be blended onto the interpolated and real backbuffer before present. Composition of this surface can be done automatically composed by the proxy swapchain or manually in the presentCallback. This method allows to present an UI unaffected by frame interpolation, however the UI will only be rendered at render rate. For applications with a largely static UI this might be a good solution without the additional overhead of rendering the UI at presentation rate.

FSR3 non async workflow

If frame generation is disabled and the UI Texture is provided, UI composition will still get executed by the frame interpolation swapchain. FSR3 non async workflow

In that case the surface needs to be registered to the swap chain by calling ffxConfigure with a ffxConfigureDescFrameGenerationSwapChainRegisterUiResourceDX12 structure.

Flags can be provided in ffxConfigureDescFrameGenerationSwapChainRegisterUiResourceDX12 to control the following:

FfxApiUiCompositionFlags memberNote
FFX_FRAMEGENERATION_UI_COMPOSITION_FLAG_USE_PREMUL_ALPHAA bit indicating that we use premultiplied alpha for UI composition.
FFX_FRAMEGENERATION_UI_COMPOSITION_FLAG_ENABLE_INTERNAL_UI_DOUBLE_BUFFERINGA bit indicating that the swapchain should doublebuffer the UI resource.
FfxResource uiColor = ffxGetResource(m_pUiTexture[m_curUiTextureIndex]->GetResource(), L"FSR3_UiTexture", FFX_RESOURCE_STATE_PIXEL_COMPUTE_READ);
ffx::ConfigureDescFrameGenerationSwapChainRegisterUiResourceDX12 uiConfig{};
uiConfig.uiResource = uiColor;
uiConfig.flags = m_DoublebufferInSwapchain ? FFX_FRAMEGENERATION_UI_COMPOSITION_FLAG_ENABLE_INTERNAL_UI_DOUBLE_BUFFERING : 0;
ffx::Configure(m_SwapChainContext, uiConfig);

The final method to handle the UI is to provide a HUDLessColor surface in the FfxFrameGenerationConfig. This surface will get used during frame interpolation to detect the UI and avoid distortion on UI elements. This method has been added for compatibility with engines that can not apply either of the other two options for UI rendering.

FSR3 non async workflow

Different HUDless Formats

An optional structure ffxCreateContextDescFrameGenerationHudless can be linked to the pNext of the ffxCreateContextDescFrameGeneration used at context-creation time to enable the application to use a different hudlessBackBufferformat (IE.RGBA8_UNORM) from backBufferFormat (IE. BGRA8_UNORM).

Distortion Field

When an application uses a distortion effect this can hinder the frame generation algorithm from correctly interpolating the motion of objects. An application can configure the frame generation context with an additional distortionField texture assigned in a ffxConfigureDescFrameGenerationRegisterDistortionFieldResource structure and applied by calling ffxConfigure.

The distortionField texture must contain distortion offset data in a 2-component (ie. RG) format. It is read by FG shaders via Sample. Resource’s xy components encodes [UV coordinate of pixel after lens distortion effect, UV coordinate of pixel before lens distortion].

Dispatch frame generation preparation

Since version 3.1.0, frame generation runs independently of FSR upscaling. To replace the resources previously shared with the upscaler, a new frame generation prepare pass is required.

After the call to ffxConfigure, fill out both a ffxDispatchDescFrameGenerationPrepare structure and a ffxDispatchDescFrameGenerationPrepareCameraInfo structure. The ffxDispatchDescFrameGenerationPrepareCameraInfo should be linked in the pNext field of the ffxDispatchDescFrameGenerationPrepare structure header. Then provide the structure and frame generation context in a call to ffxDispatch.

ffxDispatchDescFrameGenerationPrepare memberNote
frameIDIdentifier used to select internal resources when async support is enabled. Must increment by exactly one (1) for each frame. Any non-exactly-one difference will reset the frame generation logic. Set the frameID to the same value as in the ffxConfigureDescFrameGeneration structure.
flagsZero or combination of values from FfxApiDispatchFrameGenerationFlags.
commandListA command list to record frame generation commands into.
renderSizeThe dimensions used to render game content, dilatedDepth, dilatedMotionVectors are expected to be of ths size.
jitterOffsetThe subpixel jitter offset applied to the camera.
motionVectorScaleThe scale factor to apply to motion vectors.
frameTimeDeltaTime elapsed in milliseconds since the last frame.
unused_resetA (currently unused) boolean value which when set to true, indicates FrameGeneration will be called in reset mode.
cameraNearThe distance to the near plane of the camera.
cameraFarThe distance to the far plane of the camera. This is used only used in case of non infinite depth.
cameraFovAngleVerticalThe camera angle field of view in the vertical direction (expressed in radians).
viewSpaceToMetersFactorThe scale factor to convert view space units to meters.
depthThe depth buffer data.
motionVectorsThe motion vector data.

For fields also found in ffxDispatchDescUpscale, the same input requirements and recommendations apply here.

Set the frameID to the same value as in the configure description.

It is required to specify ffxDispatchDescFrameGenerationPrepareCameraInfo which must contain information about the camera position and orientation within the scene.

ffxDispatchDescFrameGenerationPrepareCameraInfo memberNote
cameraPositionThe camera position in world space.
cameraUpThe camera up normalized vector in world space.
cameraRightThe camera right normalized vector in world space.
cameraForwardThe camera forward normalized vector in world space.

Dispatch frame generation

In order to optimally invoke ffxDispatch for frame-generation it is strongly recommended that the caller supply a callback to frameGenerationCallback along with any necessary context pointer in frameGenerationCallbackUserContext. This callback should execute ffxDispatch using the provided ffxDispatchDescFrameGeneration structure pointer. This will then be written into the proper command-list during the swap-chain presentation and when allowAsyncWorkloads are enabled overlapped with other work.

It is possible, but highly discouraged to dispatch frame-generation manually. This infrastructure exists to support specific game-engine integrations such as the Unreal Engine plugin where dispatching during swap-chain presentation is unsafe. In this case the caller must acquire an interpolation command-buffer using the ffxQueryDescFrameGenerationSwapChainInterpolationCommandListDX12 structure on the swap-chain context and supply this as the commandList in a ffxDispatchDescFrameGeneration which can then be used to dispatch via ffxDispatch. In this case the caller is responsible for understanding whether it is safe to enable allowAsyncWorkloads.

ffxDispatchDescFrameGeneration memberNote
commandListThe command list on which to register render commands.
presentColorThe current presentation color, this will be used as source data.
outputsDestination targets (1 for each frame in numGeneratedFrames).
numGeneratedFramesThe number of frames to generate from the passed in color target.
resetA boolean value which when set to true, indicates the camera has moved discontinuously.
backbufferTransferFunctionThe transfer function use to convert frame generation source color data to linear RGB. One of the values from FfxApiBackbufferTransferFunction.
minMaxLuminanceMin and max luminance values, used when converting HDR colors to linear RGB.
generationRectThe area of the backbuffer that should be used for generation in case only a part of the screen is used e.g. due to movie bars.
frameIDIdentifier used to select internal resources when async support is enabled. Must increment by exactly one (1) for each frame. Any non-exactly-one difference will reset the frame generation logic.

Shutdown

During shutdown, disable UI handling and frame generation in the proxy swapchain and destroy the contexts:

// disable frame generation before destroying context
// also unset present callback, HUDLessColor and UiTexture to have the swapchain only present the backbuffer
m_FrameGenerationConfig.frameGenerationEnabled = false;
m_FrameGenerationConfig.swapChain = GetSwapChain()->GetImpl()->DX12SwapChain();
m_FrameGenerationConfig.presentCallback = nullptr;
m_FrameGenerationConfig.HUDLessColor = FfxApiResource({});
ffx::Configure(m_FrameGenContext, m_FrameGenerationConfig);
ffx::ConfigureDescFrameGenerationSwapChainRegisterUiResourceDX12 uiConfig{};
uiConfig.uiResource = {};
uiConfig.flags = 0;
ffx::Configure(m_SwapChainContext, uiConfig);
// Destroy the contexts
ffx::DestroyContext(m_UpscalingContext);
ffx::DestroyContext(m_FrameGenContext);

Finally, destroy the proxy swap chain by releasing the handle, destroying the context with ffxDestroyContext and re-create the normal DX12 swap chain.

Thread safety

The ffx-api context is not guarranted to be thread safe. In this technique, FrameGenContext and SwapChainContext are not thread safe. Race condition symptom includes access violation error crash, interpolation visual artifact, and infinite wait in Dx12CommandPool destructor when releasing swapchain. It’s not obvious but FrameInterpolationSwapchainDX12::Present() actually accesses SwapChainContext and FrameGenContext (for dispatching Optical Flow and Frame Generation). A race condition occurs if app threads can simutaneously call FrameInterpolationSwapchainDX12::Present() and Dispatch(m_FrameGenContext, DispatchDescFrameGenerationPrepare). Another race condition occurance is if app threads can simutaneously call FrameInterpolationSwapchainDX12::Present() and DestroyContext(SwapChainContext). App could acquire mutex lock before calling ffx functions that access FrameGenContext or SwapChainContext to guarantee at any time there is at most 1 thread that can access the context.

Resource Lifetime

When UiTexture composition mode is used

If FFX_FRAMEGENERATION_UI_COMPOSITION_FLAG_ENABLE_INTERNAL_UI_DOUBLE_BUFFERING is set:

The currentUI gets copied to an internal resource on the game queue The currentUI may be reused on the GFX queue immediately in the next frame

If FFX_FRAMEGENERATION_UI_COMPOSITION_FLAG_ENABLE_INTERNAL_UI_DOUBLE_BUFFERING is not set:

The application is responsible to ensure currentUI persists until composition of the real frame is finished This is typically in the middle of the next frame, so the currentUI should not be used during the next frame. The application must ensure double buffering of the UITexture

When HUDLess composition mode is used:

The HUDLess texture will be used during FrameInterpolation The application is responsible to ensure it persists until FrameInterpolation is complete If allowAsyncWorkloads is true: Frameinterpolation happens on an async compute queue so the HUDLess texture needs to be double buffered by the application If allowAsyncWorkloads is false: Frameinterpolation happens on the game GFX queue, so app can safely modify HUDLess texture in the next frame

When distortionField texture is registered to FrameInterpolation:

The application is responsible to ensure distortionField texture persists until FrameInterpolation is complete If allowAsyncWorkloads is true: Frameinterpolation happens on an async compute queue so the distortionField texture needs to be double buffered by the application If allowAsyncWorkloads is false: Frameinterpolation happens on the game GFX queue, so app can safely modify distortionField texture in the next frame

Debug Checker

Enable debug checker to validate application supplied inputs at dispatch upscale. It is recommended this is enabled only in development builds of game.

Passing FFX_FRAMEGENERATION_ENABLE_DEBUG_CHECKING flag within FfxApiCreateContextFramegenerationFlags will output textual warnings from frame generation to debugger TTY by default. Calling ffxConfigure with fpMessage within a ffxConfigureDescGlobalDebug1 structure to a suitable function allows the application to receive the debug messages issued.

An example of the kind of output that can occur when debug checker observes possible issues is below:

FSR_API_DEBUG_WARNING: ffxDispatchDescFrameGenerationPrepareCameraInfo needs to be passed as linked struct. This is a required input to FSR3.1.4 and onwards for best quality.

Debug output

The frame interpolation API supports several debug visualisation options:

When FFX_FRAMEGENERATION_FLAG_DRAW_DEBUG_TEAR_LINES is set in the flags attribute of ffxDispatchDescFrameGenerationPrepare, the inpainting pass will add bars of changing color on the left and right border of the interpolated image. This will assist visualizing if interpolated frames are getting presented and if the frames are presented with tearing enabled.

When FFX_FRAMEGENERATION_FLAG_DRAW_DEBUG_RESET_INDICATORS is set in the flags attribute of ffxDispatchDescFrameGenerationPrepare, the debug reset indicators will be drawn to the generated output.

When FFX_FRAMEGENERATION_FLAG_DRAW_DEBUG_VIEW is set in the flags attribute of ffxDispatchDescFrameGenerationPrepare, the FrameInterpolationSwapChain will only present interpolated frames and execute an additional pass to render debug data from internal surfaces onto the interpolated frame, to allow you to debug.

When FFX_FRAMEGENERATION_FLAG_DRAW_DEBUG_PACING_LINES is set in the flags attribute of ffxDispatchDescFrameGenerationPrepare, the debug pacing lines will be drawn to the generated output.

Frame interpolation debug overlay

Building the sample

To build the FSR sample, please follow the following instructions:

  1. Download and install the following software developer tool minimum versions:

  2. Open the Visual Studio solution:

    Terminal window
    > <installation path>\Samples\FidelityFX_FSR\dx12\FidelityFX_FSR_2022.sln
  3. First time vcpkg installation

  • If vcpkg has not previously been initialized in Visual Studio, please do so with the following:
    • From the menu, select Tools and then Visual Studio Command Prompt to bring up the terminal.
    • Type vcpkg integrate install and hit enter key.
    • Close and re-open the solution.

Building the project

The Visual Studio solution file (.sln) will contain all projects needed to build the effect sample. To build the projects in the solution, you should click on Build and then Build solution from the menu at the top of Visual Studio. This will build all dependencies of the sample (such as the FidelityFX Cauldron Framework), then build the sample application.

Running the project

To run a project from Visual Studio:

  1. If not already highlighted, select the sample project in the solution explorer.

  2. Right-click the project and select Set as Start-up project.

  3. Right-click the project and select Properties

  4. Under Configuration Properties, click on the Debugging entry.

  5. Set Working Directory to $(TargetDir) for all configurations (Debug and Release).

  6. Click Apply then OK to close the properties panel.

  7. Click on the Build menu item from the toolstrip, and then click on Run.

Limitations

FSR requires a GPU with typed UAV load and R16G16B16A16_UNORM support.

Version history

VersionDate
1.1.12023-11-28
1.1.22024-06-05
1.1.32025-05-08
3.1.52025-08-20

Refer to changelog for more detail on versions.

See also