Copyright © 2018-2022 LunarG, Inc.

GFXReconstruct API Capture and Replay

Desktop Version

This document describes the GFXReconstruct software for capturing and replaying Vulkan API calls on Desktop systems.

Index

1. Capturing API calls
   1. Enabling the Capture Layer
   2. Capture Options
   3. Capture Files
   4. Capture Script
2. Replaying API Calls
   1. Command Line Arguments
   2. Key Controls
   3. Virtual Swapchain
3. Other Capture File Processing Tools
   1. Capture File Info
   2. Capture File Compression
   3. Shader Extraction
   4. Trimmed File Optimization
   5. JSON Lines Conversion
   6. Command Launcher

Capturing API calls

The GFXReconstruct capture layer is a Vulkan layer that intercepts Vulkan API calls and logs them to a GFXReconstruct capture file.

Enabling the Capture Layer

The path to the layer's VkLayer_gfxreconstruct.json file and corresponding VkLayer_gfxreconstruct library must be added to VK_LAYER_PATH environment variable for the Vulkan loader to find the layer.

After VK_LAYER_PATH has been updated, the layer may be enabled through one of the following methods:

• Adding the VK_LAYER_LUNARG_gfxreconstruct name string to the VkInstanceCreateInfo::ppEnabledLayerNames value when calling vkCreateInstance
• Adding the VK_LAYER_LUNARG_gfxreconstruct name string to the list of layer names specified through the VK_INSTANCE_LAYERS environment variable

Setting VK_LAYER_PATH

The VK_LAYER_PATH environment variable can be used to tell the Vulkan loader where to find the GFXReconstruct capture layer. If you are using GFXReconstruct from a Vulkan SDK installation or a Linux package install, there is no need to set VK_LAYER_PATH - the installation process will have set up the GFXReconstruct capture layer so that the Vulkan loader can find and load it.

Setting VK_LAYER_PATH for Windows

If you are not using GFXReconstruct from a Vulkan SDK, use the following as an example of how to update the Windows VK_LAYER_PATH environment variable for the GFXReconstruct capture layer. The example uses the C:\gfxreconstruct path to represent the location of directory containing the GFXReconstruct project source, and the build folder name to represent the sub-directory specified to CMake as the location to place the build binaries. The following command would be executed from the command prompt to add the Debug build of the layer to VK_LAYER_PATH:

set VK_LAYER_PATH=C:\gfxreconstruct\build\layer\Debug;%VK_LAYER_PATH%

Setting VK_LAYER_PATH for Linux

If you are not using GFXReconstruct from the Vulkan SDK or a Linux package, use the following as an example of how to update the Linux VK_LAYER_PATH environment variable for the GFXReconstruct capture layer. The example uses the /gfxreconstruct path to represent the location of directory containing the GFXReconstruct project source, and the build folder name to represent the sub-directory specified to CMake as the location to place the build binaries. The following command would be executed from the command line to add the layer to VK_LAYER_PATH:

export VK_LAYER_PATH=/gfxreconstruct/build/layer:$VK_LAYER_PATH

Enabling the layer with VK_INSTANCE_LAYERS

The VK_INSTANCE_LAYERS environment variable may be used so that the the GFXReconstruct layer is loaded by the Vulkan loader.

Enabling the layer for Windows

The following command would be executed from the command prompt to set the VK_INSTANCE_LAYERS environment variable:

set VK_INSTANCE_LAYERS=VK_LAYER_LUNARG_gfxreconstruct

Enabling the layer for Linux

The following command would be executed from the command line to set the VK_INSTANCE_LAYERS environment variable:

export VK_INSTANCE_LAYERS=VK_LAYER_LUNARG_gfxreconstruct

Capture Options

The GFXReconstruct layer supports several options, which may be enabled through environment variables or a layer settings file.

Windows Options

The following example demonstrates how to set the layer's log level to "warning" from the Windows command prompt:

set GFXRECON_LOG_LEVEL=warning

Linux Options

The following example demonstrates how to set the layer's log level to "warning" from the Linux command line:

export GFXRECON_LOG_LEVEL=warning

Supported Options

Options with the BOOL type accept the following values:

• A case-insensitive string value 'true' or a non-zero integer value indicate true.
• A case-insensitive string value 'false' or a zero integer value indicate false.

The capture layer will generate a warning message for unrecognized or invalid option values.

Memory Tracking Known Issues

There is a known issue with the page guard memory tracking method. The logic behind that method is to apply a memory protection to the guarded/shadowed regions so that accesses made by the user to trigger a segmentation fault which is handled by GFXReconstruct. If the access is made by a system call (like fread()) then there won't be a segmentation fault generated and the function will fail. As a result the mapped region will not be updated.

Settings File

Capture options may also be specified through a layer settings file. The layer settings file will be loaded before the environment variables are processed, allowing environment variables to override individual settings file entries.

The VK_LAYER_SETTINGS_PATH environment variable is used to enable a settings file. The environment variable may be set as either the path to the folder containing a file named vk_layer_settings.txt or the full path to a file with a custom name. When set to a folder, the capture layer will try to open a file in that folder named vk_layer_settings.txt. When set to a file, the capture layer will try to open a file with the specified name.

The settings file may be combined with settings files for other layers. The capture layer will ignore entries that do not start with the 'lunarg_gfxreconstruct.' prefix.

A sample layer settings file, documenting each available setting, can be found in the GFXReconstruct GitHub repository at layer/vk_layer_settings.txt. Most binary distributions of the GFXReconstruct software will also include a sample settings file.

Selecting Settings for the page_guard Memory Tracking Mode

The default settings selected for the page_guard memory tracking mode are the settings that are most likely to work on a given platform, but may not provide the best performance for all cases.

For Windows, setting GFXRECON_PAGE_GUARD_EXTERNAL_MEMORY to true is recommended. If capture does not work with this setting, or a different operating system is being used, try the default settings.

If capture performs poorly with the the default settings, try setting GFXRECON_PAGE_GUARD_PERSISTENT_MEMORY to true.

If corruption is observed during capture, try setting GFXRECON_PAGE_GUARD_ALIGN_BUFFER_SIZES to true. If this does not help, try setting GFXRECON_PAGE_GUARD_SEPARATE_READ to false.

Capture Files

Capture files are created on the first call to vkCreateInstance, when the Vulkan loader loads the capture layer, and are closed on vkDestroyInstance, when the last active instance is destroyed and the layer is unloaded.

If multiple instances are active concurrently, only one capture file will be created. If multiple instances are active consecutively (i.e. an instance is created and destroyed before the next instance is created), the creation of each instance will generate a new file. For applications that create multiple instances consecutively, it will be necessary to enable capture file timestamps to prevent each new instance from overwriting the file created by the previous instance.

If the layer fails to open the capture file, it will make the call to vkCreateInstance fail, returning VK_ERROR_INITIALIZATION_FAILED.

Specifying Capture File Location

The capture file's save location can be specified by setting the GFXRECON_CAPTURE_FILE environment variable, described above in the Layer Options section.

Timestamps

When capture file timestamps are enabled, a timestamp with an ISO 8601-based format will be added to the name of every file created by the layer. The timestamp is generated when the capture file is created by the layer's vkCreateInstance function and is added to the base filename specified through the GFXRECON_CAPTURE_FILE environment variable. Timestamps have the form:

_yyyymmddThhmmss

where the lower-case letters stand for: Year, Month, Day, Hours, Minutes, Seconds. The T is a designator that separates the date and time components. Time is reported for the local timezone and is specified with the 24-hour format.

The following example shows a timestamp that was added to a file that was originally named gfxrecon_capture.gfxr and was created at 2:35 PM on November 25, 2018: gfxrecon_capture_20181125T143527.gfxr

Capture Script

The gfxrecon-capture.py tool is a convenience script that can be used to start a capture and specify the capture options using a single command.

usage: gfxrecon-capture.py [-h]
                           [-w dir]
                           [-o captureFile]
                           [-f captureFrames]
                           [--no-file-timestamp]
                           [--trigger {F1-F12,TAB,CTRL}]
                           [--compression-type {LZ4,ZLIB,ZSTD,NONE}]
                           [--file-flush]
                           [--log-level {debug,info,warn,error,fatal}]
                           [--log-file <file>]
                           [--memory-tracking-mode {page_guard,assisted,unassisted}]
                           <program> [<programArgs>]

Create a capture of a Vulkan program.

positional arguments:
  <program> [<program args>]
                        Program to capture, optionally followed by program
                        arguments

optional arguments:
  -h, --help            show this help message and exit
  -w <dir>, --working-dir <dir>
                        Set CWD to this directory before running the program
  -o <captureFile>, --capture-file <captureFile>
                        Name of the capture file, default is
                        gfxrecon_capture.gfxr
  -f <captureFrames>, --capture-frames <captureFrames>
                        List of frames to capture, default is all frames
  --no-file-timestamp   Do not add a timestamp to the capture file name
  --trigger {F1,F2,F3,F4,F5,F6,F7,F8,F9,F10,F11,F12,TAB,CTRL}
                        Specify a hotkey to start/stop capture
  --compression-type {LZ4,ZLIB,ZSTD,NONE}
                        Specify the type of compression to use in the capture
                        file, default is LZ4
  --file-flush          Flush output stream after each packet is written to
                        capture file
  --log-level {debug,info,warn,error,fatal}
                        Specify highest level message to log, default is info
  --log-file <logFile>  Write log messages to a file at the specified path.
                        Default is: Empty string (file logging disabled)
  --memory-tracking-mode {page_guard,assisted,unassisted}
                        Method to use to track changes to memory mapped objects:
                           page_guard: use guard pages to track changes (default)
                           assisted:   application will call vkFlushMappedMemoryRanges
                                       for memory to be written to the capture file
                           unassisted: all mapped memory will be written to the
                                       capture file during VkQueueSubmit and VkUnmapMemory

Most of the options for gfxrecon-capture.py result in the script setting the appropriate capture layer environment variable, then invoking the program to be captured. Environment variables not set by gfxrecon-capture.py can be set manually before running gfxrecon-capture.py and they will be detected by the capture layer.

The gfxrecon-capture.py tool is a Python3 script. In order to use it, a Python3 interpreter must first be installed. Once Python3 is installed, you should be able to invoke gfxrecon-capture.py by simply typing:

gfxrecon-capture.py -o vkcube.gfxr vkcube

On Windows, after installing Python3, be sure to associate the .py file extension with the Python3 interpreter before you run the script.

Replaying API Calls

The GFXReconstruct Replay tool, gfxrecon-replay, can be used to replay files captured with or generated by other GFXReconstruct components.

Command Line Arguments

The gfxrecon-replay tool for desktop accepts the following command line arguments:

gfxrecon-replay         [-h | --help] [--version] [--gpu <index>]
                        [--pause-frame <N>] [--paused] [--sync] [--screenshot-all]
                        [--screenshots <N1(-N2),...>] [--screenshot-format <format>]
                        [--screenshot-dir <dir>] [--screenshot-prefix <file-prefix>]
                        [--sfa | --skip-failed-allocations] [--replace-shaders <dir>]
                        [--opcd | --omit-pipeline-cache-data] [--wsi <platform>]
                        [--surface-index <N>] [--remove-unsupported]
                        [-m <mode> | --memory-translation <mode>]
                        [--use-captured-swapchain-indices]
                        [--log-level <level>] [--log-file <file>] [--log-debugview]
                        <file>

Required arguments:
  <file>                Path to the capture file to replay.

Optional arguments:
  -h                    Print usage information and exit (same as --help).
  --version             Print version information and exit.
  --log-level <level>   Specify highest level message to log. Options are:
                        debug, info, warning, error, and fatal. Default is info.
  --log-file <file>     Write log messages to a file at the specified path.
                        Default is: Empty string (file logging disabled).
  --log-debugview       Log messages with OutputDebugStringA. Windows only.
  --gpu <index>         Use the specified device for replay, where index
                        is the zero-based index to the array of physical devices
                        returned by vkEnumeratePhysicalDevices.  Replay may fail
                        if the specified device is not compatible with the
                        original capture devices.
  --gpu-group <index>   Use the specified device group for replay, where index
                        is the zero-based index to the array of physical device group
                        returned by vkEnumeratePhysicalDeviceGroups.  Replay may fail
                        if the specified device group is not compatible with the
                        original capture device group.
  --pause-frame <N>     Pause after replaying frame number N.
  --paused              Pause after replaying the first frame (same
                        as --pause-frame 1).
  --screenshot-all
                        Generate screenshots for all frames.  When this
                        option is specified, --screenshots is ignored.
  --screenshots <N1[-N2][,...]>
                        Generate screenshots for the specified frames.
                        Target frames are specified as a comma separated
                        list of frame ranges.  A frame range can be specified
                        as a single value, to specify a single frame, or as
                        two hyphenated values, to specify the first and last
                        frames to process.  Frame ranges should be specified in
                        ascending order and cannot overlap.  Note that frame
                        numbering is 1-based (i.e. the first frame is frame 1).
                        Example: 200,301-305 will generate six screenshots.
  --screenshot-format <format>
                        Image file format to use for screenshot generation.
                        Available formats are:
                            bmp         Bitmap file format.  This is the default format.
  --screenshot-dir <dir>
                        Directory to write screenshots.  Default is the current
                        working directory.
  --screenshot-prefix <file-prefix>
                        Prefix to apply to the screenshot file name.  Default is
                        "screenshot", producing file names similar to
                        "screenshot_frame8049.bmp".
  --sfa                 Skip vkAllocateMemory, vkAllocateCommandBuffers, and
                        vkAllocateDescriptorSets calls that failed during
                        capture (same as --skip-failed-allocations).
  --replace-shaders <dir> Replace the shader code in each CreateShaderModule
                        with the contents of the file <dir>/sh<handle_id> if found, where
                        <handle_id> is the handle id of the CreateShaderModule call.
                        See gfxrecon-extract.
  --opcd                Omit pipeline cache data from calls to
                        vkCreatePipelineCache and skip calls to
                        vkGetPipelineCacheData (same as
                        --omit-pipeline-cache-data).
  --wsi <platform>      Force replay to use the specified wsi platform.
                        Available platforms are: auto,win32,xlib,xcb,wayland
  --surface-index <N>   Restrict rendering to the Nth surface object created.
                        Used with captures that include multiple surfaces.  Default
                        is -1 (render to all surfaces).
  --sync                Synchronize after each queue submission with vkQueueWaitIdle.
  --remove-unsupported  Remove unsupported extensions and features from instance
                        and device creation parameters.
  -m <mode>             Enable memory translation for replay on GPUs with memory
                        types that are not compatible with the capture GPU's
                        memory types.  Available modes are:
                            none        No memory translation is performed.  This
                                        is the default behavior.
                            remap       Attempt to map capture memory types to
                                        compatible replay memory types, without
                                        altering memory allocation behavior.
                            realign     Adjust memory allocation sizes and
                                        resource binding offsets based on
                                        replay memory properties.
                            rebind      Change memory allocation behavior based
                                        on resource usage and replay memory
                                        properties.  Resources may be bound
                                        to different allocations with different
                                        offsets.  Uses VMA to manage allocations
                                        and suballocations.
  --use-captured-swapchain-indices
                        Use the swapchain indices stored in the capture directly on the swapchain
                        setup for replay. The default without this option is to use a Virtual Swapchain
                        of images which match the swapchain in effect at capture time and which are
                        copied to the underlying swapchain of the implementation being replayed on.

Key Controls

The gfxrecon-replay tool for Desktop supports the following key controls:

Virtual Swapchain

During replay, swapchain indices for present can be different from captured indices. Causes for this can include the swapchain image count differing between capture and replay, and vkAcquireNextImageKHR returning a different pImageIndex at replay to the one that was captured. These issues can cause unexpected rendering or even crashes.

Virtual Swapchain insulates higher layers in the Vulkan stack from these problems by creating a set of images, exactly matching the swapchain configuration at capture time, which it exposes for them to render into. Before a present, it copies the virtual image to a target swapchain image for display. Since this issue can happen in many situations, virtual swapchain is the default setup. If the user wants to bypass the feature and use the captured indices to present directly on the swapchain of the replay implementation, they should add the --use-captured-swapchain-indices option when invoking gfxrecon-replay.

Other Capture File Processing Tools

Capture File Info

The gfxrecon-info tool prints statistics for a GFXReconstruct capture file, including information about the application, the physical device , device memory allocation, and device pipelines.

gfxrecon-info - Print statistics for a GFXReconstruct capture file.

Usage:
  gfxrecon-info [-h | --help] [--version] <file>

Required arguments:
  <file>      The GFXReconstruct capture file to be processed.

Optional arguments:
  -h          Print usage information and exit (same as --help).
  --version   Print version information and exit.

Capture File Compression

The gfxrecon-compress tool compresses or decompresses GFXReconstruct capture files. It can also be used to change the compression format used in a capture file.

gfxrecon-compress - A tool to compress/decompress GFXReconstruct capture files.

Usage:
  gfxrecon-compress [-h | --help] [--version] <input_file> <output_file> <compression_format>

Required arguments:
  <input_file>    Path to the input file to process.
  <output_file>   Path to the output file to generate.
  <compression_format>  Compression format to apply to the output file.
                        Options are:
                          LZ4  - Use LZ4 compression.
                          ZLIB - Use zlib compression.
                          ZSTD - Use Zstandard compression.
                          NONE - Remove compression.

Optional arguments:
  -h              Print usage information and exit (same as --help).
  --version       Print version information and exit.

Shader Extraction

The gfxrecon-extract tool extracts all shaders in a GFXReconstruct capture file. The extracted shaders are placed into a specified directory.

gfxrecon-extract - Extract shaders from a GFXReconstruct capture file.

Usage:
  gfxrecon-extract [-h | --help] [--version] [--dir <dir>] <file>

Optional arguments:
  -h          Print usage information and exit (same as --help).
  --version   Print version information and exit.
  --dir <dir> Place extracted shaders into directory <dir>. Otherwise
              use <file>.shaders in working directory. Create directory
              if necessary. Each shader is placed in individual file
              named sh<handle_id> where handle_id is handle id of the
              CreateShaderModule call. See gfxrecon-replay --replace-shaders.
Required arguments:
  <file>      The GFXReconstruct capture file to be processed.

Trimmed File Optimization

The gfxrecon-optimize tool removes unused buffer and image initialization data from trimmed capture files.

For trimmed capture files, a snapshot of the Vulkan API state is written at the start of the file. This state snapshot includes the data for all buffers and images that were live at the time that capture started. Some of the buffer and image objects captured in the state snapshot may go unreferenced by the captured frames and their data can be removed from the capture file. The gfxrecon-optimize tool will process a trimmed file to identify buffer and image objects that were initialized in the state snapshot, but were not used by any of the captured frames, and generate a new capture file that omits the data for these unused buffer and image objects.

gfxrecon-optimize - Remove unused resource initialization data from trimmed
                    GFXReconstruct capture files.

Usage:
  gfxrecon-optimize [-h | --help] [--version] <input-file> <output-file>

Required arguments:
  <input-file>          The trimmed GFXReconstruct capture file to be
                        processed.
  <output-file>         The name of the new GFXReconstruct capture file to be
                        created.

Optional arguments:
  -h                    Print usage information and exit (same as --help).
  --version             Print version information and exit.

JSON Lines Conversion

The gfxrecon-convert tool converts a capture file into a series of JSON documents, one per line following the JSON Lines standard. The JSON document on each line is designed to be parsed by tools such as simple Python scripts as well as being useful for inspection by eye after pretty printing, for example by piping through a command-line tool such as jq. For these post-processing use cases, gfxrecon-convert can be used to stream from binary captures directly, without having to save the intermediate JSON files to storage. Because each JSON object is on its own line, line oriented tools such as grep, sed, head, and split can be applied ahead of JSON-aware ones which are heavier-weight to reduce their workload on large captures.

The file begins with a header object containing some metadata, followed by a series of objects representing the sequence of Vulkan calls stored in the capture. More details of the file format can be found in the tool's README.

gfxrecon-convert - A tool to convert GFXReconstruct capture files to text.

Usage:
  gfxrecon-convert [-h | --help] [--version] <file>

Required arguments:
  <file>		Path to the GFXReconstruct capture file to be converted
                to text.

Optional arguments:
  -h			        Print usage information and exit (same as --help).
  --version		        Print version information and exit.
  --output file         'stdout' or a path to a file to write JSON output
                        to. Default is the input filepath with "gfxr" replaced
                        by "jsonl".
  --no-debug-popup      Disable the 'Abort, Retry, Ignore' message box
                        displayed when abort() is called (Windows debug only).

Command Launcher

The gfxrecon.py tool is a utility that can be used to launch all of the GFXReconstruct commands.

usage: gfxrecon.py [-h] command ...

GFXReconstruct utility launcher.

positional arguments:
  command     Command to execute. Valid options are [capture, compress, convert,
              extract, info, optimize, replay]
  args        Command-specific argument list. Specify -h after command name for
              command help.

optional arguments:
  -h, --help  show this help message and exit

The gfxrecon.py tool is a Python3 script. In order to use it, a Python3 interpreter must first be installed. Once Python3 is installed, you should be able to invoke gfxrecon.py by simply typing:

gfxrecon.py capture -o vkcube.gfxr vkcube

On Windows, after installing Python3, be sure to associate the .py file extension with the Python3 interpreter before you run the script.