Dumping resources

Index

  1. Introduction
    1. General
    2. Block index and object ID
    3. Rules for providing command indices
    4. Secondary command buffers
    5. Simple examples
    6. Index vector dimensionality
    7. A more complex example
    8. Culling dumped descriptors
  2. Vulkan dump resources options
    1. gfxrecon-replay command line parameters
    2. Input json options
  3. Output
    1. Json file output
    2. Image file output
    3. Buffer file output

Introduction

General

GFXReconstruct offers the capability to dump resources when replaying a capture file. These resources can be:

  1. Render targets

    All images used as render target attachments (both as color and depth attachments) either in render passes or with dynamic rendering.

  2. Vertex and index buffers

    Buffers bound as vertex and index buffers referenced by draw calls which have been marked for dumping.

  3. Results of compute (vkCmdDispatch and its variants) and ray tracing (vkCmdTraceRays) shaders.

    All result images and buffers used as descriptor bindings by dispatch and ray tracing shaders.

  4. Descriptor bindings used as inputs in all shader stages.

    All images and buffers used as descriptor bindings by draw calls, dispatch and ray tracing shaders.

The resources are dumped into files and can either be image files (bmp or png) or binary files.

Dumping can take place only while replaying a capture file either on desktop with the gfxrecon-replay tool or when replaying a capture file on Android with the replay application.

In order to enable the dump resources feature the DrawCalls and/or Dispatch, and/or TraceRays for which the related resources is desired to be dumped, need to be specified in some way. This is can be done by using the block index with which these commands are recorded inside the capture file (for more details see [Block index])(#block-index-and-object-id).

It is possible to dump resources from multiple draw calls and/or Dispatch/TraceRays in a single run by specifying multiple indices.

Block index and object ID

Each command (either a Vulkan command issued by the application or a meta command generated by GFXReconstruct internally) stored in the capture file can be identified by a unique, monotonically increasing number that each command is assigned. This is called the command's block index. In order to specify commands for dumping their resources this index needs to be specified for each command.

The simplest way to find the command block index for each Vulkan command is to use the gfxrecon-convert tool on a capture file which will convert the capture into a human readable json file. Use the --format jsonl option to produce a json file with one command per line. The resulting jsonl file looks something like this:

{"index":301,"function":{"name":"vkBeginCommandBuffer","thread":2,"return":"VK_SUCCESS","args":{"commandBuffer":80, ... }}},
{"index":302,"function":{"name":"vkCmdBeginRenderPass","thread":2,"cmd_index":1,"args":{"commandBuffer":80, ... }}},
{"index":303,"function":{"name":"vkCmdBindPipeline","thread":2,"cmd_index":2,"args":{"commandBuffer":80, ... }}},
{"index":304,"function":{"name":"vkCmdBindDescriptorSets","thread":2,"cmd_index":3,"args":{"commandBuffer":80, ... }}},
{"index":305,"function":{"name":"vkCmdSetViewport","thread":2,"cmd_index":4,"args":{"commandBuffer":80, ... }}},
{"index":306,"function":{"name":"vkCmdSetScissor","thread":2,"cmd_index":5,"args":{"commandBuffer":80, ... }}},
{"index":307,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":80, ... }}},

The block index is the index key used in each entry.

In a similar manner each vulkan object is assigned an object ID inside the capture file. When an object is referenced by a command in capture file, the object is referenced by its ID. In the above example the VkCommandBuffer with object ID 80 is referenced and is common between all commands.

Rules for providing command indices

Apart from the DrawCall/Dispatch/TraceRays indices, indices for several other Vulkan commands need to specified. In summary, the commands that can be specified are the following:

  1. DrawCalls This includes the indices of vkCmdDraw and all supported variants:

    • vkCmdDraw
    • vkCmdDrawIndexed
    • vkCmdDrawIndirect
    • vkCmdDrawIndexedIndirect
    • vkCmdDrawIndirectCount
    • vkCmdDrawIndexedIndirectCount
    • vkCmdDrawIndirectCountKHR
    • vkCmdDrawIndexedIndirectCountKHR
  2. Dispatch This includes the indices of vkCmdDispatch and all supported variants:

    • vkCmdDispatch
    • vkCmdDispatchIndirect
  3. Trace rays This includes the indices of vkCmdTraceRaysKHR and all supported variants:

    • vkCmdTraceRaysKHR
    • VkTraceRaysIndirectCommandKHR

Depending on the type of the commands that dumping is requested for, additional command indices need to be specified. These commands are:

  1. BeginCommandBuffer This is the index of the vkBeginCommandBuffer inside which the indices of the dump-able commands are provided.

  2. Render pass Render pass indices are required only for draw calls. All render pass indices which surround the provided draw calls must be specified - the indices for vkCmdBeginRenderPass, vkCmdNextSubpass (if any), and vkCmdEndRenderPass. In case of dynamic rendering the indices of vkCmdBeginRendering and vkCmdEndRendering must be provided instead.

  3. QueueSubmit The index of the vkQueueSubmit (or vkQueueSubmit2) in which the command buffer that includes the desired commands are submitted needs to be provided.

  4. ExecuteCommands Dumping resources from commands that are recorded in secondary command buffers requires different handling. The secondary's BeginCommandBuffer index must be specified like with the primary command buffers. The index of the vkCmdExecuteCommands from which the specific commands should be dumped needs to be specified in the ExecuteCommands json array along with BeginCommandBuffer of the secondary that is desired to be dumped.

Simple examples

Assuming the following imaginary excerpt from a capture file that contains the following commands:

{"index":301,"function":{"name":"vkBeginCommandBuffer", ... } },
{"index":302,"function":{"name":"vkCmdBeginRenderPass", ... } },
{"index":303,"function":{"name":"vkCmdBindPipeline", ... } },
{"index":304,"function":{"name":"vkCmdBindDescriptorSets", ... } },
{"index":305,"function":{"name":"vkCmdSetViewport", ... } },
{"index":306,"function":{"name":"vkCmdSetScissor", ... } },
{"index":307,"function":{"name":"vkCmdDraw", ... } },
{"index":308,"function":{"name":"vkCmdDraw", ... } },
{"index":309,"function":{"name":"vkCmdDraw", ... } },
{"index":310,"function":{"name":"vkCmdDraw", ... } },
{"index":311,"function":{"name":"vkCmdDraw", ... } },
{"index":312,"function":{"name":"vkCmdDraw", ... } },
{"index":313,"function":{"name":"vkCmdEndRenderPass", ... } },
{"index":314,"function":{"name":"vkEndCommandBuffer", ... } },
{"index":315,"function":{"name":"vkQueueSubmit", ... } },

It is possible to dump the depth and color attachments of all vkCmdDraw commands by providing the following json input file:

{
    "BeginCommandBuffer": [ 301 ],
    "Draw": [ [ 307, 308, 309, 310, 311, 312 ] ],
    "RenderPass": [ [ [ 302, 313 ] ] ],
    "QueueSubmit": [ 315 ]
}

An example involving secondary command buffers

{"index":754,"function":{"name":"vkBeginCommandBuffer","args":{"commandBuffer":230 ... }}}, {"index":761,"function":{"name":"vkCmdDrawIndexed","args":{"commandBuffer":230, ...}}}

{"index":736,"function":{"name":"vkBeginCommandBuffer","args":{"commandBuffer":226, ...}}}, {"index":3948,"function":{"name":"vkCmdBeginRenderPass","args":{"commandBuffer":226, ...}}}, {"index":3949,"function":{"name":"vkCmdExecuteCommands","args":{"commandBuffer":226,"commandBufferCount":357,"pCommandBuffers":[227,230,232,233,...]}}} {"index":3950,"function":{"name":"vkCmdEndRenderPass","args":{"commandBuffer":226}}} {"index":3952,"function":{"name":"vkQueueSubmit","args":{"queue":6,"submitCount":1,"pSubmits":[{"commandBufferCount":1,"pCommandBuffers":[226]}...]...}}}

In order to dump the draw call from the secondary command buffer 230 the following json input file should be provided:

{
    "BeginCommandBuffer": [ 754, 736 ],
    "Draw": [ [ 761 ], [ ] ],
    "RenderPass": [ [ [ ] ], [ [ 3948, 3950 ] ] ],
    "ExecuteCommands": [ [ [ ] ], [ [ 3949, 754 ] ] ],
    "QueueSubmit": [ 3952 ]
}

Index vector dimensionality

Indices are provided in GFXReconstruct as vectors of indices. Each index vector, depending on the type of the command it describes can vary and is important, otherwise errors will be generated while parsing the input or the expected commands will not be dumped. Each vector's dimensionality is the following:

Commands recorded in multiple command buffers can be dumped in a single run. For each command buffer the index of the vkBeginCommandBuffer must be provided. The index of the vkQueueSubmit in which the command buffer is submitted must be provided. Both these vectors must have the same number of indices.

These vectors are two dimensional. The first dimension corresponds to BeginCommandBuffer each vector belongs to. I.e.:

{
    "BeginCommandBuffer" :  [ 10, 20 ],
    "TraceRays":            [ [ 220, 230, 240 ], [] ],
    "Dispatch" :            [ [], [ 250, 260, 270 ] ],
    "QueueSubmit" :         [ 350, 360 ]
}

In the example above the vkCmdTraceRays [ 220, 230, 240 ] belong to vkBeginCommandBuffer with block index 10 and are submitted in vkQueueSubmit with block index 350.

vkCmdDispatch with indices [ 250, 260, 270 ] belong to command buffer with vkBeginCommandBuffer with block index 20 and are submitted in vkQueueSubmit with block index 360.

Inside a command buffer, Vulkan allows multiple render passes with multiple sub-passes. In order to support that for multiple command buffers a 3D array is required.

A more complex example

A hypothetical json output of the gfxrecon-convert tool could look something like the following:

{"index":10,"function":{"name":"vkBeginCommandBuffer","thread":2,"return":"VK_SUCCESS","args":{"commandBuffer":59, ... }}},
{"index":11,"function":{"name":"vkBeginCommandBuffer","thread":2,"return":"VK_SUCCESS","args":{"commandBuffer":60, ... }}},
{"index":12,"function":{"name":"vkCmdBeginRenderPass","thread":2,"cmd_index":1,"args":{"commandBuffer":59, ... }}},
{"index":13,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":14,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":15,"function":{"name":"vkCmdNextSubpass","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":16,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":17,"function":{"name":"vkCmdDraw","thread":3,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":18,"function":{"name":"vkCmdEndRenderPass","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":19,"function":{"name":"vkCmdBeginRenderPass","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":20,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":21,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":22,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":23,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":24,"function":{"name":"vkCmdEndRenderPass","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":25,"function":{"name":"vkEndCommandBuffer","thread":2,"return":"VK_SUCCESS","args":{"commandBuffer":59}}}
{"index":26,"function":{"name":"vkCmdBeginRenderPass","thread":2,"cmd_index":1,"args":{"commandBuffer":60, ... }}},
{"index":27,"function":{"name":"vkCmdDrawIndexed","thread":2,"cmd_index":6,"args":{"commandBuffer":60, ... }}}
{"index":28,"function":{"name":"vkCmdDrawIndexed","thread":2,"cmd_index":6,"args":{"commandBuffer":60, ... }}}
{"index":29,"function":{"name":"vkCmdDrawIndexed","thread":2,"cmd_index":6,"args":{"commandBuffer":60, ... }}}
{"index":30,"function":{"name":"vkCmdEndRenderPass","thread":2,"cmd_index":6,"args":{"commandBuffer":60, ... }}}
{"index":31,"function":{"name":"vkEndCommandBuffer","thread":2,"return":"VK_SUCCESS","args":{"commandBuffer":60}}}
...
{"index":50,"function":{"name":"vkQueueSubmit", ... ,"commandBufferCount":1,"pCommandBuffers":[59], ... }},
{"index":51,"function":{"name":"vkQueueSubmit", ... ,"commandBufferCount":1,"pCommandBuffers":[60], ... }},

The indices submitted to gfxrecon-replay for dumping are the following:

{
    "BeginCommandBuffer" : [ 10, 11 ],
    "Draw" :               [ [ 13, 14, 16, 17, 20, 21, 22, 23 ],
                             [ 27, 28, 29 ] ],
    "RenderPass" :         [ [ [ 12, 15, 18 ], [ 19, 24 ] ], [ [ 26, 30 ] ] ],
    "QueueSubmit" :        [ 50, 51 ]
}

In this example two command buffers are submitted for dumping, one with object ID 59 and one with object ID 60.

Culling dumped descriptors

There is an option to ask for specific descriptors and image subresources to be dumped instead of dumping all bound descriptors. This can be done by providing for each command index the 1) descriptor set index, 2) binding index and, 3) array index of the descriptors to be dumped for the specific command. This is an example:

{
    "BeginCommandBuffer": [
        2525
    ],
    "QueueSubmit": [
        2548
    ],
    "Draw": [
        [
            {
                "Index": 2533,
                "Descriptors": [
                    {
                        "Set": 0,
                        "Binding": 1,
                        "ArrayIndex": 0
                    },
                    {
                        "Set": 0,
                        "Binding": 0,
                        "ArrayIndex": 0
                    }
                ]
            },
            2537
        ]
    ],
    "RenderPass": [
        [
            [
                2526,
                2538
            ]
        ]
    ]
}

Image descriptors can be fine grained further by specifying the desired subresources with a VkImageSubresourceRange like this:

"Index": 2533,
"Descriptors": [
    {
        "Set": 0,
        "Binding": 1,
        "ArrayIndex": 0,
        "SubresourceRange": {
            "AspectMask": "VK_IMAGE_ASPECT_DEPTH_BIT | VK_IMAGE_ASPECT_STENCIL_BIT",
            "BaseMipLevel": 2,
            "LevelCount": 1,
            "BaseArrayLayer": 2,
            "LayerCount": 1
        }
    }
]

VK_REMAINING_MIP_LEVELS and VK_REMAINING_ARRAY_LAYERS can be used in LevelCount and LayerCount respectively.

Vulkan dump resources options

gfxrecon-replay command line parameters

Dump resources feature can be controled in several ways. This is done either by providing the appropriate options via the command line or using the input json. The majority of the options are only available via the input json. The command line options, which can be passed to either to the gfxrecon-replay tool or to the Android application through the gfxrecon.py script are the following:

Input json options

The rest of the dump resources options are available through the input json file. They are expected to be inside a json object named DumpResourcesOptions. The possible options are the following:

A json example specifying the above options:

{
  "DumpResourcesOptions": {
      "Scale": 0.2,
      "OutputDir": "",
      "ColorAttachmentIndex": -1,
      "OutputDir": "",
      "ImageFormat": "png",
      "DumpBeforeCommand": false,
      "DumpDepth": false,
      "DumpVertexIndexBuffer": false,
      "DumpAllDescriptors": true,
      "DumpAllImageSubresources": true,
      "DumpRawImages": false,
      "DumpSeparateAlpha": false,
      "DumpUnusedVertexBindings": false,
      "BinaryFileCompressionType": "None",
      "DumpBuildAccelerationStructuresInputBuffers": true
  },

  "BeginCommandBuffer": [ 448 ],
  "TraceRays": [[ 451 ]],
  "QueueSubmit": [ 466 ]
}

Output

Json file output

The dump resources feature generates a number of output files. A json file with an entry for each set of resources that are dumped is generated, as well as image and binary files. The json output file name is derived from the input file name, with the ".gfxr" extension replaced with "_rd.json", i.e. an input file name of "vulkanCapture.gfxr" will result in a json output file name of "vulkanCapture_rd.json".

The output json consists of 4 main entries:

  1. Header

The header contains the path to the capture file, GFXR and vulkan version and the dump resources parameters used when replaying.

  1. Draw call commands

The draw calls are listed in an array. Each draw call entry contains information regarding:

  1. Dispatch commands and Trace Rays commands

Here is an example of a json output file:

[
{
  "header": {
    "source-path": "vulkanCapture.gfxr",
    "gfxrecon-version": "1.0.3"
    "vulkan-version": "1.3.275"
    "dumpResourcesOptions": {
      "scale": 1.0,
      "dumpResourcesOutputDir": "",
      "dumpResourcesColorAttachmentIndex": -1,
      "dumpResourcesBefore": false,
      "dumpResourcesDumpDepth": true,
      "dumpResourcesDumpVertexIndexBuffer": true,
      "dumpResourcesDumpImmutableResources": true,
      "dumpResourcesDumpAllImageSubresources": false
    }
  }
},
{
   "drawCallCommands": [
    {
      "drawIndex": 407,
      "beginCommandBufferIndex": 399,
      "queueSubmitIndex": 461,
      "parameters": {
        "drawCallType": "vkCmdDraw",
        "vertexCount": 24576,
        "instanceCount": 1,
        "firstVertex": 0,
        "firstInstance": 0
      },
      "colorAttachments": [
        {
          "imageId": 15,
          "format": "VK_FORMAT_B8G8R8A8_UNORM",
          "type": "VK_IMAGE_TYPE_2D",
          "aspect": "COLOR",
          "dimensions": [
            1280,
            720,
            1
          ],
          "mipLevel": 0,
          "arrayLayer": 0,
          "file": "Draw_407_qs_461_bcb_399_att_0_aspect_color.bmp"
        }
      ],
      "depthAttachments": [
        {
          "imageId": 32,
          "format": "VK_FORMAT_D32_SFLOAT_S8_UINT",
          "type": "VK_IMAGE_TYPE_2D",
          "aspect": "DEPTH",
          "dimensions": [
            1280,
            720,
            1
          ],
          "mipLevel": 0,
          "arrayLayer": 0,
          "file": "Draw_407_qs_461_bcb_399_depth_att_aspect_depth.bmp"
        },
     ...
      ],
      "vertexBuffers": [
        {
          "bufferId": 75,
          "vertexBufferBinding": 0,
          "file": "VertexBuffers_qs_461_bcb_399_dc_407_binding_0.bin"
        }
      ],
      "descriptors": {
        "vertex": [
          {
            "type": "VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER",
            "set": 0,
            "binding": 2,
            "arrayIndex": 0,
            "descriptor": {
              "bufferId": 79,
              "file": "Buffer_79_qs_461_bcb_399_rp_0.bin"
            }
          }
        ],
        "fragment": [
          {
            "type": "VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER",
            "set": 0,
            "binding": 1,
            "arrayIndex": 0,
            "descriptor": [
              {
                "imageId": 68,
                "format": "VK_FORMAT_R8G8B8A8_UNORM",
                "type": "VK_IMAGE_TYPE_2D",
                "aspect": "COLOR",
                "dimensions": [
                  256,
                  1,
                  1
                ],
                "mipLevel": 0,
                "arrayLayer": 0,
                "file": "Image_68_qs_461_bcb_399_rp_0_aspect_color.bmp"
              }
            ]
          },
          ...
        ]
      }
    },
  ]
},
{
  "dispatchCommands": [
    {
      "dispatchIndex": 294,
      "beginCommandBufferIndex": 290,
      "queueSubmitIndex": 723,
      "parameters": {
        "dispatchType": "vkCmdDispatch",
        "groupCountX": 96,
        "groupCountY": 1,
        "groupCountZ": 1
      },
      "outputs": {
        "buffers": [
          {
            "type": "VK_DESCRIPTOR_TYPE_STORAGE_BUFFER",
            "set": 0,
            "binding": 0,
            "arrayIndex": 0,
            "bufferId": 75,
            "file": "Dispatch_294_qs_723_bcb_290_stage_compute_set_0_binding_0_index_0_buffer.bin"
          }
        ]
      },
      "descriptors": [
        {
          "type": "VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER",
          "set": 0,
          "binding": 1,
          "arrayIndex": 0,
          "descriptor": {
            "bufferId": 90,
            "file": "Buffer_90_qs_723_bcb_290.bin"
          }
        },
        {
          "type": "VK_DESCRIPTOR_TYPE_STORAGE_BUFFER",
          "set": 0,
          "binding": 0,
          "arrayIndex": 0,
          "descriptor": {
            "bufferId": 75,
            "file": "Buffer_75_qs_723_bcb_290.bin"
          }
        }
      ]
    },
    ...
  ]
}]

Image file output

The image files that created are either images (in one of the supported image formats), or raw binary files (.bin). Raw binary files are created when the dumped resource is an image with a format which cannot be converted into a plain 32bit RGBA layout.

Buffer file output

All buffers are dumped as raw binary files (.bin).