Dumping resources
Index
Introduction
General
GFXReconstruct offers the capability to dump resources when replaying a capture file. These resources can be:
Render targets
All images used as render target attachments (both as color and depth attachments) either in render passes or with dynamic rendering.
Vertex and index buffers
Buffers bound as vertex and index buffers referenced by draw calls which have been marked for dumping.
Results of compute (vkCmdDispatch and its variants) and ray tracing (vkCmdTraceRays) shaders.
All result images and buffers used as descriptor bindings by dispatch and ray tracing shaders.
Descriptor bindings used as inputs in all shader stages.
All images and buffers used as descriptor bindings by draw calls, dispatch and ray tracing shaders.
The resources are dumped into files and can either be image files (bmp or png) or binary files.
Dumping can take place only while replaying a capture file either on
desktop with the gfxrecon-replay tool or when replaying a
capture file on Android with the replay application.
In order to enable the dump resources feature the DrawCalls and/or Dispatch, and/or TraceRays for which the related resources is desired to be dumped, need to be specified in some way. This is can be done by using the block index with which these commands are recorded inside the capture file (for more details see [Block index])(#block-index-and-object-id).
It is possible to dump resources from multiple draw calls and/or Dispatch/TraceRays in a single run by specifying multiple indices.
Block index and object ID
Each command (either a Vulkan command issued by the application or a meta command generated by GFXReconstruct internally) stored in the capture file can be identified by a unique, monotonically increasing number that each command is assigned. This is called the command's block index. In order to specify commands for dumping their resources this index needs to be specified for each command.
The simplest way to find the command block index for each Vulkan
command is to use the gfxrecon-convert tool on a capture
file which will convert the capture into a human readable json file. Use
the --format jsonl option to produce a json file with one
command per line. The resulting jsonl file looks something like
this:
{"index":301,"function":{"name":"vkBeginCommandBuffer","thread":2,"return":"VK_SUCCESS","args":{"commandBuffer":80, ... }}},
{"index":302,"function":{"name":"vkCmdBeginRenderPass","thread":2,"cmd_index":1,"args":{"commandBuffer":80, ... }}},
{"index":303,"function":{"name":"vkCmdBindPipeline","thread":2,"cmd_index":2,"args":{"commandBuffer":80, ... }}},
{"index":304,"function":{"name":"vkCmdBindDescriptorSets","thread":2,"cmd_index":3,"args":{"commandBuffer":80, ... }}},
{"index":305,"function":{"name":"vkCmdSetViewport","thread":2,"cmd_index":4,"args":{"commandBuffer":80, ... }}},
{"index":306,"function":{"name":"vkCmdSetScissor","thread":2,"cmd_index":5,"args":{"commandBuffer":80, ... }}},
{"index":307,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":80, ... }}},
The block index is the index key used in each entry.
In a similar manner each vulkan object is assigned an object ID
inside the capture file. When an object is referenced by a command in
capture file, the object is referenced by its ID. In the above example
the VkCommandBuffer with object ID 80 is referenced and is
common between all commands.
Rules for providing command indices
Apart from the DrawCall/Dispatch/TraceRays indices, indices for several other Vulkan commands need to specified. In summary, the commands that can be specified are the following:
DrawCalls This includes the indices of
vkCmdDrawand all supported variants:vkCmdDrawvkCmdDrawIndexedvkCmdDrawIndirectvkCmdDrawIndexedIndirectvkCmdDrawIndirectCountvkCmdDrawIndexedIndirectCountvkCmdDrawIndirectCountKHRvkCmdDrawIndexedIndirectCountKHR
Dispatch This includes the indices of
vkCmdDispatchand all supported variants:vkCmdDispatchvkCmdDispatchIndirect
Trace rays This includes the indices of
vkCmdTraceRaysKHRand all supported variants:vkCmdTraceRaysKHRVkTraceRaysIndirectCommandKHR
Depending on the type of the commands that dumping is requested for, additional command indices need to be specified. These commands are:
BeginCommandBuffer This is the index of the
vkBeginCommandBufferinside which the indices of the dump-able commands are provided.Render pass Render pass indices are required only for draw calls. All render pass indices which surround the provided draw calls must be specified - the indices for
vkCmdBeginRenderPass,vkCmdNextSubpass(if any), andvkCmdEndRenderPass. In case of dynamic rendering the indices ofvkCmdBeginRenderingandvkCmdEndRenderingmust be provided instead.QueueSubmit The index of the
vkQueueSubmit(orvkQueueSubmit2) in which the command buffer that includes the desired commands are submitted needs to be provided.ExecuteCommands Dumping resources from commands that are recorded in secondary command buffers requires different handling. The secondary's
BeginCommandBufferindex must be specified like with the primary command buffers. The index of thevkCmdExecuteCommandsfrom which the specific commands should be dumped needs to be specified in theExecuteCommandsjson array along withBeginCommandBufferof the secondary that is desired to be dumped.
Simple examples
Assuming the following imaginary excerpt from a capture file that contains the following commands:
{"index":301,"function":{"name":"vkBeginCommandBuffer", ... } },
{"index":302,"function":{"name":"vkCmdBeginRenderPass", ... } },
{"index":303,"function":{"name":"vkCmdBindPipeline", ... } },
{"index":304,"function":{"name":"vkCmdBindDescriptorSets", ... } },
{"index":305,"function":{"name":"vkCmdSetViewport", ... } },
{"index":306,"function":{"name":"vkCmdSetScissor", ... } },
{"index":307,"function":{"name":"vkCmdDraw", ... } },
{"index":308,"function":{"name":"vkCmdDraw", ... } },
{"index":309,"function":{"name":"vkCmdDraw", ... } },
{"index":310,"function":{"name":"vkCmdDraw", ... } },
{"index":311,"function":{"name":"vkCmdDraw", ... } },
{"index":312,"function":{"name":"vkCmdDraw", ... } },
{"index":313,"function":{"name":"vkCmdEndRenderPass", ... } },
{"index":314,"function":{"name":"vkEndCommandBuffer", ... } },
{"index":315,"function":{"name":"vkQueueSubmit", ... } },
It is possible to dump the depth and color attachments of all
vkCmdDraw commands by providing the following json input
file:
{
"BeginCommandBuffer": [ 301 ],
"Draw": [ [ 307, 308, 309, 310, 311, 312 ] ],
"RenderPass": [ [ [ 302, 313 ] ] ],
"QueueSubmit": [ 315 ]
}
An example involving secondary command buffers
{"index":754,"function":{"name":"vkBeginCommandBuffer","args":{"commandBuffer":230 ... }}}, {"index":761,"function":{"name":"vkCmdDrawIndexed","args":{"commandBuffer":230, ...}}}
{"index":736,"function":{"name":"vkBeginCommandBuffer","args":{"commandBuffer":226, ...}}}, {"index":3948,"function":{"name":"vkCmdBeginRenderPass","args":{"commandBuffer":226, ...}}}, {"index":3949,"function":{"name":"vkCmdExecuteCommands","args":{"commandBuffer":226,"commandBufferCount":357,"pCommandBuffers":[227,230,232,233,...]}}} {"index":3950,"function":{"name":"vkCmdEndRenderPass","args":{"commandBuffer":226}}} {"index":3952,"function":{"name":"vkQueueSubmit","args":{"queue":6,"submitCount":1,"pSubmits":[{"commandBufferCount":1,"pCommandBuffers":[226]}...]...}}}
In order to dump the draw call from the secondary command buffer
230 the following json input file should be provided:
{
"BeginCommandBuffer": [ 754, 736 ],
"Draw": [ [ 761 ], [ ] ],
"RenderPass": [ [ [ ] ], [ [ 3948, 3950 ] ] ],
"ExecuteCommands": [ [ [ ] ], [ [ 3949, 754 ] ] ],
"QueueSubmit": [ 3952 ]
}
Index vector dimensionality
Indices are provided in GFXReconstruct as vectors of indices. Each index vector, depending on the type of the command it describes can vary and is important, otherwise errors will be generated while parsing the input or the expected commands will not be dumped. Each vector's dimensionality is the following:
"BeginCommandBuffer"and"QueueSubmit": 1D
Commands recorded in multiple command buffers can be dumped in a
single run. For each command buffer the index of the
vkBeginCommandBuffer must be provided. The index of the
vkQueueSubmit in which the command buffer is submitted must
be provided. Both these vectors must have the same number of
indices.
"Draw","Dispatch"and"TraceRays": 2D
These vectors are two dimensional. The first dimension corresponds to
BeginCommandBuffer each vector belongs to. I.e.:
{
"BeginCommandBuffer" : [ 10, 20 ],
"TraceRays": [ [ 220, 230, 240 ], [] ],
"Dispatch" : [ [], [ 250, 260, 270 ] ],
"QueueSubmit" : [ 350, 360 ]
}
In the example above the vkCmdTraceRays
[ 220, 230, 240 ] belong to
vkBeginCommandBuffer with block index 10 and
are submitted in vkQueueSubmit with block index
350.
vkCmdDispatch with indices
[ 250, 260, 270 ] belong to command buffer with
vkBeginCommandBuffer with block index 20 and
are submitted in vkQueueSubmit with block index
360.
"RenderPass": 3D
Inside a command buffer, Vulkan allows multiple render passes with multiple sub-passes. In order to support that for multiple command buffers a 3D array is required.
A more complex example
A hypothetical json output of the gfxrecon-convert tool
could look something like the following:
{"index":10,"function":{"name":"vkBeginCommandBuffer","thread":2,"return":"VK_SUCCESS","args":{"commandBuffer":59, ... }}},
{"index":11,"function":{"name":"vkBeginCommandBuffer","thread":2,"return":"VK_SUCCESS","args":{"commandBuffer":60, ... }}},
{"index":12,"function":{"name":"vkCmdBeginRenderPass","thread":2,"cmd_index":1,"args":{"commandBuffer":59, ... }}},
{"index":13,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":14,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":15,"function":{"name":"vkCmdNextSubpass","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":16,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":17,"function":{"name":"vkCmdDraw","thread":3,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":18,"function":{"name":"vkCmdEndRenderPass","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":19,"function":{"name":"vkCmdBeginRenderPass","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":20,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":21,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":22,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":23,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":24,"function":{"name":"vkCmdEndRenderPass","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":25,"function":{"name":"vkEndCommandBuffer","thread":2,"return":"VK_SUCCESS","args":{"commandBuffer":59}}}
{"index":26,"function":{"name":"vkCmdBeginRenderPass","thread":2,"cmd_index":1,"args":{"commandBuffer":60, ... }}},
{"index":27,"function":{"name":"vkCmdDrawIndexed","thread":2,"cmd_index":6,"args":{"commandBuffer":60, ... }}}
{"index":28,"function":{"name":"vkCmdDrawIndexed","thread":2,"cmd_index":6,"args":{"commandBuffer":60, ... }}}
{"index":29,"function":{"name":"vkCmdDrawIndexed","thread":2,"cmd_index":6,"args":{"commandBuffer":60, ... }}}
{"index":30,"function":{"name":"vkCmdEndRenderPass","thread":2,"cmd_index":6,"args":{"commandBuffer":60, ... }}}
{"index":31,"function":{"name":"vkEndCommandBuffer","thread":2,"return":"VK_SUCCESS","args":{"commandBuffer":60}}}
...
{"index":50,"function":{"name":"vkQueueSubmit", ... ,"commandBufferCount":1,"pCommandBuffers":[59], ... }},
{"index":51,"function":{"name":"vkQueueSubmit", ... ,"commandBufferCount":1,"pCommandBuffers":[60], ... }},
The indices submitted to gfxrecon-replay for dumping are
the following:
{
"BeginCommandBuffer" : [ 10, 11 ],
"Draw" : [ [ 13, 14, 16, 17, 20, 21, 22, 23 ],
[ 27, 28, 29 ] ],
"RenderPass" : [ [ [ 12, 15, 18 ], [ 19, 24 ] ], [ [ 26, 30 ] ] ],
"QueueSubmit" : [ 50, 51 ]
}
In this example two command buffers are submitted for dumping, one
with object ID 59 and one with object ID
60.
- The first command buffer with object ID
59andvkBeginCommandBufferwith index10contains:- The draw calls
[ 13, 14, 16, 17, 20, 21, 22, 23 ] - These draw calls are divided into two render passes:
[ 12, 15, 18 ]:vkCmdBeginRenderPass:12,vkCmdNextSubpass:15andvkCmdEndRenderPass:18[ 19, 24 ]:vkCmdBeginRenderPass:19andvkCmdEndRenderPass:24
- The draw calls
- The second command buffer with object ID
60andvkBeginCommandBufferwith index20contains:- The draw calls
[ 27, 28, 29 ] - One render pass with 1 sub pass:
[ 26, 30 ]:vkCmdBeginRenderPass:26andvkCmdEndRenderPass:30
- The draw calls
- Command buffer
59is submitted for execution withvkQueueSubmitwith index50and command buffer60is submitted invkQueueSubmitwith index51
Culling dumped descriptors
There is an option to ask for specific descriptors and image subresources to be dumped instead of dumping all bound descriptors. This can be done by providing for each command index the 1) descriptor set index, 2) binding index and, 3) array index of the descriptors to be dumped for the specific command. This is an example:
{
"BeginCommandBuffer": [
2525
],
"QueueSubmit": [
2548
],
"Draw": [
[
{
"Index": 2533,
"Descriptors": [
{
"Set": 0,
"Binding": 1,
"ArrayIndex": 0
},
{
"Set": 0,
"Binding": 0,
"ArrayIndex": 0
}
]
},
2537
]
],
"RenderPass": [
[
[
2526,
2538
]
]
]
}Image descriptors can be fine grained further by specifying the
desired subresources with a VkImageSubresourceRange like
this:
"Index": 2533,
"Descriptors": [
{
"Set": 0,
"Binding": 1,
"ArrayIndex": 0,
"SubresourceRange": {
"AspectMask": "VK_IMAGE_ASPECT_DEPTH_BIT | VK_IMAGE_ASPECT_STENCIL_BIT",
"BaseMipLevel": 2,
"LevelCount": 1,
"BaseArrayLayer": 2,
"LayerCount": 1
}
}
]VK_REMAINING_MIP_LEVELS and
VK_REMAINING_ARRAY_LAYERS can be used in
LevelCount and LayerCount respectively.
Vulkan dump resources options
gfxrecon-replay command line parameters
Dump resources feature can be controled in several ways. This is done
either by providing the appropriate options via the command line or
using the input json. The majority of the options are only available via
the input json. The command line options, which can be passed to either
to the gfxrecon-replay tool or to the Android application
through the gfxrecon.py script are the following:
--dump-resources <filename>.jsonSpecify the input json file. It should contain the command indices and the various options that control how and which the resources should be dumped. The format for the json file is documented in detail in the gfxreconstruct documentation.--dump-resources-dir <dir>Directory to write dump resources output files. Default is the current working directory.
Input json options
The rest of the dump resources options are available through the
input json file. They are expected to be inside a json object named
DumpResourcesOptions. The possible options are the
following:
Scale<float>: Scale images generated by dump resources by the given scale factor. The scale factor must be a floating point number greater than 0. Values greater than 10 are capped at 10. Default value is1.0.ImageFormat<image format>: Image file format to use for image resource dumping. Available formats are:bmpandpng. Default isbmp.DumpBeforeCommand<boolean>: In addition to dumping gpu resources after the CmdDraw, CmdDispatch and CmdTraceRays calls, also dump resources before those calls. Default isfalse.DumpDepth<boolean>: Enables dumping of the depth attachment when dumping draw calls. Default is disabled. Default isfalse.ColorAttachmentIndex<unsigned integer>: Specify which color attachment to dump when dumping draw calls. Index should be an unsigned zero based integer. Default is to dump all color attachments. By default all attachments are dumped.DumpVertexIndexBuffer<boolean>: Enables dumping of vertex and index buffers while dumping draw call resources. Default isfalse.DumpAllDescriptors<boolean>: Enables dumping all descriptors used in the shaders referenced by the commands that have been marked for dumping. Default isfalse.DumpAllImageSubresources<boolean>: Enables dumping of all image sub resources (mip map levels and array layers). Default isfalse.DumpRawImages<boolean>: When enabled all image resources will be dumped verbatim as raw bin files. Default isfalse.DumpSeparateAlpha<boolean>: When enabled alpha channel of dumped images will be dumped in a separate file. Default isfalse.DumpUnusedVertexBindings<boolean>: Bound vertex buffer bindings are filtered based on the currently active graphics pipeline. Setting this option to true will dump all bound vertex buffers. Default isfalse.JsonOutputPerCommand<boolean>: Enables storing a json output file for each dumped command. Overrides default behavior which is generating one output json file that contains the information for all dumped commands. Default isfalse.BinaryFileCompressionType<compression method>: Compress files that are dumped as binary. Available compression methods are: [none,lz4(block format),zlib,zlib,zstd]. Default isnone(no compression).DumpBuildAccelerationStructuresInputBuffers<boolean>: Dump all input buffers used in vkCmdBuildAccelerationStructures. This includes vertex, index, transformation matrix, AABB and instance buffers. Default isfalse.
A json example specifying the above options:
{
"DumpResourcesOptions": {
"Scale": 0.2,
"OutputDir": "",
"ColorAttachmentIndex": -1,
"OutputDir": "",
"ImageFormat": "png",
"DumpBeforeCommand": false,
"DumpDepth": false,
"DumpVertexIndexBuffer": false,
"DumpAllDescriptors": true,
"DumpAllImageSubresources": true,
"DumpRawImages": false,
"DumpSeparateAlpha": false,
"DumpUnusedVertexBindings": false,
"BinaryFileCompressionType": "None",
"DumpBuildAccelerationStructuresInputBuffers": true
},
"BeginCommandBuffer": [ 448 ],
"TraceRays": [[ 451 ]],
"QueueSubmit": [ 466 ]
}Output
Json file output
The dump resources feature generates a number of output files. A json file with an entry for each set of resources that are dumped is generated, as well as image and binary files. The json output file name is derived from the input file name, with the ".gfxr" extension replaced with "_rd.json", i.e. an input file name of "vulkanCapture.gfxr" will result in a json output file name of "vulkanCapture_rd.json".
The output json consists of 4 main entries:
- Header
The header contains the path to the capture file, GFXR and vulkan version and the dump resources parameters used when replaying.
- Draw call commands
The draw calls are listed in an array. Each draw call entry contains information regarding:
- The Vulkan api command parameters
- The color and/or depth attachments
- Vertex and/or index buffers (if requested)
- The immutable descriptors which can be images and/or buffers
- Dispatch commands and Trace Rays commands
- The Vulkan api command parameters
- The generated images and/or buffers
- The immutable descriptors which can be images and/or buffers
Here is an example of a json output file:
[
{
"header": {
"source-path": "vulkanCapture.gfxr",
"gfxrecon-version": "1.0.3"
"vulkan-version": "1.3.275"
"dumpResourcesOptions": {
"scale": 1.0,
"dumpResourcesOutputDir": "",
"dumpResourcesColorAttachmentIndex": -1,
"dumpResourcesBefore": false,
"dumpResourcesDumpDepth": true,
"dumpResourcesDumpVertexIndexBuffer": true,
"dumpResourcesDumpImmutableResources": true,
"dumpResourcesDumpAllImageSubresources": false
}
}
},
{
"drawCallCommands": [
{
"drawIndex": 407,
"beginCommandBufferIndex": 399,
"queueSubmitIndex": 461,
"parameters": {
"drawCallType": "vkCmdDraw",
"vertexCount": 24576,
"instanceCount": 1,
"firstVertex": 0,
"firstInstance": 0
},
"colorAttachments": [
{
"imageId": 15,
"format": "VK_FORMAT_B8G8R8A8_UNORM",
"type": "VK_IMAGE_TYPE_2D",
"aspect": "COLOR",
"dimensions": [
1280,
720,
1
],
"mipLevel": 0,
"arrayLayer": 0,
"file": "Draw_407_qs_461_bcb_399_att_0_aspect_color.bmp"
}
],
"depthAttachments": [
{
"imageId": 32,
"format": "VK_FORMAT_D32_SFLOAT_S8_UINT",
"type": "VK_IMAGE_TYPE_2D",
"aspect": "DEPTH",
"dimensions": [
1280,
720,
1
],
"mipLevel": 0,
"arrayLayer": 0,
"file": "Draw_407_qs_461_bcb_399_depth_att_aspect_depth.bmp"
},
...
],
"vertexBuffers": [
{
"bufferId": 75,
"vertexBufferBinding": 0,
"file": "VertexBuffers_qs_461_bcb_399_dc_407_binding_0.bin"
}
],
"descriptors": {
"vertex": [
{
"type": "VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER",
"set": 0,
"binding": 2,
"arrayIndex": 0,
"descriptor": {
"bufferId": 79,
"file": "Buffer_79_qs_461_bcb_399_rp_0.bin"
}
}
],
"fragment": [
{
"type": "VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER",
"set": 0,
"binding": 1,
"arrayIndex": 0,
"descriptor": [
{
"imageId": 68,
"format": "VK_FORMAT_R8G8B8A8_UNORM",
"type": "VK_IMAGE_TYPE_2D",
"aspect": "COLOR",
"dimensions": [
256,
1,
1
],
"mipLevel": 0,
"arrayLayer": 0,
"file": "Image_68_qs_461_bcb_399_rp_0_aspect_color.bmp"
}
]
},
...
]
}
},
]
},
{
"dispatchCommands": [
{
"dispatchIndex": 294,
"beginCommandBufferIndex": 290,
"queueSubmitIndex": 723,
"parameters": {
"dispatchType": "vkCmdDispatch",
"groupCountX": 96,
"groupCountY": 1,
"groupCountZ": 1
},
"outputs": {
"buffers": [
{
"type": "VK_DESCRIPTOR_TYPE_STORAGE_BUFFER",
"set": 0,
"binding": 0,
"arrayIndex": 0,
"bufferId": 75,
"file": "Dispatch_294_qs_723_bcb_290_stage_compute_set_0_binding_0_index_0_buffer.bin"
}
]
},
"descriptors": [
{
"type": "VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER",
"set": 0,
"binding": 1,
"arrayIndex": 0,
"descriptor": {
"bufferId": 90,
"file": "Buffer_90_qs_723_bcb_290.bin"
}
},
{
"type": "VK_DESCRIPTOR_TYPE_STORAGE_BUFFER",
"set": 0,
"binding": 0,
"arrayIndex": 0,
"descriptor": {
"bufferId": 75,
"file": "Buffer_75_qs_723_bcb_290.bin"
}
}
]
},
...
]
}]
Image file output
The image files that created are either images (in one of the
supported image formats), or raw binary files (.bin). Raw
binary files are created when the dumped resource is an image with a
format which cannot be converted into a plain 32bit RGBA layout.
Buffer file output
All buffers are dumped as raw binary files (.bin).