Crash Diagnostic Layer

The Crash Diagnostic Layer (CDL) is a Vulkan layer to help track down and identify the cause of GPU hangs and crashes. It works by instrumenting command buffers with completion checkpoints. When an error is detected a dump file containing incomplete command buffers is written. Often the last complete or incomplete commands are responsible for the crash. The name of the layer is VK_LAYER_LUNARG_crash_diagnostic.

Building

See the associated BUILD.md file for details on how to build the Crash Diagnostic Layer for various platforms.

Runtime requirements

CDL uses the following extensions. If an extension is not present, some functionality might be disabled.

Running

CDL can be used as an explicit or implicit layer. The loader's documentation describes the difference between implicit and explicit layers, but the relevant bit here is that implicit layers are meant to be available to all applications on the system, even if the application doesn't explicitly enable the layer. On the other hand, explicit layers are easier to use when doing application development.

Explicit Layer

To use CDL as an explicit layer, enable it with vkconfig or do the following:

  1. The directory containing the file VkLayer_crash_diagnostic.json is included in the layer search path, by including it in either the VK_LAYER_PATH or VK_ADD_LAYER_PATH environment variable or by using vkconfig.
  2. Include the layer name in the VK_INSTANCE_LAYERS environment variable or the ppEnabledLayerNames field of VkInstanceCreateInfo.

Implicit Layer

When using CDL as an implicit layer, it is disabled by default. To enable the layer's functionality, set the CDL_ENABLE environment variable to 1.

Registering the Layer

In order to be discovered by the Vulkan loader at runtime, implicit layers must be registered. The registration process is platform-specific and is discussed in detail in the Vulkan-Loader documentation. In all cases, it is the layer manifest (the .json file) that is registered; the manifest contains a relative path to the layer library, which can be in a separate directory.

Basic Usage

Once the layer is enabled, if vkQueueSubmit() or other Vulkan functions returns a fatal error code (usually VK_ERROR_DEVICE_LOST), a dump file of the command buffers (and other state) that failed to execute are written to disk.

Logging

CDL outputs messages at runtime to indicate it is enabled, to trace certain commands, and to report when a gpu crash has occurred. The message_severity, log_file, trace_on and trace_all_semaphores configuration settings control which messages are output and where they are sent.

VK_EXT_debug_utils and VK_EXT_debug_report extensions can also be used to receive log messages directly in the application.

Dump files

A unique log file directory is created every time an application is run with CDL enabled. The log file directories are named with the timestamp of the format YYYY-MM-DD-HHMMSS. This prevents subsequent runs from overwriting old data, in case the application is non-deterministic and producing different crashes on each run. The location of these files is platform-specific:

The output directory for dump files can be changed using the output_path configuration setting, described below.

The name of the dump file is cdl_dump.yaml, since the file is in the YAML format. Other files, such as dumped shaders, may be present in the dump directory.

Configuration

This layer implements the VK_EXT_layer_settings extension, so it can be configured with vkconfig, programmatically, or with environment variables. See the manifest for this layer and the layer manifest schema for full details. The discussion below uses the key field to identify each setting, the env field defines the corresponding environment variable, and the label field defines the text you will see when using vkconfig.

Layer Properties

Layer Settings Overview

Label Variables Key Type Default Value Platforms
Watchdog timeout (ms) watchdog_timeout_ms INT 30000 WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
Dump files dump GROUP `` WINDOWS_X86, WINDOWS_ARM, LINUX, ANDROID
    Output Path output_path STRING `` WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
    Dump queue submissions dump_queue_submits ENUM running WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
    Dump command buffers dump_command_buffers ENUM running WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
    Dump commands dump_commands ENUM running WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
    Dump shaders dump_shaders ENUM off WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
Logging logging GROUP `` WINDOWS_X86, WINDOWS_ARM, LINUX, ANDROID
    Message Severity message_severity FLAGS error WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
    Log file name log_file STRING stderr WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS
    Enable Tracing trace_on BOOL false WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
    Enable semaphore log tracing. trace_all_semaphores BOOL false WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
State Tracking state GROUP `` WINDOWS_X86, WINDOWS_ARM, LINUX, ANDROID
    Synchronize commands sync_after_commands BOOL false WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
    Instrument all commands instrument_all_commands BOOL false WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
    Track semaphores track_semaphores BOOL true WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID

Layer Settings Details

Watchdog timeout (ms)

If set to a non-zero number, a watchdog thread will be created. This will trigger if the application fails to submit new commands within a set time (in milliseconds) and a log will be created as if the a lost device error was encountered.

Methods:
VK_EXT_layer_settings:
int32_t data[] = { 30000 };
VkLayerSettingEXT setting {
 "VK_LAYER_LUNARG_crash_diagnostic", "watchdog_timeout_ms", VK_LAYER_SETTING_TYPE_INT32_EXT,
 static_cast<uint32_t>(std::size(data)), data };
vk_layer_settings.txt:
lunarg_crash_diagnostic.watchdog_timeout_ms = 30000
Environment variables:
export VK_LUNARG_CRASH_DIAGNOSTIC_WATCHDOG_TIMEOUT_MS=30000
export VK_CRASH_DIAGNOSTIC_WATCHDOG_TIMEOUT_MS=30000
export VK_WATCHDOG_TIMEOUT_MS=30000
export CDL_WATCHDOG_TIMEOUT_MS=30000
Android system properties:
adb setprop debug.vulkan.lunarg_crash_diagnostic.watchdog_timeout_ms 30000
adb setprop debug.vulkan.crash_diagnostic.watchdog_timeout_ms 30000
adb setprop debug.vulkan.watchdog_timeout_ms 30000

Dump files

Control of dump files.

Sub-settings: Output Path, Dump queue submissions, Dump command buffers, Dump commands, and Dump shaders.

Output Path

The directory where dump files and shader binaries are written.

Methods:
VK_EXT_layer_settings:
const char* data[] = { "" };
VkLayerSettingEXT setting {
 "VK_LAYER_LUNARG_crash_diagnostic", "output_path", VK_LAYER_SETTING_TYPE_STRING_EXT,
 static_cast<uint32_t>(std::size(data)), data };
vk_layer_settings.txt:
lunarg_crash_diagnostic.output_path = 
Environment variables:
export VK_LUNARG_CRASH_DIAGNOSTIC_OUTPUT_PATH=
export VK_CRASH_DIAGNOSTIC_OUTPUT_PATH=
export VK_OUTPUT_PATH=
export CDL_OUTPUT_PATH=
Android system properties:
adb setprop debug.vulkan.lunarg_crash_diagnostic.output_path 
adb setprop debug.vulkan.crash_diagnostic.output_path
adb setprop debug.vulkan.output_path

Dump queue submissions

Control which queue submissions are included in the dump.

Enum Value Label Description Platforms
running Running Submissions that were executing at the time of the dump. WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
pending Pending Submissions executing or pending at the time of the dump. WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
Methods:
VK_EXT_layer_settings:
const char* data[] = { "running" };
VkLayerSettingEXT setting {
 "VK_LAYER_LUNARG_crash_diagnostic", "dump_queue_submits", VK_LAYER_SETTING_TYPE_STRING_EXT,
 static_cast<uint32_t>(std::size(data)), data };
vk_layer_settings.txt:
lunarg_crash_diagnostic.dump_queue_submits = running
Environment variables:
export VK_LUNARG_CRASH_DIAGNOSTIC_DUMP_QUEUE_SUBMITS=running
export VK_CRASH_DIAGNOSTIC_DUMP_QUEUE_SUBMITS=running
export VK_DUMP_QUEUE_SUBMITS=running
export CDL_DUMP_QUEUE_SUBMITS=running
Android system properties:
adb setprop debug.vulkan.lunarg_crash_diagnostic.dump_queue_submits running
adb setprop debug.vulkan.crash_diagnostic.dump_queue_submits running
adb setprop debug.vulkan.dump_queue_submits running

Dump command buffers

Control which command buffers are included in the dump.

Enum Value Label Description Platforms
running Running Command Buffers that were executing at the time of the dump. WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
pending Pending Command Buffers executing or pending at the time of the dump. WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
all All All known Command Buffers. WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
Methods:
VK_EXT_layer_settings:
const char* data[] = { "running" };
VkLayerSettingEXT setting {
 "VK_LAYER_LUNARG_crash_diagnostic", "dump_command_buffers", VK_LAYER_SETTING_TYPE_STRING_EXT,
 static_cast<uint32_t>(std::size(data)), data };
vk_layer_settings.txt:
lunarg_crash_diagnostic.dump_command_buffers = running
Environment variables:
export VK_LUNARG_CRASH_DIAGNOSTIC_DUMP_COMMAND_BUFFERS=running
export VK_CRASH_DIAGNOSTIC_DUMP_COMMAND_BUFFERS=running
export VK_DUMP_COMMAND_BUFFERS=running
export CDL_DUMP_COMMAND_BUFFERS=running
Android system properties:
adb setprop debug.vulkan.lunarg_crash_diagnostic.dump_command_buffers running
adb setprop debug.vulkan.crash_diagnostic.dump_command_buffers running
adb setprop debug.vulkan.dump_command_buffers running

Dump commands

Control which commands are included in the dump.

Enum Value Label Description Platforms
running Running Command Buffers that were executing at the time of the dump. WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
pending Pending Commands executing or pending at the time of the dump. WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
all All All known Commands. WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
Methods:
VK_EXT_layer_settings:
const char* data[] = { "running" };
VkLayerSettingEXT setting {
 "VK_LAYER_LUNARG_crash_diagnostic", "dump_commands", VK_LAYER_SETTING_TYPE_STRING_EXT,
 static_cast<uint32_t>(std::size(data)), data };
vk_layer_settings.txt:
lunarg_crash_diagnostic.dump_commands = running
Environment variables:
export VK_LUNARG_CRASH_DIAGNOSTIC_DUMP_COMMANDS=running
export VK_CRASH_DIAGNOSTIC_DUMP_COMMANDS=running
export VK_DUMP_COMMANDS=running
export CDL_DUMP_COMMANDS=running
Android system properties:
adb setprop debug.vulkan.lunarg_crash_diagnostic.dump_commands running
adb setprop debug.vulkan.crash_diagnostic.dump_commands running
adb setprop debug.vulkan.dump_commands running

Dump shaders

Control of shader dumping.

Enum Value Label Description Platforms
off Off Never dump shaders. WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
on_crash On Crash Dump currently bound shaders after a crash is detected. WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
on_bind On Bind Dump only bound shaders. WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
all All Dump all shaders. WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
Methods:
VK_EXT_layer_settings:
const char* data[] = { "off" };
VkLayerSettingEXT setting {
 "VK_LAYER_LUNARG_crash_diagnostic", "dump_shaders", VK_LAYER_SETTING_TYPE_STRING_EXT,
 static_cast<uint32_t>(std::size(data)), data };
vk_layer_settings.txt:
lunarg_crash_diagnostic.dump_shaders = off
Environment variables:
export VK_LUNARG_CRASH_DIAGNOSTIC_DUMP_SHADERS=off
export VK_CRASH_DIAGNOSTIC_DUMP_SHADERS=off
export VK_DUMP_SHADERS=off
export CDL_DUMP_SHADERS=off
Android system properties:
adb setprop debug.vulkan.lunarg_crash_diagnostic.dump_shaders off
adb setprop debug.vulkan.crash_diagnostic.dump_shaders off
adb setprop debug.vulkan.dump_shaders off

Logging

Control of logging.

Sub-settings: Message Severity, Log file name, Enable Tracing, and Enable semaphore log tracing..

Message Severity

Comma-delineated list of options specifying the types of log messages to be reported.

Flags Label Description Platforms
error Error Report errors such as device lost or setup problems in the layer. WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
warn Warning Report non-fatal problems that may interfere with operation of the layer WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
info Info Report informational messages. WINDOWS_X86, WINDOWS_ARM, LINUX, MACOS, ANDROID
Methods:
VK_EXT_layer_settings:
const char* data[] = { "error" };
VkLayerSettingEXT setting {
 "VK_LAYER_LUNARG_crash_diagnostic", "message_severity", VK_LAYER_SETTING_TYPE_STRING_EXT,
 static_cast<uint32_t>(std::size(data)), data };
vk_layer_settings.txt:
lunarg_crash_diagnostic.message_severity = error
Environment variables:
export VK_LUNARG_CRASH_DIAGNOSTIC_MESSAGE_SEVERITY=error
export VK_CRASH_DIAGNOSTIC_MESSAGE_SEVERITY=error
export VK_MESSAGE_SEVERITY=error
Android system properties:
adb setprop debug.vulkan.lunarg_crash_diagnostic.message_severity error
adb setprop debug.vulkan.crash_diagnostic.message_severity error
adb setprop debug.vulkan.message_severity error

Log file name

none = no logging, stderr or stdout = to the console, otherwise an absolute or relative path.

Methods:
VK_EXT_layer_settings:
const char* data[] = { "stderr" };
VkLayerSettingEXT setting {
 "VK_LAYER_LUNARG_crash_diagnostic", "log_file", VK_LAYER_SETTING_TYPE_STRING_EXT,
 static_cast<uint32_t>(std::size(data)), data };
vk_layer_settings.txt:
lunarg_crash_diagnostic.log_file = stderr
Environment variables:
export VK_LUNARG_CRASH_DIAGNOSTIC_LOG_FILE=stderr
export VK_CRASH_DIAGNOSTIC_LOG_FILE=stderr
export VK_LOG_FILE=stderr
export CDL_LOG_FILE=stderr

Enable Tracing

All Vulkan API calls intercepted by the layer will be logged to the console.

Methods:
VK_EXT_layer_settings:
VkBool32 data[] = { VK_FALSE };
VkLayerSettingEXT setting {
 "VK_LAYER_LUNARG_crash_diagnostic", "trace_on", VK_LAYER_SETTING_TYPE_BOOL32_EXT,
 static_cast<uint32_t>(std::size(data)), data };
vk_layer_settings.txt:
lunarg_crash_diagnostic.trace_on = false
Environment variables:
export VK_LUNARG_CRASH_DIAGNOSTIC_TRACE_ON=false
export VK_CRASH_DIAGNOSTIC_TRACE_ON=false
export VK_TRACE_ON=false
export CDL_TRACE_ON=false
Android system properties:
adb setprop debug.vulkan.lunarg_crash_diagnostic.trace_on false
adb setprop debug.vulkan.crash_diagnostic.trace_on false
adb setprop debug.vulkan.trace_on false

Enable semaphore log tracing.

Semaphore events will be logged to console.

Methods:
VK_EXT_layer_settings:
VkBool32 data[] = { VK_FALSE };
VkLayerSettingEXT setting {
 "VK_LAYER_LUNARG_crash_diagnostic", "trace_all_semaphores", VK_LAYER_SETTING_TYPE_BOOL32_EXT,
 static_cast<uint32_t>(std::size(data)), data };
vk_layer_settings.txt:
lunarg_crash_diagnostic.trace_all_semaphores = false
Environment variables:
export VK_LUNARG_CRASH_DIAGNOSTIC_TRACE_ALL_SEMAPHORES=false
export VK_CRASH_DIAGNOSTIC_TRACE_ALL_SEMAPHORES=false
export VK_TRACE_ALL_SEMAPHORES=false
export CDL_TRACE_ALL_SEMAPHORES=false
Android system properties:
adb setprop debug.vulkan.lunarg_crash_diagnostic.trace_all_semaphores false
adb setprop debug.vulkan.crash_diagnostic.trace_all_semaphores false
adb setprop debug.vulkan.trace_all_semaphores false

State Tracking

Control of state tracking.

Sub-settings: Synchronize commands, Instrument all commands, and Track semaphores.

Synchronize commands

Add pipeline barriers after instrumented commands.

Methods:
VK_EXT_layer_settings:
VkBool32 data[] = { VK_FALSE };
VkLayerSettingEXT setting {
 "VK_LAYER_LUNARG_crash_diagnostic", "sync_after_commands", VK_LAYER_SETTING_TYPE_BOOL32_EXT,
 static_cast<uint32_t>(std::size(data)), data };
vk_layer_settings.txt:
lunarg_crash_diagnostic.sync_after_commands = false
Environment variables:
export VK_LUNARG_CRASH_DIAGNOSTIC_SYNC_AFTER_COMMANDS=false
export VK_CRASH_DIAGNOSTIC_SYNC_AFTER_COMMANDS=false
export VK_SYNC_AFTER_COMMANDS=false
export CDL_SYNC_AFTER_COMMANDS=false
Android system properties:
adb setprop debug.vulkan.lunarg_crash_diagnostic.sync_after_commands false
adb setprop debug.vulkan.crash_diagnostic.sync_after_commands false
adb setprop debug.vulkan.sync_after_commands false

Instrument all commands

All commands will be instrumented.

Methods:
VK_EXT_layer_settings:
VkBool32 data[] = { VK_FALSE };
VkLayerSettingEXT setting {
 "VK_LAYER_LUNARG_crash_diagnostic", "instrument_all_commands", VK_LAYER_SETTING_TYPE_BOOL32_EXT,
 static_cast<uint32_t>(std::size(data)), data };
vk_layer_settings.txt:
lunarg_crash_diagnostic.instrument_all_commands = false
Environment variables:
export VK_LUNARG_CRASH_DIAGNOSTIC_INSTRUMENT_ALL_COMMANDS=false
export VK_CRASH_DIAGNOSTIC_INSTRUMENT_ALL_COMMANDS=false
export VK_INSTRUMENT_ALL_COMMANDS=false
export CDL_INSTRUMENT_ALL_COMMANDS=false
Android system properties:
adb setprop debug.vulkan.lunarg_crash_diagnostic.instrument_all_commands false
adb setprop debug.vulkan.crash_diagnostic.instrument_all_commands false
adb setprop debug.vulkan.instrument_all_commands false

Track semaphores

Enable semaphore tracking.

Methods:
VK_EXT_layer_settings:
VkBool32 data[] = { VK_TRUE };
VkLayerSettingEXT setting {
 "VK_LAYER_LUNARG_crash_diagnostic", "track_semaphores", VK_LAYER_SETTING_TYPE_BOOL32_EXT,
 static_cast<uint32_t>(std::size(data)), data };
vk_layer_settings.txt:
lunarg_crash_diagnostic.track_semaphores = true
Environment variables:
export VK_LUNARG_CRASH_DIAGNOSTIC_TRACK_SEMAPHORES=true
export VK_CRASH_DIAGNOSTIC_TRACK_SEMAPHORES=true
export VK_TRACK_SEMAPHORES=true
export CDL_TRACK_SEMAPHORES=true
Android system properties:
adb setprop debug.vulkan.lunarg_crash_diagnostic.track_semaphores true
adb setprop debug.vulkan.crash_diagnostic.track_semaphores true
adb setprop debug.vulkan.track_semaphores true