time travel debugging gdb

DEV Community

Posted on Aug 8, 2023

Time-Travel Debugging Production Code

In this post, I’ll give an overview of time travel debugging (what it is, its history, how it’s implemented) and show how it relates to debugging your production code.

Normally, when we use debuggers, we set a breakpoint on a line of code, we run our code, execution pauses on our breakpoint, we look at values of variables and maybe the call stack, and then we manually step forward through our code's execution. In time-travel debugging , also known as reverse debugging , we can step backward as well as forward. This is powerful because debugging is an exercise in figuring out what happened: traditional debuggers are good at telling you what your program is doing right now, whereas time-travel debuggers let you see what happened. You can wind back to any line of code that executed and see the full program state at any point in your program’s history.

History and current state

It all started with Smalltalk-76, developed in 1976 at Xerox PARC . ( Everything started at PARC 😄.) It had the ability to retrospectively inspect checkpointed places in execution. Around 1980, MIT added a "retrograde motion" command to its DDT debugger , which gave a limited ability to move backward through execution. In a 1995 paper, MIT researchers released ZStep 95, the first true reverse debugger, which recorded all operations as they were performed and supported stepping backward, reverting the system to the previous state. However, it was a research tool and not widely adopted outside academia.

ODB, the Omniscient Debugger , was a Java reverse debugger that was introduced in 2003, marking the first instance of time-travel debugging in a widely used programming language. GDB (perhaps the most well-known command-line debugger, used mostly with C/C++) added it in 2009.

Now, time-travel debugging is available for many languages, platforms, and IDEs, including:

• Replay for JavaScript in Chrome, Firefox, and Node, and Wallaby for tests in Node
• WinDbg for Windows applications
• rr for C, C++, Rust, Go, and others on Linux
• Undo for C, C++, Java, Kotlin, Rust, and Go on Linux
• Various extensions (often rr- or Undo-based) for Visual Studio, VS Code, JetBrains IDEs, Emacs, etc.

Implementation techniques

There are three main approaches to implementing time-travel debugging:

• Record & Replay : Record all non-deterministic inputs to a program during its execution. Then, during the debug phase, the program can be deterministically replayed using the recorded inputs in order to reconstruct any prior state.
• Snapshotting : Periodically take snapshots of a program's entire state. During debugging, the program can be rolled back to these saved states. This method can be memory-intensive because it involves storing the entire state of the program at multiple points in time.
• Instrumentation : Add extra code to the program that logs changes in its state. This extra code allows the debugger to step the program backwards by reverting changes. However, this approach can significantly slow down the program's execution.

rr uses the first (the rr name stands for Record and Replay), as does Replay . WinDbg uses the first two, and Undo uses all three (see how it differs from rr ).

Time-traveling in production

Traditionally, running a debugger in prod doesn't make much sense. Sure, we could SSH into a prod machine and start the process handling requests with a debugger and a breakpoint, but once we hit the breakpoint, we're delaying responses to all current requests and unable to respond to new requests. Also, debugging non-trivial issues is an iterative process: we get a clue, we keep looking and find more clues; discovery of each clue is typically rerunning the program and reproducing the failure. So, instead of debugging in production, what we do is replicate on our dev machine whatever issue we're investigating and use a debugger locally (or, more often, add log statements 😄), and re-run as many times as required to figure it out. Replicating takes time (and in some cases a lot of time, and in some cases infinite time), so it would be really useful if we didn't have to.

While running traditional debuggers doesn't make sense, time-travel debuggers can record a process execution on one machine and replay it on another machine. So we can record (or snapshot or instrument) production and replay it on our dev machine for debugging (depending on the tool, our machine may need to have the same CPU instruction set as prod). However, the recording step generally doesn't make sense to use in prod given the high amount of overhead—if we set up recording and then have to use ten times as many servers to handle the same load, whoever pays our AWS bill will not be happy 😁.

But there are a couple scenarios in which it does make sense:

• Undo only slows down execution 2–5x , so while we don't want to leave it on just in case, we can turn it on temporarily on a subset of prod processes for hard-to-repro bugs until we have captured the bug happening, and then we turn it off.
• When we're already recording the execution of a program in the normal course of operation.

The rest of this post is about #2, which is a way of running programs called durable execution .

Durable execution

What's that.

First, a brief backstory. After Amazon (one of the first large adopters of microservices) decided that using message queues to communicate between services was not the way to go (hear the story first-hand here ), they started using orchestration. And once they realized defining orchestration logic in YAML/JSON wasn't a good developer experience, they created AWS Simple Workfow Service to define logic in code. This technique of backing code by an orchestration engine is called durable execution, and it spread to Azure Durable Functions , Cadence (used at Uber for > 1,000 services ), and Temporal (used by Stripe, Netflix, Datadog, Snap, Coinbase, and many more).

Durable execution runs code durably—recording each step in a database, so that when anything fails, it can be retried from the same step. The machine running the function can even lose power before it gets to line 10, and another process is guaranteed to pick up executing at line 10, with all variables and threads intact. 1 It does this with a form of record & replay: all input from the outside is recorded, so when the second process picks up the partially-executed function, it can replay the code (in a side-effect–free manner) with the recorded input in order to get the code into the right state by line 10.

Durable execution's flavor of record & replay doesn't use high-overhead methods like software JIT binary translation , snapshotting, or instrumentation. It also doesn't require special hardware. It does require one constraint: durable code must be deterministic (i.e., given the same input, it must take the same code path). So it can't do things that might have different results at different times, like use the network or disk. However, it can call other functions that are run normally ( "volatile functions" , as we like to call them 😄), and while each step of those functions isn't persisted, the functions are automatically retried on transient failures (like a service being down).

Only the steps that require interacting with the outside world (like calling a volatile function, or calling sleep('30 days') , which stores a timer in the database) are persisted. Their results are also persisted, so that when you replay the durable function that died on line 10, if it previously called the volatile function on line 5 that returned "foo", during replay, "foo" will immediately be returned (instead of the volatile function getting called again). While yes, it adds latency to be saving things to the database, Temporal supports extremely high throughput (tested up to a million recorded steps per second). And in addition to function recoverability and automatic retries, it comes with many more benefits , including extraordinary visibility into and debuggability of production.

Debugging prod

With durable execution, we can read through the steps that every single durable function took in production. We can also download the execution’s history, checkout the version of the code that's running in prod, and pass the file to a replayer (Temporal has runtimes for Go, Java, JavaScript, Python, .NET, and PHP) so we can see in a debugger exactly what the code did during that production function execution. Read this post or watch this video to see an example in VS Code. 2

Being able to debug any past production code is a huge step up from the other option (finding a bug, trying to repro locally, failing, turning on Undo recording in prod until it happens again, turning it off, then debugging locally). It's also a (sometimes necessary) step up distributed tracing.

I hope you found this post interesting! If you'd like to learn more about durable execution, I recommend reading:

• Building reliable distributed systems
• How durable execution works

and watching:

• Introduction to Temporal
• Why durable execution changes everything

Thanks to Greg Law, Jason Laster, Chad Retz, and Fitz for reviewing drafts of this post.

Technically, it doesn't have line-by-line granularity. It only records certain steps that the code takes—read on for more info ☺️.  ↩

The astute reader may note that our extension uses the default VS Code debugger, which doesn’t have a back button 😄. I transitioned from talking about TTD to methods of debugging production code via recording, so while Temporal doesn’t have TTD yet, it does record all the non-deterministic inputs to the program and is able to replay execution, so it’s definitely possible to implement. Upvote this issue or comment if you have thoughts on implementation!  ↩

Top comments (0)

Templates let you quickly answer FAQs or store snippets for re-use.

Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink .

Hide child comments as well

For further actions, you may consider blocking this person and/or reporting abuse

5 Dicas de Flutter

Suami Rocha - Apr 14

How to remove background of an image with CSS

Madhu Saini - Apr 3

4 Habits to Avoid Bugs related to Hoisting in your JavaScript Code

Shubh Sharma - Apr 15

The Illusion of Expertise: A Critique of Modern E-Learning's Superficial Certificates

Leandro Nuñez - Apr 14

We're a place where coders share, stay up-to-date and grow their careers.

Documentation

Find developer guides, API references, and tutorials to get started with time travel debugging, develop advanced troubleshooting skills, and learn how to integrate LiveRecorder and UDB into your application or workflow.

Undo documentation ¶

Your guide to UDB and LiveRecorder

This documentation explains how to use UDB and LiveRecorder to understand complex code and fix bugs faster.

The UDB time travel debugger allows you to go back to any point in the execution history of a Linux process, including stepping backwards and forwards by individual instructions, source lines, function calls and so on. It can regenerate the complete state of the process at any point in its execution, including information that is destroyed during execution.

LiveRecorder includes:

the LiveRecorder library which can be used by any Linux application to give it the ability to record its execution;

the live-record tool which records the execution of a process specified on the command line;

the usave [filename] and uload [-e|-goto-end] filename functionality inside UDB , which enables time travel debugging of LiveRecorder recordings .

Version History ¶

Release 7.2

Headline improvements in UDB:

The new undo command provides a common entry point for all Undo command-line tools and can be invoked with no arguments to print the list of available tools and an explanation of their purpose. UDB sessions can be now be exported and imported using the usession export and usession import commands, allowing session state such as bookmarks and breakpoints to be handed off between developers.

Headline improvements in LiveRecorder:

Nondeterministic events are now stored more efficiently in the event log, meaning that the portion of a recording file allocated to the event log will typically be 30% smaller and 50% faster to load, depending on the characteristics of the recorded program. The live-record tool has gained three new capabilities: The --retry-for DURATION option causes live-record to run the program repeatedly to generate a recording until DURATION has expired. The --list option causes live-record to print a list of LiveRecorder sessions that are running on the current system and their corresponding process IDs, allowing sessions to be terminated or discarded more easily. The existing --thread-fuzzing option now adds support for Thread Fuzzing on ARM64 platforms for parity with x86.

Release 7.1

UDB now saves the command history from all of your sessions by default. Use the up arrow key (or control-P) to access previously typed commands. After a period of inactivity at the prompt, UDB now provides command suggestions that could enhance your productivity with time travel debugging workflows.

Progress indicators are displayed for recording save operations in UDB and the command-line live-record tool. Recordings of x86-64 applications which use instructions from the Intel SHA instruction set extension (identified by the sha_ni CPU flag) can now be replayed on a machine with a CPU which does not implement those instructions. A new LiveRecorder API function creates point recordings which capture program state at a single point in time. When replaying these recordings, functions in the program can be evaluated in the same way as in a full LiveRecorder recording. A new LiveRecorder API function provides the option to save a recording file and stop recording when the event log is full , instead of rotating the circular event log.

For more information on these changes, see the release notes that are included in the docs/release-notes/ directory in your UDB release archive.

The GDB distribution that is bundled with UDB is updated from version 10.2 to 13.2. The new version enables multithreaded symbol loading by default amongst many other fixes and improvements.

Users can now optimise ioctl calls without revealing any sensitive hardware information to us, by writing a JSON configuration file with definitions of the ioctl calls that are to be optimised. See Self-service configuration .

Symbol files can now be encrypted using the encryption utility included in the UDB release archive and the encrypted symbol files can be loaded into UDB using the existing symbol-file command. See Symbol encryption .

Version 7.0 provides the first public early-access release of UDB and LiveRecorder for the 64-bit ARM architecture family (AArch64), targeting the Graviton2 CPU as used in AWS EC2.

Session state is now restored when loading a previously-loaded LiveRecorder recording. This includes breakpoint and watchpoint settings, current time and thread information, the ugo undo / ugo redo stack, minimum / maximum time limits, the wall-clock timezone, and signal handling information.

The live-record tool supports the new --record-on option, which defers recording until execution reaches a specified function, or when the name of the program or any of its descendants matches a specified pattern. See Using the LiveRecorder tool .

UDB now uses multiple threads to unpack files when loading recordings saved by UDB and LiveRecorder 7.0. This can reduce the time to load recordings by up to 40%, when the recording contains many large files, and when reading from the disk is fast enough.

UDB now uses up to four CPU cores to load event history from recordings, resulting in a performance improvement of between 1.3x and 4x when compared with the previous UDB release.

Recordings made on a machine which has a CPU which supports AVX-512 and uses GLIBC 2.34 or later can now be replayed on a machine with a CPU which does not support AVX-512, provided that certain conditions are met.

The live-record tool takes a new --verbose option, which emits extra information before and after recording.

Post Failure Logging can now watch expressions and generate logging statements any time the value of the expression changes.

LiveRecorder and UDB are able to attach to processes that link against recent glibc versions which use Restartable Sequences .

UDB is faster to load recordings of programs with a very large memory footprint or a large number of files with debugging information.

Shell behaviour under UDB is aligned with that of GDB, with UDB now supporting shell metacharacters in argument lists when starting applications.

User experience is improved when debugging inline functions, so that UDB now steps more reliably to inline call sites, and from inline call sites into inline functions.

Up to 4x faster reverse execution in freshly-loaded recordings where less than 15% of the maximum event log size was used.

New UDB last command which jumps to the last time in execution history when the value of a variable or expression was modified.

Reduced the time required to load LiveRecorder recordings.

Reverse execution commands in a freshly-loaded LiveRecorder recording are 3-4x faster, due to increased number of snapshots saved in recording files.

UDB displays progress indicators for long-running operations when in TUI mode and when using the Time Travel Debug for C/C++ extension for Microsoft Visual Studio Code .

Parallel Search - a performance-enhancement option that makes use of multi-core systems to improve the performance of reverse navigation commands - enabled in UDB by default.

Faster saving and loading of LiveRecorder recordings via LZ4 compression.

UDB displays a progress indicator when loading LiveRecorder recordings.

Upgraded the version of Python that is packaged with UDB and that is used by C++ pretty-printers from 3.6 to 3.10.

Dropped support for RHEL & CentOS 6.

Parallel Search - a performance-enhancement option that makes use of multi-core systems to improve the performance of reverse navigation commands. The speed-up achieved will depend on how far back in history the command moves; the more history executed, the greater the benefit.

While performing a reverse execution command (including reverse-step , reverse-next and the corresponding Reverse Step Into and Reverse Step Out buttons in Visual Studio Code) UDB will stop the program’s execution if the program hits a breakpoint or watchpoint.

The stepi and reverse-stepi commands now step one instruction forwards or backwards within the current thread. You can also use reverse-stepi -any-thread to step back exactly one instruction back in history, potentially switching threads in the process.

Documentation on using the Time Travel Debug for C/C++ extension for Microsoft Visual Studio Code is included in the UDB documentation .

Added the delete bookmark command for deleting bookmarks.

Time Travel Debug for C/C++ extension for Microsoft Visual Studio Code . Enables UDB’s Time Travel debugging capabilities in Microsoft’s Visual Studio Code editor. Requires UDB 6.5.

Added syntax highlighting for source code displayed by UDB, for instance when stopping at a breakpoint, when invoking the list command, or in TUI mode.

Added support for debuginfod in UDB. This allows UDB to fetch debug info and source code from debuginfod servers specified with the space-separated environment variable DEBUGINFOD_URLS .

Reverse commands now uniformly respect the same arguments (repeat count) and context (current frame) as their forwards equivalents.

Upgraded the version of GDB that is packaged with UDB from 9.2 to 10.2. Details of the various bugs that this fixes can be found at GDB News .

Deprecated all commands starting with the uinfo , ushow , uset and uclear prefixes in favour of the equivalent GDB prefixes:

uinfo → info

ushow → show

uclear → unset

Prefixes which don’t have a GDB equivalent, like ugo , were not renamed.

Status Prompt: UDB’s prompt displays the status of the debugged program and the progress through its execution history.

Undo/Redo: New UDB commands ugo undo ( uu ) and ugo redo ( ur ) support undoing and redoing navigation and time-travel commands.

Changed the default UDB event log mode to circular . This means that when the event log reaches its maximum size, old events are dropped to make room for new ones, allowing the program to continue running.

[Beta] Time Travel Debug for C/C++ extension for Visual Studio Code. Enables UDB’s time travel debugging capabilities in Microsoft’s enormously popular Visual Studio Code (VS Code) editor. Download the extension from Visual Studio Code Marketplace .

Reverse step commands ( reverse-finish , reverse-next , reverse-nexti , reverse-step , reverse-stepit , reverse-until ) remain in the same thread and are more consistently performant. The stepi and reverse-stepi commands can still be used to forward or reverse step single instructions while following thread switches.

[Experimental] A replay-time option provides 2-6x speedup by exploiting multiple CPU cores when performing reverse navigation commands including reverse-next , reverse-finish and reverse-continue .

Bookmarks that you place when replaying a LiveRecorder recording in UDB are automatically persisted across debugging sessions.

UDB displays a progress indicator when performing a long-running time travel operation.

Further improved the performance of the reverse-next command and other reverse operations, and improved the correctness of these operations when debugging highly optimized code.

UDB now uses styles and colours in the text printed at startup.

Improved performance of reverse operations in a number of situations including long-running operations and operations which stay in the same thread.

Deprecated all command line options starting with --undodb- in favour of options without any prefix. Refer to the changelog.txt file or product documentation for details.

UDB is the new name for UndoDB. Existing --undodb command-line options will be retained for 6.x releases.

Upgraded the version of GDB that is packaged with UDB from 8.1 to 9.2. New GDB features, such as new commands and styled output, are described at GDB News under the relevant sections.

Upgraded the version of Python that is packaged with UDB and that is used by C++ pretty-printers from 2.7 to 3.6. Python 2 only pretty printers shipped by older supported distros are converted automatically at load time into Python 3.

The Log Jump feature is now enabled for all existing and new UDB and LiveRecorder customers.

Support for applications written in the Java language.

Recordings are portable across machines with different CPU microarchitectures. See Recording Portability for details.

Core engine 25%-100% faster than 5.0 on diverse workloads.

• Getting started with UDB
• Invoking UDB
• Configuring UDB
• Initialization files
• UDB commands reference
• Breakpoints
• When did an expression last change value?
• Deferred recording
• Inferior calls
• Automation API: introduction
• Remote debugging
• Convenience variables
• Explicit libthread_db selection
• Symbol encryption
• Usage statistics collection

LiveRecorder

• Using the LiveRecorder tool
• Using the LiveRecorder API
• Loading and saving recordings
• Querying recordings
• Updating recordings
• Post Failure Logging
• Multi-Process Correlation for Shared Memory
• Thread Fuzzing
• AVX-512 recording portability

IDE integration

• Visual Studio Code

The Undo Engine

• System requirements
• Technical details
• Time notation
• Limitations
• Asynchronous I/O
• Self-service configuration

Tips and Tricks

• Tip and tricks when bugs won’t reproduce
• Setting up notifications for long operations

Time-Travel Debugging Production Code

Loren Sands-Ramshaw

Developer Relations Engineer

In this post, I’ll give an overview of time-travel debugging (what it is, its history, how it’s implemented) and show how it relates to debugging your production code.

Normally, when we use debuggers, we set a breakpoint on a line of code, we run our code, execution pauses on our breakpoint, we look at values of variables and maybe the call stack, and then we manually step forward through our code’s execution. In time-travel debugging , also known as reverse debugging , we can step backward as well as forward. This is powerful because debugging is an exercise in figuring out what happened: traditional debuggers are good at telling you what your program is doing right now, whereas time-travel debuggers let you see what happened. You can wind back to any line of code that executed and see the full program state at any point in your program’s history.

History and current state

It all started with Smalltalk-76, developed in 1976 at Xerox PARC . ( Everything started at PARC 😄.) It had the ability to retrospectively inspect checkpointed places in execution. Around 1980, MIT added a “retrograde motion” command to its DDT debugger , which gave a limited ability to move backward through execution. In a 1995 paper, MIT researchers released ZStep 95, the first true reverse debugger, which recorded all operations as they were performed and supported stepping backward, reverting the system to the previous state. However, it was a research tool and not widely adopted outside academia.

ODB, the Omniscient Debugger , was a Java reverse debugger that was introduced in 2003, marking the first instance of time-travel debugging in a widely used programming language. GDB (perhaps the most well-known command-line debugger, used mostly with C/C++) added it in 2009.

Now, time-travel debugging is available for many languages, platforms, and IDEs, including:

• Replay for JavaScript in Chrome, Firefox, and Node, and Wallaby for tests in Node
• WinDbg for Windows applications
• rr for C, C++, Rust, Go, and others on Linux
• Undo for C, C++, Java, Kotlin, Rust, and Go on Linux
• Various extensions (often rr- or Undo-based) for Visual Studio, VS Code, JetBrains IDEs, Emacs, etc.

Implementation techniques

There are three main approaches to implementing time-travel debugging:

• Record & Replay : Record all non-deterministic inputs to a program during its execution. Then, during the debug phase, the program can be deterministically replayed using the recorded inputs in order to reconstruct any prior state.
• Snapshotting : Periodically take snapshots of a program’s entire state. During debugging, the program can be rolled back to these saved states. This method can be memory-intensive because it involves storing the entire state of the program at multiple points in time.
• Instrumentation : Add extra code to the program that logs changes in its state. This extra code allows the debugger to step the program backwards by reverting changes. However, this approach can significantly slow down the program’s execution.

rr uses the first (the rr name stands for Record and Replay), as does Replay . WinDbg uses the first two, and Undo uses all three (see how it differs from rr ).

Time-traveling in production

Traditionally, running a debugger in prod doesn’t make much sense. Sure, we could SSH into a prod machine and start the process handling requests with a debugger and a breakpoint, but once we hit the breakpoint, we’re delaying responses to all current requests and unable to respond to new requests. Also, debugging non-trivial issues is an iterative process: we get a clue, we keep looking and find more clues; discovery of each clue is typically rerunning the program and reproducing the failure. So, instead of debugging in production, what we do is replicate on our dev machine whatever issue we’re investigating and use a debugger locally (or, more often, add log statements 😄), and re-run as many times as required to figure it out. Replicating takes time (and in some cases a lot of time, and in some cases infinite time), so it would be really useful if we didn’t have to.

While running traditional debuggers doesn’t make sense, time-travel debuggers can record a process execution on one machine and replay it on another machine. So we can record (or snapshot or instrument) production and replay it on our dev machine for debugging (depending on the tool, our machine may need to have the same CPU instruction set as prod). However, the recording step generally doesn’t make sense to use in prod given the high amount of overhead—if we set up recording and then have to use ten times as many servers to handle the same load, whoever pays our AWS bill will not be happy 😁.

But there are a couple scenarios in which it does make sense:

• Undo only slows down execution 2–5x , so while we don’t want to leave it on just in case, we can turn it on temporarily on a subset of prod processes for hard-to-repro bugs until we have captured the bug happening, and then we turn it off.
• When we’re already recording the execution of a program in the normal course of operation.

The rest of this post is about #2, which is a way of running programs called durable execution .

Durable execution

What’s that.

First, a brief backstory. After Amazon (one of the first large adopters of microservices) decided that using message queues to communicate between services was not the way to go (hear the story first-hand here ), they started using orchestration. And once they realized defining orchestration logic in YAML/JSON wasn’t a good developer experience, they created AWS Simple Workfow Service to define logic in code. This technique of backing code by an orchestration engine is called durable execution, and it spread to Azure Durable Functions , Cadence (used at Uber for > 1,000 services ), and Temporal (used by Stripe, Netflix, Datadog, Snap, Coinbase, and many more).

Durable execution runs code durably—recording each step in a database, so that when anything fails, it can be retried from the same step. The machine running the function can even lose power before it gets to line 10, and another process is guaranteed to pick up executing at line 10, with all variables and threads intact.[^1] It does this with a form of record & replay: all input from the outside is recorded, so when the second process picks up the partially-executed function, it can replay the code (in a side-effect–free manner) with the recorded input in order to get the code into the right state by line 10.

Durable execution’s flavor of record & replay doesn’t use high-overhead methods like software JIT binary translation , snapshotting, or instrumentation. It also doesn’t require special hardware. It does require one constraint: durable code must be deterministic (i.e., given the same input, it must take the same code path). So it can’t do things that might have different results at different times, like use the network or disk. However, it can call other functions that are run normally ( “volatile functions” , as we like to call them 😄), and while each step of those functions isn’t persisted, the functions are automatically retried on transient failures (like a service being down).

Only the steps that require interacting with the outside world (like calling a volatile function, or calling sleep('30 days') , which stores a timer in the database) are persisted. Their results are also persisted, so that when you replay the durable function that died on line 10, if it previously called the volatile function on line 5 that returned “foo”, during replay, “foo” will immediately be returned (instead of the volatile function getting called again). While yes, it adds latency to be saving things to the database, Temporal supports extremely high throughput (tested up to a million recorded steps per second). And in addition to function recoverability and automatic retries, it comes with many more benefits , including extraordinary visibility into and debuggability of production.

Debugging prod

With durable execution, we can read through the steps that every single durable function took in production. We can also download the execution’s history, checkout the version of the code that’s running in prod, and pass the file to a replayer (Temporal has runtimes for Go, Java, JavaScript, Python, .NET, and PHP) so we can see in a debugger exactly what the code did during that production function execution. Read this post or watch this video to see an example in VS Code.[^2]

Being able to debug any past production code is a huge step up from the other option (finding a bug, trying to repro locally, failing, turning on Undo recording in prod until it happens again, turning it off, then debugging locally). It’s also a (sometimes necessary) step up from distributed tracing.

💬 Discuss on Hacker News , Reddit , Twitter , or LinkedIn .

I hope you found this post interesting! If you’d like to learn more about durable execution, I recommend reading:

• Building reliable distributed systems
• How durable execution works

and watching:

• Introduction to Temporal
• Why durable execution changes everything

Thanks to Greg Law, Jason Laster, Chad Retz, and Fitz for reviewing drafts of this post.

[^1]: Technically, it doesn’t have line-by-line granularity. It only records certain steps that the code takes—read on for more info ☺️. [^2]: The astute reader may note that our extension uses the default VS Code debugger, which doesn’t have a back button 😄. I transitioned from talking about TTD to methods of debugging production code via recording, so while Temporal doesn’t have TTD yet, it does record all the non-deterministic inputs to the program and is able to replay execution, so it’s definitely possible to implement. Upvote this issue or comment if you have thoughts on implementation!

Practical Debugging at Scale pp 109–116 Cite as

Time Travel Debugging

• Shai Almog 2  
• First Online: 03 January 2023

599 Accesses

I think there’s no better quote for a chapter about time travel debuggers than this. A forgettable Shakespeare quote became significant as Huxley plucked it from relative obscurity with his groundbreaking book. This is the perfect analogy for time travel debuggers, a.k.a. back-in-time debuggers.

This is a preview of subscription content, log in via an institution .

Buying options

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

https://vmblog.com/archive/2021/12/03/lightrun-2022-predictions-logging-debugging-and-observability-in-2022.aspx

https://github.com/rr-debugger/rr/

Author information

Authors and affiliations.

Tel Aviv, Israel

You can also search for this author in PubMed   Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature

About this chapter

Cite this chapter.

Almog, S. (2023). Time Travel Debugging. In: Practical Debugging at Scale. Apress, Berkeley, CA. https://doi.org/10.1007/978-1-4842-9042-2_5

Download citation

DOI : https://doi.org/10.1007/978-1-4842-9042-2_5

Published : 03 January 2023

Publisher Name : Apress, Berkeley, CA

Print ISBN : 978-1-4842-9041-5

Online ISBN : 978-1-4842-9042-2

eBook Packages : Professional and Applied Computing Apress Access Books Professional and Applied Computing (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

AdaCore Blog

Time travel debugging in gnat studio with gdb and rr, by ghjuvan lacambre – mar 17, 2020, intro­duc­tion #.

Tra­di­tion­al debug­ging tools usu­al­ly let you place break­points and fol­low the program’s flow. You can see the evo­lu­tion of the program’s state, but if you want to know how the pro­gram arrived to a cer­tain state, you have to add log­ging facil­i­ties to your pro­gram in order to be able to retrace state’s his­to­ry. The ​ “ reach a cer­tain point in the pro­gram -> dis­cov­er that a vari­able doesn’t have the right val­ue -> add log­ging to fig­ure out what changed it -> launch the pro­gram again” cycle can be a very time con­sum­ing process which can be made even slow­er by non-deter­min­is­tic, hard to repro­duce bugs. Note that time trav­el debug­ging is usu­al­ly very os-depen­dent and that the two tools described here are linux-only.

Time trav­el with GDB #

One solu­tion to this prob­lem is time trav­el debug­ging. GDB imple­ments time trav­el debug­ging with its record com­mand. To use it in GNAT Stu­dio, one just needs to make sure that GDB is used rather than GDB mi in GNAT Studio’s pref­er­ences (Edit > Pref­er­ences > Debug­ger), as shown in the screen­shot below.

Then, after build­ing the project in debug mode, and start­ing it with the ​ ‘ start ’ com­mand, GDB ’s ​ ‘ record full ’ com­mand will start the record­ing of the program’s exe­cu­tion. Break­points can be then placed in points of inter­est and, when going back is time is need­ed, the ​ ‘ reverse-next ’, ​ ‘ reverse-step ’ and ​ ‘ reverse-finish ’ com­mands can be used.

GDB ’s record and replay func­tions are extreme­ly use­ful: they sup­port a wide vari­ety of archi­tec­tures and allow record­ing only parts of a program’s exe­cu­tion. They have down­sides though: some­times the buffer used for record­ing exe­cu­tion will get full and you’ll have to dis­card your record his­to­ry. You can only record exe­cu­tion from GDB , they make exe­cu­tion slow­er and do not work with pro­grams that use AVX / SSE instructions.

Time trav­el with RR #

If the archi­tec­ture you’re tar­get­ing is x 86 _​ 64 , there is a solu­tion to the above prob­lems: RR . RR (which stands for Record &  Replay) is a tool devel­opped by Mozil­la to help with debug­ging Fire­fox. RR records a whole program’s exe­cu­tion, from start to fin­ish and lets you replay it as many times as you want, even across dif­fer­ent GDB ses­sions. It does make pro­gram exe­cu­tion slow­er, but not as much as GDB ’s record and replay feature.

In order to use RR from GNAT Stu­dio, first install RR (instal­la­tion instruc­tions are avail­able at the bot­tom of the RR web­site ). Once this is done, make sure that the set­ting that con­trols the gran­u­lar­i­ty of perf event report­ing in your ker­nel is set to 1 (this can be done by cat /proc/sys/kernel/perf_event_paranoid' and set­ting it to 1  is as sim­ple as 'echo 1 | sudo tee /proc/sys/kernel/perf_event_paranoid ). Just like with time trav­el with GDB , make sure that you are using ​ ‘ gdb ’ and not ​ ‘ gdb mi ’ in your GNAT Stu­dio debug­ging preferences.

Here’s a mini-tuto­r­i­al that shows how to use RR with GNAT Stu­dio. In order to fol­low along, cre­ate a new ​ “ Sim­ple Ada Project” in GNAT Stu­dio. Set the con­tent of ​ ‘ src/main.adb ’ to the following:

This pro­gram has a  10 % chance of crash­ing, so we will first need to record a crash­ing run. To do that, com­pile the project in debug mode (‘ View > Scenario ’ , set ​ ‘ Build Mode ’ to ​ ‘ debug ’ and then press ‘ <F4> ’). This will pro­duce a bina­ry in your project’s ​ ‘ obj/debug ’ direc­to­ry. Nav­i­gate to this direc­to­ry in a shell and try to record the program’s exe­cu­tion with RR with the fol­low­ing com­mand: ​ ‘ rr record ./main ’. You can then run ​ ‘ rr replay ’ to have RR start GDB for you (but don’t do this — we’ll use GNAT Stu­dio instead!).

You might have been lucky enough to get a crash on your first record­ing, but if that is not the case, record­ing exe­cu­tions until you do get a crash is easy: when they raise an excep­tion that isn’t caught, Ada pro­grams exit with a non-zero exit code. We can use this to have the shell record exe­cu­tions until we do get a fail­ure, like this: ​ ‘ while rr record ./main ; do true ; done ’.

Now that we have record­ed a crash, we could debug it in GDB from our shell (with ​ ‘ rr replay ’), or from GNAT Stu­dio. We’ll try GNAT Studio.

In your shell, run ​ ‘ rr replay -s 12345 & ’. RR will print a mes­sage telling you to run GDB , which you shouldn’t do (because we’ll use a  GDB process start­ed by GNAT Stu­dio). Copy the path print­ed at the end of the GDB com­mand to your clip­board. In GNAT Stu­dio, start the debug­ger (‘ Debug > Initialize > Main ’). You should get a  GDB win­dow at the bot­tom of your screen. In that win­dow, run the fol­low­ing commands:

GNAT Studio’s debug­ger should now be con­nect­ed to RR . This means that all GDB for time trav­el (reverse-next, reverse-step, reverse-fin­ish…) are avail­able to us. Since we know the error hap­pens on line 36 , we can just put a break­point there: either click on line 36  in the num­ber col­umn or type ​ ‘ break main.adb:36 ’ in GNAT Studio’s GDB win­dow. We can exe­cute the pro­gram until we reach that break­point with ​ ‘ continue ’, by press­ing ‘ <F8> ’ or by click­ing on the ​ “ play” but­ton at the top of GNAT Stu­dio. ​ ‘ print Bug ’ (or hov­er­ing the mouse over the vari­able) shows that the pro­gram reached this line because the vari­able named ​ ‘ Bug ’ is set to true. But what pro­ce­dure set ​ ‘ Bug ​ ” s val­ue to true? Right click­ing on ​ ‘ Bug ’ and select­ing ​ ‘ Debug > Set watchpoint on Bug ’ (or run­ning ​ ‘ watch Bug ’ in the GDB win­dow) and typ­ing ​ ‘ reverse-continue ’ in the GDB win­dow will tell us that it hap­pened either in ​ ‘ Make_Bug ’ or ​ ‘ Do_Bug ’.

This is a very sim­ple exam­ple, but it hope­ful­ly shows how use­ful RR can be and how sim­ple it is to use it from GNAT Studio!

Posted in #IDE     #gdb    

About Ghjuvan Lacambre

Ghjuvan Lacambre is a GNAT compiler engineer at AdaCore. Passionate about everything more or less related to programming, he has a particular interest in programming language and static analysis and spends most of his free time improving the tools he uses for programming.

Last updated Fri Oct 18 17:19

• mailing list
• #rr on chat.mozilla.org

what rr does

Rr aspires to be your primary c/c++ debugging tool for linux, replacing — well, enhancing — gdb. you record a failure once, then debug the recording, deterministically, as many times as you want. the same execution is replayed every time., rr also provides efficient reverse execution under gdb. set breakpoints and data watchpoints and quickly reverse-execute to where they were hit., rr works on real applications and is used by many developers to fix real bugs. it makes debugging hard bugs much easier, but also speeds up debugging of easy bugs..

getting started

Build from source, or in fedora:.

You can also install rr directly from the package manager.

Or in Ubuntu:

Background and motivation, rr in context, limitations, further reference.

This browser is no longer supported.

Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.

Time Travel Debugging - Overview

• 1 contributor

What is Time Travel Debugging?

Time Travel Debugging is a tool that allows you to capture a trace of your process as it executes and then replay it later both forwards and backwards. Time Travel Debugging (TTD) can help you debug issues easier by letting you "rewind" your debugger session, instead of having to reproduce the issue until you find the bug.

TTD allows you to go back in time to better understand the conditions that lead up to the bug and replay it multiple times to learn how best to fix the problem.

TTD can have advantages over crash dump files, which often miss the state and execution path that led to the ultimate failure.

In the event you can't figure out the issue yourself, you can share the trace with a coworker and they can look at exactly what you're looking at. This can allow for easier collaboration than live debugging, as the recorded instructions are the same, whereas the address locations and code execution will differ on different PCs. You can also share a specific point in time to help your coworker figure out where to start.

TTD is efficient and works to add as little as possible overhead as it captures code execution in trace files.

TTD includes a set of debugger data model objects to allow you to query the trace using LINQ. For example, you can use TTD objects to locate when a specific code module was loaded or locate all of the exceptions.

Requirements

Time Travel Debugging is integrated with WinDbg , providing seamless recording and replay experience.

To use TTD, you need to run the debugger elevated. Install WinDbg using an account that has administrator privileges and use that account when recording in the debugger. In order to run the debugger elevated, select and hold (or right-click) the WinDbg icon in the Start menu, and then select More > Run as Administrator.

The recording may contain personally identifiable or security related information

The created trace file that contains the recording may contain personally identifiable or security related information, including but not necessarily limited to file paths, registry, memory or file contents. Exact information depends on target process activity while it was recorded. Be aware of this when sharing recording files with other people.

TTD.exe command line recording utility

In addition to recording traces in the WinDbg UI, there is a TTD.exe command line utility available to record a trace.

You may have scenarios where only the TTD command line recorder is required: recording on a PC without installing the debugger, advanced recording scenarios, test automation, etc. In these scenarios you can install just the TTD command line recorder through a URL. For more information, see Time Travel Debugging - TTD.exe command line utility .

Comparison of Debugging Tools

This table summarizes the pros and cons of the different debugging solutions available.

Video Training

To learn more about TTD see these videos.

Defrag Tools 185 - Ivette and JamesP go over the basics of TTD and demo some features in WinDbg

Defrag Tools 186 - Jordi and JCAB demo more great features of TTD in WinDbg

CppCon (YouTube) - Jordi, Ken and JamesM presented TTD in WinDbg at CppCon 2017

Trace file basics

Trace file size.

The trace file can get big and the user of TTD needs to make sure that there is adequate free space available. If you record a program for even a few minutes, the trace files can quickly grow to be several gigabytes. TTD does not set a maximum size of trace files to allow for complex long running scenarios. Quickly re-creating the issue, will keep the trace file size as small as possible.

Trace and index files

A trace file ( .run ) stores the code execution during recording.

Once the recording is stopped, an index file ( .idx ) is created to optimize access to the trace information. Index files are also created automatically when WinDbg opens trace files.

Index files can also be large, typically twice as large as the trace file.

You can recreate the index file from the trace file using the !tt.index command.

Recording errors and other recording output is written to a WinDbg log file.

All of the output files are stored in a location configured by the user. The default location is in the users document folder. For example, for User1 the TTD files would be stored here:

For more information on working the trace files, see Time Travel Debugging - Working with trace files .

Things to look out for

Anti-virus incompatibilities.

You may encounter incompatibilities because of how TTD hooks into process to record them. Typically issues arise with anti-virus or other system software that is attempting to track and shadow system memory calls. If you run into issues of with recording, such as an insufficient permission message, try temporarily disabling any anti-virus software.

Other utilities that attempt to block memory access, can also be problematic, for example, the Microsoft Enhanced Mitigation Experience Toolkit.

Another example of an environment that conflicts with TTD, would be the electron application framework. In this case the trace may record, but a deadlock or crash of the process being recorded is also possible.

User mode only

TTD currently supports only user mode operation, so tracing a kernel mode process is not possible.

Read-only playback

You can travel back in time, but you can't change history. You can use read memory commands, but you can't use commands that modify or write to memory.

System Protected Processes

Some Windows system protected processes, such as Protected Process Light (PPL) process are protected, so the TTD cannot inject itself into the protected process to allow for the recording of the code execution.

Performance impact of recording

Recording an application or process impacts the performance of the PC. The actual performance overhead varies based upon the amount and type of code being executed during recording. You can expect about a 10x-20x performance hit in typical recording scenarios. Sometimes there will not be a noticeable slowdown but for more resource intensive operations (i.e. File Open dialog) you can see the impact of recording.

Trace file errors

There are some cases where trace file errors can occur. For more information, see Time Travel Debugging - Troubleshooting .

Advanced Features of Time Travel Debugging

Here's some of the most notable TTD advanced features.

Timelines are a visual representation of events that happen during the execution. These events can be locations of: breakpoints, memory read/writes, function calls and returns, and exceptions. For more information about timelines, see WinDbg - Timelines .

Debugger data model support

• Built in data model support - TTD includes data model support. Using LINQ queries to analyze application failures can be a powerful tool. You can use the data model window in WinDbg to work with an expandable and browsable version of ‘dx’ and ‘dx -g’, letting you create tables using NatVis, JavaScript, and LINQ queries.

For general information about the debugger data model, see WinDbg - Data model . For information about working with the TTD debugger object model, see Time Travel Debugging - Introduction to Time Travel Debugging objects .

Scripting support

• Scripting Automation - Scripting support for JavaScript and NatVis allows for the automation of problem investigation. For more information, see Time Travel Debugging - JavaScript Automation .

For general information about working with JavaScript and NatVis, see WinDbg - Scripting .

TTD.exe Command line utility

The TTD.exe command line utility to record traces is available. For more information, see Time Travel Debugging - TTD.exe command line utility .

Managed code TTD support

You can use the SOS debugging extension (sos.dll) running in 64 bit mode to debug managed code using TTD in WinDbg. For more information, see Debugging Managed Code Using the Windows Debugger .

Getting started with TTD

Review these topics to record and replay a trace file as well as for information on working with trace files and troubleshooting.

• Time Travel Debugging - Record a trace
• Time Travel Debugging - Replay a trace
• Time Travel Debugging - Working with trace files
• Time Travel Debugging - Troubleshooting
• Time Travel Debugging - Sample App Walkthrough

These topics describe additional advanced functionality in time travel debugging.

• Time Travel Debugging - Introduction to Time Travel Debugging objects
• Time Travel Debugging - JavaScript Automation

Was this page helpful?

Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see: https://aka.ms/ContentUserFeedback .

Submit and view feedback for

Additional resources