Design to Debug

Introduction

For a complex project, most development time is spent debugging, but often the experience is very poor.

Furthermore, the ability to understand and debug a program limits the complexity that a software developer can handle.

Therefore, a successful program should be designed to be debuggable - easy to inspect and modify.

Understanding State

A clear understanding of program state speeds debugging and improves design.

Context should expressed in a way that can be navigated quickly.

Diagrams

• Most powerful tool while debugging
• Detailed Graphviz diagrams of internal graph representation
• Visually detect problems quickly
• Provides clues for deeper analysis

Structured Logs

• Context automatically added on event
• Indentation indicates context
• Deferred string rendering for minimal overhead

Closer Inspection

Once the area of interest had been located, tools are necessary to pick apart the details.

Often, the program will be run many times, only making small changes, while looking at different aspects, so this is the "inner loop" of debugging.

LLDB

• Nicer interface than GDB, Python scripting
• Conditional breakpoints to stop at reduction of a particular cell
• Commands for often repeated tasks, such as generating diagrams

Application Breakpoints

• faster than LLDB/GDB alone
• Break on log messages
  • On the same message afterwards
  • On a range of log messages
• Log tags are four character alphanumeric tags that are easy to read and remember

Application Watchpoints

• Much faster and less tedious
• Break on cell throughout life cycle
  • Allocation, reduction, trace, free
  • Relies on repeatable allocation

Breakpoint & Watchpoint Graph Integration

• Generate graphs at each breakpoint/watchpoint
  • Labeled with a log tag
  • Any cells indices logged are highlighted in the graph

Detecting bugs

The ability to quickly verify changes gives confidence and allows aggressive development while maintaining stability.

Asserts

• assert_error() is used liberally
• Most functions have preconditions asserted
• Selective mark-and-sweep to check reference counts
• Very useful in conjunction with afl

Tests

• As easy as void test_foo() {...}
• Run with eval -t foo
• make test runs all tests and diffs with stored test output

American Fuzzy Lop (afl)

• Powerful fuzzer which is good at triggering asserts
• Not all asserts are bad; assert_throw() for expected failures
  • Assert_throw() doesn't count as a crash
  • Also allows for cleaner handling of user errors

Tooling development

• Note tooling improvements that would be nice to have
• Work on them when burnt out on the main task
• Typically, completing a new tool inspires work on the main task with the new tool