From Linkage chapter of the Rust Reference, we can see there are seven kinds of crates: bin, lib, dylib, staticlib, cdylib, rlib, and proc-macro.

⊕ I had originally intended to cover all seven, but the time got late and the post got long and I decided that proc-macro can be dealt with another day.

What this post is going to do is walk through the first six of the crate types listed above and demonstrate: how to build an example of such a crate, how to link to it, and how to run with that linked crate.

In later posts, I will explore the various attributes and command-line flags that can influence the linking step. But right now, I want to establish the foundation for that later discussion.

⊕ Some phenomena require mixing multiple crate types in order to observe corner cases that are worth addressing. I am leaving that for a future post as well.

These initial examples are as simple as possible. We will want to actually demonstrate each case running. Since most crate types are not executable, that means we will need multiple crates in almost all of our examples.

Also, I will not be using Cargo in any of my examples. I will try to note points where things I am doing are deviating from what you would normally do if you were using Cargo (such as my use of extern crate in these examples), and hopefully a later post will explore the status quo of how Cargo handles various linkage scenarios. But, I am taking baby steps here; lets focus on rustc alone for now.

At a very high level, we will be looking at object file structures that look something like this:

That is: Given some input like simple-bin.rs, you can feed into rustc and get an executable as its output.

The actual code for a case like the above is trivial:

1
2
3

// simple-bin.rs
#![crate_type="bin"] // ("bin" is the default; other examples vary here.)
fn main() { println!("Running main from {}", file!()); }

With that code in place, you can compile and run the program. (In practice developers would usally do this via cargo run.)

1
2
3
4
5
6

% rustc --out-dir out/bins/ps/dsb simple-bin.rs
% ls out/bins/ps/dsb
simple-bin
% out/bins/ps/dsb/simple-bin
Running main from simple-bin.rs
%

⊕ I am using a bespoke directory naming convention unique to this blog post. For example, in --out-dir in this invocation: All output goes under out. Executable binaries go under out/bin. If static linkage was preferred during the build, then it goes under out/bin/ps (for “prefer static”); if dynamic linkage was preferred, then under out/bin/pd (for “prefer dynamic”); the post explains this “preference” further down, with the discussion of -Cprefer-dynamic. Finally, I added a directory named via a unique key for each test (here, dsb, for “demo-simple-bin”), so that the ls invocations are tidy. In these examples, I will be overriding the default output directory so that each example will have its own direcrory. This forces the examples to specify precisely where it is getting its libraries from; it is a good way to double-check one’s understanding of what is actually happening under-the-hood.

Simple linkage of a lib crate

Of course, since our subject of interest is linking, we will want to look at examples that involve library crates. Here is perhaps the simplest intance of that:

What is the advantage of separating your code like this, into separate libraries that are subsequently linked? One reason to do this is to identify which code is under active development, and focus on its development separately from the overall product. So, if simple-lib.rs were under development, we could focus just on that, and not have our tools spend time processing demo-simple-lib.rs. Or vice-versa: If demo-simple-lib.rs were under development, then we could focus on that, and our tools would process, at most, the libsimple_lib.rlib library produced by running rustc on simple-lib.rs.

Here are the code and commands that correspond to the picture above for compiling demo-simple-lib.

1
2
3
4
5
6
7

// simple-lib.rs

// generate a library (what kind? The "default" for this platform)
#![crate_type="lib"]

// Here's the function we'll provide to our clients.
pub fn main() { println!("Running main from {}", file!()); }

1
2
3
4
5
6
7

// demo-simple-lib.rs

// link to library built from simple-lib.rs
extern crate simple_lib;

// call function it exports
fn main() { simple_lib::main(); }

We have omitted #![crate_type="bin"] from demo-simple-lib.rs because that is the default crate type.

(In practice, most modern Rust code does not use extern crate; instead, people let Cargo handle injecting compiler options that achieve a similar effect. For now, we will use extern crate in our initial examples, so that the Rust code itself indicates its dependence on a separate library.)

With the above two files in place, we can compile each of them and run the resulting binary.

1
2
3
4
5
6
7
8
9

% rustc --out-dir out/ps/sl simple-lib.rs
% ls out/ps/sl
libsimple_lib.rlib
% rustc --out-dir out/bins/ps/dsl demo-simple-lib.rs -Lout/ps/sl
% ls out/bins/ps/dsl
demo-simple-lib
% out/bins/ps/dsl/demo-simple-lib
Running main from simple-lib.rs
%

⊕ If you’re squinting at the diagram and wondering how you might confirm that the relations it describes properly reflect what these commands are doing, I recommend you check out the “Who is using that library?” appendix at the end of this post. It walks through some of the issues that arise when linking crates together.

The first rustc invocation, compiling simple-lib.rs, is much like our simple-bin.rs example. The second rustc invocation, compiling demo-simple-lib.rs, has something new: it is passing the option -Lout/sl, which tells the compiler “if you need to resolve any external crates, you should add out/sl to the list of paths you will search for them.”

The demo-simple-lib example demonstrates Rust’s lib crate type, which is specified as a compiler-defined choice from one of the flavors of libraries that Rust supports.

At the time of this blog post, Rust’s lib crate type maps to rlib, at least on my machine. We will talk more about rlib further down; but for now, you can just keep in mind the points made above for simple-lib: the compiler itself will read the metadata stored in rlib crates, and the linker also extracts definitions from rlib crates as well.

We will now step through other supported crate types, and provide a similar demonstration of how they operate.

⊕ Re-reading the overall post now, I am thinking that I should have put dylib at the end, due to the number of rabbit holes it opens up. Maybe I will edit the post in the future and do that rearrangement, so that people can see the “easy cases” first.

Dynamic Libraries: dylib

The next crate type listed in the Rust reference is dylib.

This is a similar picture to the one presented above for lib. In fact, if you look at the two pictures side-by-side, most of the differences can be attributed to uniform renaming … except for one: There is now an extra arc in the diagram, connecting the demo-simple-dylib executable directly to the libsimple_dylib.so library file.

This is because a dylib crate is a dynamic library; it is meant to be loaded dynamically, when the program is executed. There are a couple different reasons one might want this: Perhaps the program is meant to support a “plug-in” architecture, where one can get load up new behaviors by swapping in a different dynamic library. Another common reason is to reduce executable binary sizes: if many programs depend on the same external crate, then it might be more efficient to have them all share the same dylib.

Note: The dylib format is not guaranteed to remain stable between different versions of the Rust compiler. Therefore, if you use the dylib format, you need to ensure that all crates that are sharing the same dylib and the dylib itself are all built with the same version of the Rust compiler.

Here are the code and commands that correspond to the picture above for compiling `demo-simple-dylib".

1
2
3

// simple-dylib.rs
#![crate_type="dylib"]
pub fn main() { println!("Running main from {}", file!()); }

1
2
3

// demo-simple-dylib
extern crate simple_dylib;
fn main() { simple_dylib::main(); }

With the above two files in place, we can compile each of them and run the resulting binary.

1
2
3
4
5
6
7

% rustc --out-dir out/pd/sd -C prefer-dynamic simple-dylib.rs
% ls out/pd/sd
libsimple_dylib.so
% rustc --out-dir out/bins/ps/dsd demo-simple-dylib.rs -Lout/pd/sd
% ls out/bins/ps/dsd
demo-simple-dylib
% LD_LIBRARY_PATH=out/pd/sd:$(rustc --print=sysroot)/lib out/bins/ps/dsd/demo-simple-dylib

You might have noticed that our demo-simple-dylib code is very similar to the demo-simple-lib code; the only effective change to the source is the difference in crate_type for simple-dylib.rs. On the other hand, the commands to compile these inputs and run the executable have had some pretty severe changes applied to them. Let us explore why those changes were necessary.

Adapting command lines to meet needs of crate type

If we tried to reuse the demo-simple-lib sequence of steps to compile these files and run the presumably generated executable, we hit some roadblocks.

Why was -Cprefer-dynamic added

First, if we tried to compile simple-dylib.rs using the same command that was used for simple-lib.rs, it seems to work at first:

1
2
3
4
5

% mkdir -p out/ps/sd
% rustc --out-dir out/ps/sd simple-dylib.rs
% ls out/ps/sd
libsimple_dylib.so
%

The problem arises when we try to use that generated dylib that was compiled; that step causes the following error output:

1
2
3
4
5
6
7
8
9
10

```sh
% rustc --out-dir out/bins/ps/dsd demo-simple-dylib.rs -Lout/ps/sd
error: cannot satisfy dependencies so `std` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away

error: cannot satisfy dependencies so `core` only shows up once
  |
  = help: having upstream crates all available in one format will likely make this go away
[...]

and so on, for all of the upstram crates std, core, compiler_builtins, rustc_std_workspace_core, alloc, libc, unwind, cfg_if, hashbrown, rustc_std_workspace_alloc, std_detect, rustc_demangle, addr2line, gimli, object, memchr, miniz_oxide, adler, and panic_unwind.

Understanding why this happens requires we take a step back.

The compiler needs to decide, in the absence of explicit indication from the programmer, how each dependency should be incorporated into the executable binary being generated. Namely, should a given dependency be “statically linked” into the output object (which effectively means that anything the object needs will be copied into the output from the referenced library), or should the given dependency be “dynamically linked” (which means the executable carries a dependence on the dynamic library, that will need to be resolved at runtime). This is a non-trivlal decision, because Rust allows individual crates to opt-into supporting multiple distinct crate types: a single crate can say “I can be used as an rlib or a dylib; I will let my downstream client decide which object file they want to pull in for their needs.”

But if no one tells the compiler what choice to make, the Rust compiler applies a simple-minded tactic for guessing what options would be best, and currently, that tactic is heavily influenced by the presence or absence of -C prefer-dynamic.

When simple-dylib.rs was compiled without -C prefer-dynamic, the compiler interpreted the absence of that flag as a signal that the compiler should attempt to link all dependencies of simple-dylib.rs statically.

Furthermore, the compiler manages to succeed at this static linkage of those dependencies, but in doing so, it has it impossible to link the resulting statically-linked crate into demo-simple-dylib.rs.

⊕ A reasonable person might note here: “Why is the linkage of std from demo-simple-dylib treated as its own distinct thing? In other words, why doesn’t the compiler just let simple-dylib, which has already statically-linked in those crates, provide them to demo-simple-dylib? And you wouldn’t be alone in thinking this: Alex Crichton, who is responsible for the basic logic in use here, made the same suggestion in rust-lang/rust#34909. I plan to explore this question more in a future blog post. The demo-simple-dylib crate also wants to link to all the same upstream crates, and the compiler rejects this, saying it is not legal for both the simple-dylib and the demo-simple-dylib crates to have duplicate copies of those dependencies.

⊕ The semantics described here dates from RFC 404, which was introduced in 2014 before Rust had even hit 1.0 status. That RFC itself refers to the comments in the code as the documentation; those comments have moved as the compiler has gone through various refactorings, but the most recent version can be found here in rustc_metadata::dependency_format

In the absence of -Cprefer-dynamic and --extern flags (and also absent any constraints forcing everything to be statically linked), the default logic of the compiler when trying to decide which upstream crate types to use is as follows:

1. First try to statically link all of the upstream dependencies via their .rlib libraries. If that succeeds, we are done.

2. If static linking failed, then either linking in general is impossible, or at least one dependency will have to be a dynamic library. Since at least one dependency has to be dynamic, the compiler, as a simple-minded tactic, tries to satisfy as many upstream dependencies as possible via their dylib object files.

However, if one does provide -Cprefer-dynamic, then that tells the compiler to not attempt the static link step, and instead to prefer to use dylib whenever possible for its upstream dependencies. And that is the fix we need here: we need simple-lib.dylib to prefer the dylib version of std.

1

% rustc --out-dir out/pd/sd -C prefer-dynamic simple-dylib.rs

Once that is in place, everything else follows suit.

First, consider how demo-simple-dylib.rs is built.

1

% rustc --out-dir out/bins/ps/dsd demo-simple-dylib.rs -Lout/pd/sd

⊕ Note that this story here is simple in part because simple-dylib.rs was only compiled to one crate type. If we had generated both dylib and rlib crates for it, then the presence/absence of -Cprefer-dynamic would become significant for building demo-simple-dylib.

Once we have established that the simple-dylib crate is only available as a dylib, then it does not matter whether we pass -Cprefer-dynamic or not when building demo-simple-dylib: if we leave it off, then all that happens is the compiler will first explore trying to statically link all of the dependencies. Once it determines it cannot (due to simple-dylib), it will go back to trying to use dynamic libraries for as much as possible, and thus it will resolve both simple-dylib and std to dynamic libraries.

Next, we consider the way that demo-simple-dylib needs to be invoked:

1

% LD_LIBRARY_PATH=out/pd/sd:$(rustc --print=sysroot)/lib out/bins/ps/dsd/demo-simple-dylib

The main point of interest here is that we had to add some entries to the LD_LIBRARY_PATH: when the program runs, it needs to satisfy its upstream dylib dependencies, which we just finished establishing are simple-dylib (in out/pd/sd) and std (which we map to a directory by asking rustc itself where those support files all live by running rustc --print=sysroot).

Rust libraries: rlib

Phew, that was exhausting.

Lets try to go through an easier case next: the Rust library type, rlib.

This is not necessarily a simple case, but it is an obvious one to deal with, because we’ve been discussing it this whole time; we just did not say that we were.

Specifically: the demo-simple-lib example covered Rust’s lib crate type, which is specified as a compiler-defined choice from one of the flavors of libraries that Rust supports. At the time of this blog post, Rust’s lib crate type maps to rlib, at least on my machine.

So, the usage patterns and issues we described up above for lib all apply to rlib.

For completeness, here is a diagram showing how rlib is used; it will look very familiar.

Likewise, here is the source code for the files listed in the diagram.

1
2
3

// simple-rlib.rs
#![crate_type="rlib"]
pub fn main() { println!("Running main from {}", file!()); }

1
2
3

// demo-simple-rlib.rs
extern crate simple_rlib;
fn main() { simple_rlib::main(); }

Finally, the command invocations that implement the diagram above.

1
2
3
4
5
6
7
8
9

% rustc --out-dir out/ps/sr simple-rlib.rs
% ls out/ps/sr
libsimple_rlib.rlib
% rustc --out-dir out/bins/ps/dsr demo-simple-rlib.rs -Lout/ps/sr
% ls out/bins/ps/dsr
demo-simple-rlib
% out/bins/ps/dsr/demo-simple-rlib
Running main from simple-rlib.rs
%

But, there are no surprises here.

Static libraries for non Rust code: staticlib

If you want to call Rust from another language, you can do that by compiling your Rust code into a static library, and then linking to that static library from your foreign program.

Now, for purposes of demonstration in this example, I am using Rust to implement demo-simple-staticlib; but, crucially, I didn’t have to. It could have been written in C, or I could have used Java and JNI to interface with it. (Or simple-staticlib could have been a Python extension, et cetera.)

Here’s what our linkage picture looks like for staticlib:

If we compare this against our picture for rlib, the main difference now is that there is no longer an arc from rustc to libsimple_staticlib.a. This is actually a pretty big difference!

1. The generated archive file, libsimple_staticlib.a, is not a Rust crate. It does not have the metadata the Rust compiler would need to intepret it as a crate. Instead, it is just another library archive, like the others typically used in a C project.

2. When compiling demo-simple-staticlib.rs, we will have to pass flags to the compiler that tell it to link to the static library. The compiler will no longer magically figure this out for us.

3. Since libsimple_staticlib.a is not a Rust crate, we have to provide explicit declarations in demo-simple-staticlib.rs for the functions it provides.

The differences above can arguably be summed up in: It is like you are programming in C, in terms of having to deal with keeping function signatures consistent and juggling linker flags. If you have experience with that, none of this should seem surprising.

That said, here is the source code for the files in the diagram.

1
2
3
4

// simple-staticlib.rs
#![crate_type="staticlib"]
#[no_mangle]
pub extern "C" fn staticlib_main() { println!("Running staticlib_main from {}", file!()); }

1
2
3
4

fn main() {
    extern "C" { fn staticlib_main(); }
    unsafe { staticlib_main(); }
}

And here are the command invocations that complete the diagram above.

1
2
3
4
5
6
7
8
9

% rustc --out-dir out/ps/ss simple-staticlib.rs
% ls out/ps/ss
libsimple_staticlib.a
% rustc --out-dir out/bins/ps/dss demo-simple-staticlib.rs -Lout/ps/ss -lsimple_staticlib
% ls out/bins/ps/dss
demo-simple-staticlib
% out/bins/ps/dss/demo-simple-staticlib
Running staticlib_main from simple-staticlib.rs
%

Dynamic libraries for non Rust code: cdylib

Conceptually so far we have covered three cells in the following 2x2 matrix, and cdylib will finish the table.

The diagram for using cdylib should have elements that remind you of both the table for dylib and the table for staticlib.

Namely, here we see:

1. Much like demo-simple-staticlib, compiling demo-simple-cdylib.rs does not attempt to extract metadata from the simple-cdylib.so or even treat it as a Rust crate; instead, one must provide the right linker flags to the compiler, and the right extern function signatures in the source code for demo-simple-cdylib.rs.

2. Much like demo-simple-dylib, the execution of demo-simple-cdylib will itself load the shared library demo-simple-cdylib.so and link to its code dynamically.

The source code for this demostration is just like that of staticlib.

1
2
3
4
5
6

// simple-cdylib.rs
#![crate_type="cdylib"]
#[no_mangle]
pub extern "C" fn cdylib_main() {
    println!("Running cdylib_main from {}", file!());
}

1
2
3
4
5

// demo-simple-cdylib.rs
fn main() {
    extern "C" { fn cdylib_main(); }
    unsafe { cdylib_main(); }
}

The command sequence here is interesting: we are no longer forced to use -C prefer-dynamic.

1
2
3
4
5
6
7
8

% rustc --out-dir out/ps/sc simple-cdylib.rs
% ls out/ps/sc
libsimple_cdylib.so
% rustc --out-dir out/bins/ps/dsc demo-simple-cdylib.rs -Lout/ps/sc -lsimple_cdylib
% ls out/bins/ps/dsc
demo-simple-cdylib
% LD_LIBRARY_PATH=out/ps/sc out/bins/ps/dsc/demo-simple-cdylib
Running cdylib_main from simple-cdylib.rs

⊕ This hypothesis is one of many items that I want to follow up on in a future post. I believe this is because we end up treating the two components (simple-cdylib.so and demo-simple-dylib) as completely divorced entities: thus, they each get their own copy of the functions they use from the Rust standard library statically linked into them.

Conclusion

That was quite a romp!

And yet, we have not even gotten to some of the hairier stuff, like:

• mixing distinct upstream crate types into the same project,
• having multiple crate-types for the same crate available,
• using --extern to specify which crate type should be used for a given crate type,
• mixing crates built with and without -Cprefer-dynamic (how to get it to work today, and how should it work in ideal world?), or
• varying whether the program will statically or dynamically link to the platforms C runtime.

Furthermore, I did not really dig into what static linking means, especially when it comes to crate types like rlib, which explicitly do not have link-time dependencies. I want to elaborate on that too, preferably by demonstrating how to use objdump to learn things about the generated code.

So, lots of material for future posts.

(And also, lots of opportunities to try to clean up the presentation of this post.)

Appendix: Who is using that library?

Consider this diagram:

Here is an important question you should ask yourself: is libsimple_lib.rlib actually used by the linker? Or is it solely used as input to rustc (i.e., potentially used in the generation of demo-simple-lib.obj itself). Or is it used by both rustc and the linker?

We can test this question directly, with some slight tweaks to our commands.

Proving the link step’s dependence on the library

First, we can test whether its used by the linker at all by separating the link invocation from the rest of the compiler steps, and then modifying it and running it on its own.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

% mkdir -p out/sl out/bins/ps/dsl2step
% rustc --out-dir out/ps/sl simple-lib.rs
% ls out/ps/sl
libsimple_lib.rlib
% LINKER_COMMAND=$(rustc --out-dir out/bins/ps/dsl2step demo-simple-lib.rs -Lout/ps/sl -Csave-temps --print=link-args -Ccodegen-units=1)
% ls out/bins/ps/dsl2step
demo-simple-lib                                                demo-simple-lib.demo_simple_lib.66aa33f3-cgu.0.rcgu.o
demo-simple-lib.demo_simple_lib.66aa33f3-cgu.0.rcgu.bc         demo-simple-lib.hlsxcd1s0pxo20a.rcgu.bc
demo-simple-lib.demo_simple_lib.66aa33f3-cgu.0.rcgu.no-opt.bc  demo-simple-lib.hlsxcd1s0pxo20a.rcgu.o
% ./out/bins/ps/dsl2step/demo-simple-lib
Running main from simple-lib.rs
% rm ./out/bins/ps/dsl2step/demo-simple-lib
% eval $LINKER_COMMAND
% ./out/bins/ps/dsl2step/demo-simple-lib
Running main from simple-lib.rs
%

The significance of the above sequence: It runs rustc -Csave-temps --print=link-args which will preserve the generated object files and also print out the linker invocation it runs.

⊕ In practice when doing these kinds of experiments, you should not blindly use eval in the manner I have shown here, but instead echo the linker command to the screen and confirm that it is something you trust running. We can run the generated binary, delete the binary, and then re-run that linker command ourselves (eval $LINKER_COMMAND), and re-run the binary again. This confirms that this linker invocation does indeed generate that binary.

With that in place, one way we could exercise the linker’s use of the file: we could just delete libsimple_lib.rlib, and run the original linker invocation:

1
2
3
4
5
6

% eval $LINKER_COMMAND
% rm out/ps/sl/libsimple_lib.rlib
% ls out/ps/sl/
% eval $LINKER_COMMAND
/usr/bin/ld: cannot find /media/pnkfelix/Rust/Linking/out/ps/sl/libsimple_lib.rlib: No such file or directory
collect2: error: ld returned 1 exit status

Another slightly more complicated way we could expose the dependence of our object code on that file: We can remove the reference to libsimple_lib.rlib from the linker invocation, and run the resulting new linker invocation:

1
2
3
4
5

% NEW_COMMAND=$(echo "$LINKER_COMMAND" | sed -e 's@"-Wl,-Bstatic" .*/libsimple_lib.rlib"@@')
% eval $NEW_COMMAND
/usr/bin/ld: out/bins/ps/dsl2step/demo-simple-lib.demo_simple_lib.66aa33f3-cgu.0.rcgu.o: in function `demo_simple_lib::main':
demo_simple_lib.66aa33f3-cgu.0:(.text._ZN15demo_simple_lib4main17h9794b393d760d697E+0x3): undefined reference to `simple_lib::main'
collect2: error: ld returned 1 exit status

This gives you an idea of the kinds of nasty error messages you have to deal with when you start playing games with your build artifacts: The linker is rightfully complaining that the generated object code for demo_simple_lib::main has some reference to simple_lib::main (indeed,the very definition of the former is just a single invocation of the latter), and yet that reference cannot be satisfied. (It is up to the user to read that message and infer that the core problem is that the linker is no longer receiving the path to libsimple_lib.rlib as one of its command line arguments.)

Proving the compiler’s dependence on the library

The previous section established that the linker needs the .rlib file in place. But, maybe that’s only necessary for the link step alone, and rustc itself doesn’t need the actual file?

We can test this theory too, in a similar form of direct experimentation on the build artifacts.

First, lets try removing the file (same as illustrated in the previous section)

1
2
3
4
5
6
7
8
9
10
11
12
13

% rm -f out/ps/sl/libsimple_lib.rlib
% ls out/ps/sl
% rustc --out-dir out/bins/ps/dsl demo-simple-lib.rs -Lout/ps/sl
error[E0463]: can't find crate for `simple_lib`
 --> demo-simple-lib.rs:1:1
  |
1 | extern crate simple_lib;
  | ^^^^^^^^^^^^^^^^^^^^^^^^ can't find crate

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
%

This shows that the compiler is definitely using the file libsimple_lib.rlib as part of its compilation of demo-simple-lib.rs.

If we try to force the compiler to make forward progress by giving it a dummy file, it will still complain:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

% touch out/ps/sl/libsimple_lib.rlib
% ls -s out/ps/sl/
total 0
0 libsimple_lib.rlib
% rustc --out-dir out/bins/ps/dsl demo-simple-lib.rs -Lout/ps/sl
error[E0786]: found invalid metadata files for crate `simple_lib`
 --> demo-simple-lib.rs:1:1
  |
1 | extern crate simple_lib;
  | ^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = note: failed to mmap file '/media/pnkfelix/Rust/Linking/out/ps/sl/libsimple_lib.rlib': memory map must have a non-zero length

error: aborting due to previous error

For more information about this error, try `rustc --explain E0786`.

Upon reflection, this all makes perfect sense: As part of compiling demo-simple-lib.rs, the compiler is going to reference external metadata (i.e. the function signatures and type definitions from the simple-lib crate).

⊕ This quote is attributed to Theodore Levitt, but there is significant evidence that the adage predated his use of it.

People don’t want to buy a quarter-inch drill, they want a quarter-inch hole.

It is a good line.

⊕ Some people I talk to make the leap from safety to another property: security. The two topics are related, since safety issues can sometimes be exploited and thus yield security issues. But you can theoretically achieve security atop an unsafe language; and safety alone does not ensure security. So it is a messy relationship at best.

The adage came to my mind recently in a discussion of Rust’s selling points. An oft-cited selling point, at least for a Programming Language enthusiast like myself, is that Rust offers safety.

I was not familiar with the adage before last year. I first heard it used as the punchline to a significantly longer parable, set at the keynote for a sales convention for a drill company. The lesson of the adage is that to be customer-focused, you must keep in mind that drills are only a means to an end for your customer.

Safety is important: it eliminates a significant class of bugs that plague software written in low-level systems languages.

But, thinking of the drill/hole adage, I stopped and asked myself: “Who wants safety? Is it an end in itself? If it is merely a means to an end, then to what end?”

At this point, while reflecting on that topic, I decided to look more into the origins of the drill/hole adage; that diversion led me to a lovely post critiquing the adage, saying it “doesn’t go far enough.” The heart of the argument there is that people don’t want holes either. The hole is itself another means to an end, such as hooks or shelving. But do people want hooks and shelving? No, they want to store objects on the wall. Why? ⊕ A shelf of objects can also serve as decor; my favorite walls are filled with books. Because they want room to store more things, or they want to be able to fetch and replace things more efficiently, yielding more time.

⊕ The post includes the caveat that intermediate results like the hole are not irrelevant. Its just important to keep the end-goal somewhere in your mind, if only in the back of your mind, as you identify your more immediate (and usually intermediate) goals.

That is what people want: Extra capacity for “stuff”, or to reduce the time they spend searching through clutter.

That is such an important point, and I think it provides the right perspective that we will need to answer the question of “What is Rust’s hole purpose?” (Or, if you prefer a pun-free version: “What is Rust’s value proposition?”)

The Promise(s) of Rust

⊕ I am using “sub-” and “super-” here in a mathematical sense, as in “every safe Rust program is also an Unsafe Rust program, but there are some Unsafe Rust programs that are not part of safe Rust.” There is no value judgement being made as to one being “superior” to the other.

Rust provides two distinct languages: the safe Rust (sub)language, and the Unsafe Rust (super)language. You can read the Rustonomicon for more discussion of this distinction.

An over-simplified way of describing the benefits of the safe Rust language is this “promise”: You will get predictable behavior from your program. If you write a program in safe Rust, it will not surprise you.

⊕ After all, if a program were incapable of surprising anyone, then would it be worth executing in the first place? But, that promise is a lie: There’s plenty of ways to observe unpredictable outcomes, in most any programming language of interest. (Consider for example psuedorandom number generation.)

Here is a somewhat improved promise: You will never get Undefined Behavior from a safe Rust program. It won’t ever access memory after its been freed, and it won’t ever have two threads racing to read and write the same location in memory.

But in fact, this is still somewhat of a lie!

For one thing, we allow crates written in safe Rust to link to crates that are written in Unsafe Rust. Should that be considered an instance of a “safe Rust” program? If it is, then its easy to see a counter-example to the improved promise: just have safe crate that calls out to a helper function oops in an unsafe crate, where oops demonstrates undefined behavior.

Even if you say “no, I do not consider such a program to be an instance of safe Rust. You need all of your crates to be written in safe Rust to make me happy”, you will still run afoul of problems. (1.) Rust’s standard library is using unsafe code that might not always maintain global safety invariants. Even worse, (2.) the Rust compiler itself may generate incorrect output.

So, your program might still exhibit Undefined Behavior even if you restrict yourself to crate graphs where all crates are written in safe Rust.

Such an outcome is not what anyone desires; but it is a reality that we have to deal with, at least with the state of the art today.

⊕ This title implicitly references the work of Findler and Felleisen (2002) on using contracts to assign proper blame when one is dealing with higher-order functions.

Blame Assignment

Here is my favorite statement of Rust’s promise: Undefined Behavior is never a bug in your (safe) code.

Now this is a promise I can get behind.

This is important, because it clearly delineates the “Trusted Computing Base” for your program.

⊕ In other words, The TCB is TCB: The Trusted Computing Base is Taking Care of Business.

In safe Rust, you are trusting the compiler and standard library to get the safety conditions right. You expect that any crates you link to will properly ensure any preconditions necessary for their unsafe blocks, if any.

Why does this matter: Compare against other environments, like C projects, where people can debate for ages about whether a given example is violating the rules of an API or of the language itself, and that leads to issues lying unaddressed, because its unclear who is responsible for the issue.

In Rust, this occurs far less frequently. If someone finds unsound behavior via a program example that has no occurrences of unsafe { ... }, then the Rust community tends to immediately say “that must be a bug!”, and the only task then is to identify whether it is from some unsafe { ... } code elsewhere, or if the Rust compiler itself has an issue.

Of course, Rust is not alone in enjoying this property. Any safe language worth its salt can make the same statement. But Rust is different, in that we make it really easy for people who want to drop into the Unsafe Rust superlanguage to do so. ⊕ Again, no value judgement is being made here. Really! Safe and unsafe Rust are mathematically incomparable when it comes to inherent value! That lets developers get their work done faster, because they can hyper-optimize their generated machine code. Or they can easily call out to a foreign library and avoid implementing a tricky algorithm in the first place!

I claim: Compared to developers using unsafe languages, safe Rust developers spend less time debating about who is responsible when soundness issues arise. Compared to developers using safe languages, Unsafe Rust developers spend less time figuring out how to deliver a performant solution.

The end value provided by Rust to its developers (and thus to their customers and employers) is less time arguing linguistic minutia and less time wrestling with a managed language environment, and more time focused on what actually matters to each of those developers (be it business logic, or family time).

But, notice that I used two distinct categories for the subjects of the two claims. Am I comfortable strengthening my claim? Can I just say “Rust developers spend less time debating about who is responsible when issues arise, and they spend less time figuring out how to get a performant solution into shape”?

The Holes in the Argument

I would like to make that stronger claim. I think we have some evidence to support it. But we are not all the way there yet.

From safe to unsafe

To me, the biggest open problem is: How does one make the transition from safe Rust developer to unsafe Rust developer? How do you prove to yourself that your unsafe { ... } code is not introducing an soundness issue?

We have some ongoing work in this space, but it is not a solved problem.

How to stay safe

The other obvious open problem is: When is safe Rust not performant enough, and why?

There are some inherent overheads that safe Rust is simply going to pay (e.g. bounds checking on array accesses). Some of those cases can be side-stepped in many cases by better understanding of what the language offers (e.g. iterating rather than indexing will skip the bounds checks), which is arguably an instance of “wrestling with your environment” to reap more performance.

Then there are overheads that are incidental rather than inherent. The Rust developers have tried to leave many doors open for future improvements to our output code quality. (Some of these cases overlap with our need for better ways to catch bugs in unsafe code. Others are a matter of “just” putting in the work.)

Conclusion

If you are interested in helping solve any of these problems, please reach out!

Last year, I gave a talk at QCon that promoted the use of rr as a tool for exploring Rust code. I wanted to convey an excitement about debugging, about using this tool to dive into the weeds of your program’s behavior.

Since then, I have continued to talk to developers, promoting rr (and the associated service pernos.co). And I heard Rust developers say that they are happy with their experience using instrumentation such as logging statements or the tracing infrastructure.

I have to agree: Logging statements are useful, and sometimes they are all you need. Here is a great quote from “The Practice of Programming”, by Kernighan and Pike:

As personal choice, we tend not to use debuggers beyond getting a stack trace or the value of a variable or two. One reason is that it is easy to get lost in details of complicated data structures and control flow; we find stepping through a program less productive than thinking harder and adding output statements and self-checking code at critical places. Clicking over statements takes longer than scanning the output of judiciously-placed displays. It takes less time to decide where to put print statements than to single-step to the critical section of code, even assuming we know where that is. More important, debugging statements stay with the program; debugging sessions are transient.

This quote is apt for many reasons; I will revisit it again before the end of this post.

I do not know the typical way that people use debuggers. But I can imagine it looks much like the image depicted above: you start your program, you use a single-step command to go through it line by line, and you examine raw memory dumps to try to figure out what the machine is doing. After that initial exploration, you might set breakpoints at specific lines of code, and then tell the debugger to continue running until it hits any of those breakpoints.

⊕ Advanced debugger users will set conditional breakpoints, which only halt the program if a specific predicate holds when it hits the breakpoint, or even scripted breakpoints, which will automatically execute a series of debugger commands each time they hit the breakpoint. One neat trick is that you often can end a script with continue, so that it seems as though the program never stopped at all, except for the side-effects on the debugger itself (such as setting other breakpoints).

But that typical usage pattern doesn’t really provide much value over just adding log statements. If all you want to do is learn what steps the computer took, and what data was in certain variables at those steps, printing to a log is a better way to achieve that.

So, why am I advocating for more people to include a debugger as one of the tools to keep ready on their Rust development utility belt?

There are four main features that a debugger gives me that are not always fulfilled by adding log statements:

• object code inspection,
• raw control-flow,
• memory exploration, and
• hardware watchpoints.

Of the four, i think hardware watchpoints are the best example of a capability that is both broadly useful and not typically available outside of a debugger. I will delve into each of these in turn below.

Object Code Inspection

By “object code inspection”, I mean that I can attach a debugger to any binary, without having to recompile anything. When your workflow is to add new print statements each time you want to inspect something new, that means editing your code and recompiling it. That edit-recompile-debug cycle has two problems: it can be slow (especially the recompilation step). Worse still, the edits can cause bugs to mysteriously go away, aka the “observer effect”. (To be fair, using a debugger can also cause those Heisenbugs to change behavior. This is where tools like rr can really shine.)

How valuable is “object code inspection”? I have given two reasons to avoid the edit-recompile steps: reduced latency, and reducing the observer effect.

Honestly: It is not a frequent need for my current work. Much of my work involves either fixing bugs that reproduce readily in the face of source code changes, or adding features (which inherently requires source code changes).

⊕ However, in practice this can be very painful: We do not have much support for building an unoptimized rustc (it has a very long bootstrap time), but once you enable optimizations, the debugging experience degrades quite a bit in Rust today.

There is one important exception here: rustc itself has a long bootstrap time; so I have often tried to use a debugger to debug rustc itself, rather than rely on an edit-recompile-debug cycle.

Raw Control-Flow

By “raw control-flow”, I mean that one can step through the individual assembly code instructions to see exactly what the machine is executing.

For example, at a method invocation, one can step forward and discover how exactly to which function that invocation dispatches as it goes through Deref implementations until it finally reaches the target method.

As another example, one can step through the the instructions corresponding to the subcomponents of a match pattern. This way, one might discover which parts matched and which failed to match, and in what order they were evaluated.

Here is a concrete example of the latter:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

struct P { a: char, b: char, c: &'static str }

fn main() {
    let p = P { a: 'h', b: 'i', c: "world" };
    foo(p);
}

fn foo(p: P) {
    match p {
        P { b: 'i', a: 'y', c } => println!("yi, {}", c),
        P { a: 'h', b, c } => println!("h{}, {}", b, c),
        _ => panic!("unmatched"),
    }
}

After compiling the above with debuginfo enabled and running it under gdb, I can investigate the body of foo while its running:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

(gdb) step
boom::foo (p=...) at boom.rs:9
9     match p {
(gdb) disassemble
Dump of assembler code for function _ZN4boom3foo17h3035bcff6328b12bE:
   0x000055555555db70 <+0>:   sub    $0x128,%rsp
   0x000055555555db77 <+7>:   mov    %rdi,0x38(%rsp)
=> 0x000055555555db7c <+12>:    cmpl   $0x69,0x14(%rdi)
   0x000055555555db80 <+16>:  jne    0x55555555db8d <_ZN4boom3foo17h3035bcff6328b12bE+29>
   0x000055555555db82 <+18>:  mov    0x38(%rsp),%rax
   0x000055555555db87 <+23>:  cmpl   $0x79,0x10(%rax)
   0x000055555555db8b <+27>:  je     0x55555555db9d <_ZN4boom3foo17h3035bcff6328b12bE+45>
   0x000055555555db8d <+29>:  mov    0x38(%rsp),%rax
   0x000055555555db92 <+34>:  cmpl   $0x68,0x10(%rax)
   0x000055555555db96 <+38>:  je     0x55555555dbeb <_ZN4boom3foo17h3035bcff6328b12bE+123>
   0x000055555555db98 <+40>:  jmp    0x55555555dc6e <_ZN4boom3foo17h3035bcff6328b12bE+254>

In the above, we can see three comparison instructions: a comparison of 0x14(%rdi) with $0x69 (the character ‘i’), then a comparison of 0x10(%rax) with $0x79 (the character ‘y’). If they’re both equal, then it jumps to 0x55555555db9d (which is presumably the first arm’s body). If they’re not, then it does a comparison of 0x10(%rax) with $0x68 (the character ‘h’), and if that is equal, then it jumps to 0x55555555dbeb (which is preumably the second arm’s body).

⊕ The people I imagine doing this kind of investigation are the ones working out the low-level semantic guarantees when it comes to investigating things like rust#87520. Pretty low-level details, admittedly. Someone new to Rust cannot be expected to do this kind of investigation to gain insight into the language.

In short, this seems like a case where Kernighan and Pike’s quote above is especially apt: we can easily get lost in the details of the control flow as rendered here.

So, how valuable is “raw control-flow”?

Honestly: The more experience I gain in a language or a particular project, the less and less I find myself relying on this kind of spulunking. The evaluation order of match shouldn’t matter in most cases, so I would not optimize the debugging UX for that. Seeing how method dispatch resolves is useful, but in practice logging statements provide enough insight that people rarely find themselves wanting to step through the individual assembly instructions.

Insight into the raw control-flow is probably most useful for someone peeking into low-level details of the semantics of the language (or a given project, especially a code base with tricky macros that generate control-flow structures). One might hope that that group generalizes to beginners who are learning the language. But I am not certain that raw control-flow will efficiently provide practical insight. For example, consider the following:

1
2
3
4
5
6
7
8
9
10
11
12
13

#[derive(PartialEq, Eq, PartialOrd, Ord, Debug)]
struct D(&'static str);

fn main() {
    let d = D("my name is D");
    println!("{:?}", d);
}

impl Drop for D {
    fn drop(&mut self) {
        println!("Goodbye, {}", self.0);
    }
}

A newcomer to Rust who is not familiar with the destructor system might think that the above code only prints a single line of text. Running the code (even outside of a debugger) will show that to be incorrect. But: Will stepping through it in a debugger provide the insight about where the <D as Drop>::drop call comes from? It is not clear to me that it would.

Memory Exploration

By “memory exploration”, I mean that when my debugger is attached to a paused program, I can inspect values on the stack or heap, walking down the nodes of a binary tree or the entries in a hashtable. A good debugger will do some amount of memory traversal and semantic interpretation on my behalf, using type information from the program’s associated debuginfo to determine how best to interpret the bits it finds and print it nicely (e.g. showing comma-separated array elements).

How valuable is “memory exploration”?

Honestly: In Rust, one can add #[derive(Debug)] to type definitions to get methods that print the structure for their values. The debugger printouts were useful when the language didn’t provide it out of the box (like in C and C++), but they less value in Rust.

⊕ Another counter-argument to this supposed first benefit: In a language like Rust, anyone can use unsafe code to transmute the value to raw bits, and subsequently print them. This power is not solely in the debugger’s hands. One benefit of using a debugger here, rather than just using log statements, is that a debugger will let you immediately see the concrete representation (such as concrete hashtable layout) in places where the developer has chosen a more abstract view for the Debug trait implementation. But again we hear echos of the Kernighan and Pike quote: Seeing that concrete representation may be crucial in some scenarios, but in many cases it will just be unnecessary detail distracting you from the problem at hand.

A second benefit with respect to memory exploration is that a debugger will also let you inspect values that were not part of the original log statement. But, one can view that as a special case of bypassing the edit-recompile cycle, which we already discussed above. If the edit-recompile cycle is short, then there is little cost to adding a new log statement that includes the value of interest.

Hardware Watchpoints

So we have come to perhaps my favorite debugger feature: Hardware watchpoints.

This is the main feature that most programming languages do not provide.

The idea is simple: the gdb command watch -location <place> will break the next time the memory at the given place is modified. For example, watch -location array[i] will evaluate &array[i] to its address, and then the next time array[i] is changed, the debugger will stop and print out both the old value and the new value.

Consider the following (buggy) code for merging two sorted sequences into a single sorted sequence:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

fn main() {
    let mut r = [99, 98, 97, 96, 95];
    merge(&mut r, &[10, 30], &[20, 40, 60]);
    dbg!(r);
}

// Supposedly merges sorted `a` and `b` into  `recv`.
fn merge(recv: &mut [i32], a: &[i32], b: &[i32]) {
    let mut ai = a.iter();
    let mut bi = b.iter();
    let (mut a_cur, mut b_cur) = (ai.next(), bi.next());
    for i in 0..recv.len() {
        match (a_cur, b_cur) {
            (Some(x), Some(y)) => {
                if *x <= *y {
                    recv[i] = *x;
                    a_cur = ai.next();
                } else {
                    recv[i] = *y;
                    a_cur = ai.next();
                }
            }
            (Some(x), None) => {
                recv[i] = *x;
                a_cur = ai.next();
            }
            (None, Some(y)) => {
                recv[i] = *y;
                b_cur = bi.next();
            }
            (None, None) => {
                return;
            }
        }
    }
}

When I run the code above, I get the output:

1
2
3
4
5
6
7

[boom.rs:4] r = [
    10,
    20,
    20,
    40,
    60,
]

which is not quite right! The third element of the generated array is 20, instead of 30.

This toy example is small enough that you might be able to see the bug from immediate inspection. Adding more test cases to the run might also make the problem apparent. But I am going to use it to illustrate how to use a hardware watchpoint to jump immediately to the point in the control flow where the third element of the recv array is overwritten.

After starting up the debugger, we let it evaluate the line that creates the destination array:

1
2
3
4
5
6

(gdb) start
Temporary breakpoint 1, boom::main () at boom.rs:2
2     let mut r = [99, 98, 97, 96, 95];
(gdb) next
3     merge(&mut r, &[10, 30], &[20, 40, 60]);
(gdb)

Next, lets first check that we understand what we are looking at:

1
2
3
4

(gdb) print r
$1 = [99, 98, 97, 96, 95]
(gdb) print r[2]
$2 = 97

Yep: We have our destination array r, and it currently has 97 as its third element r[2].

So we tell the debugger that we want it to break as soon as that location is changed, and then let the program run!

1
2
3
4
5
6
7
8
9
10
11
12

(gdb) watch -location r[2]
Hardware watchpoint 2: -location r[2]
(gdb) continue
Continuing.

Hardware watchpoint 2: -location r[2]

Old value = 97
New value = 20
boom::merge (recv=&mut [i32](size=5) = {...}, a=&[i32](size=2) = {...}, b=&[i32](size=3) = {...}) at boom.rs:29
29                    b_cur = bi.next();
(gdb)

We are now within the body of fn merge instead of fn main (as you might have expected), immediately after the mutation occured:

1
2
3
4

        (None, Some(y)) => {
            recv[i] = *y;
            b_cur = bi.next();
        }

And indeed, if we print out recv at this point, we can see that the array has been “corrupted”, since we now see the two copies of 20 in the output:

1
2
3

(gdb) p recv
$3 = &mut [i32](size=5) = {10, 20, 20, 96, 95}
(gdb) ```

I am running with the rust-gdb command here, which has extensions for printing slices nicely. If you were doing this under just normal gdb, then you would see something like this instead.

1
2

(gdb) p recv
$3 = &mut [i32] {data_ptr: 0x7fffffffe238, length: 5}

This kind of break-on-write is most useful for unsafe languages, when you have data being corrupted due to out of bounds accesses. One certainly can encode examples of that in unsafe Rust, but we also do not expect them to arise as frequently.

But honestly (a ha): Is this actually revealing anything about the core problem here? The logic at this match-arm is entirely sound: if we have (None, Some(y)) for our two cursors traversing the slices, then we should pass along y and keep traversing down the right cursor. Whatever the bug is, it is arising somewhere else, before we even get here.

Reversible Debugging

If I were restricting my attention to gdb/rust-gdb alone, this would be a point where I would start talking about re-running the program and stepping through it carefully, or trying to hypothesize points of interest based on analyzing the source code and setting breakpoints at those points. But if we were going to do that, then we might as well invest our time adding some of those logging statements discussed above, because I think those would show the problem here just as fast, if not faster!

My goal is to show you an alternative strategy, that works for code bases that you are not that familiar with.

Lets continue looking at the merge example, but let us now run it under rr.

First we record a run of interest:

1
2
3
4
5
6
7
8
9
10

% rr record ./boom
rr: Saving execution to trace directory `/home/pnkfelix/.local/share/rr/boom-2'.
[boom.rs:4] r = [
    10,
    20,
    20,
    40,
    60,
]
%

With that recording in place, we can replay the run under rr, which gives an interface similar to gdb. (And we will tell rr that we want it to use rust-gdb as our debugger to get those extensions for rendering Rust collections.)

1

% rr replay -d ~/.cargo/bin/rust-gdb

Our initial interaction with rr will be much like how we interacted with gdb above: we’ll start the program, set a hardware watchpoint for the third array element, and let it run forward.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

(rr) break main
Breakpoint 1 at 0x55e4f162f437: main. (2 locations)
(rr) c
Continuing.

Breakpoint 1, 0x000055e4f162fae0 in main ()
(rr) next
Single stepping until exit from function main,
which has no line number information.

Breakpoint 1, boom::main () at boom.rs:2
2     let mut r = [99, 98, 97, 96, 95];
(rr)
3     merge(&mut r, &[10, 30], &[20, 40, 60]);
(rr) watch -location r[2]
Hardware watchpoint 2: -location r[2]
(rr) c
Continuing.

Hardware watchpoint 2: -location r[2]

Old value = 97
New value = 20
boom::merge (recv=&mut [i32](size=5) = {...}, a=&[i32](size=2) = {...}, b=&[i32](size=3) = {...}) at boom.rs:29
29                    b_cur = bi.next();
(rr)

But now that we have reached this point in replay, we can rewind time. We can ask questions like “how did a_cur (the left-hand side of the tuple) become None already, without us ever emitting 30 to the recv array? Well, the question doesn’t quite look like that. But we can ask "When was a_cur last written to before this point?”, which ends up effectively the same question.

1
2
3

(rr) watch a_cur
Hardware watchpoint 3: a_cur
(rr)

Now we tell the system to run backwards.

1
2
3
4
5
6
7
8
9
10

(rr) reverse-continue
Continuing.

Hardware watchpoint 2: -location r[2]

Old value = 20
New value = 97
0x000055e4f162fa95 in boom::merge (recv=&mut [i32](size=5) = {...}, a=&[i32](size=2) = {...}, b=&[i32](size=3) = {...}) at boom.rs:28
28                    recv[i] = *y;
(rr)

⊕ The gdb output is potentially confusing because the notions of “Old” and “New” are inverted here. To dive deeper into this topic, you might consider the watching the films Primer or TENET.

We still have our first watch point in place for r[2], so it halted almost immediately. (When we were running forward, a watchpoint makes us stop right after the write occurs. When we run backwards, a watchpoint makes us stop right before the write occurs.)

Let us keep going backwards.

1
2
3
4
5
6
7
8
9
10

(rr) reverse-continue
Continuing.

Hardware watchpoint 3: a_cur

Old value = core::option::Option<&i32>::None
New value = core::option::Option<&i32>::Some(0x55e4f1664004)
boom::merge (recv=&mut [i32](size=5) = {...}, a=&[i32](size=2) = {...}, b=&[i32](size=3) = {...}) at boom.rs:20
20                        a_cur = ai.next();
(rr)

And now we see where a_cur is being assigned None: line 20:

1
2
3
4
5
6
7
8
9
10
11
12

    match (a_cur, b_cur) {
        (Some(x), Some(y)) => {
            if *x <= *y {
                recv[i] = *x;
                a_cur = ai.next();
            } else {
                recv[i] = *y;
                a_cur = ai.next(); // here's the assignment
            }
        }
        ...
    }

And if you look at this, you might see the problem: in that arm of the if, we are copying over the right-hand tuple value (*y, from b_cur), but we are advancing the left-hand cursor, a_cur.

With that insight, we can make an easy fix: replace line 20 with

1

                b_cur = bi.next(); // was `a_cur = ai.next();`

With that change in place, the code works:

1
2
3
4
5
6
7

[boom.rs:4] r = [
    10,
    20,
    30,
    40,
    60,
]

⊕ This is a case where the ability to explore memory and add printouts from the debugger is crucial: Because now we are exploring the behavior of one specific trace, and we do not always have the opportunity to try to recreate the trace on a new version of the program.

There are other advantages provided by rr. For example, record/replay means that you can investigate a trace repeatedly. Each time you replay the trace, you’ll observe the same execution; and you can add breakpoints (and print statements on those breakpoints) on that exact trace as it replays. This is invaluable for tracking down bugs that are intermittent or that depend on subtle event orderings.

Closing Thoughts

I started writing this post thinking “I’m excited about debuggers! Why is no one in the Rust community using them?” As I dug into the topic, I kept asking myself “what feature is provided here that people cannot more readily get via their logging instrumentation? What value am I getting from this?”

As you might infer from my “Honestly” notes above, I had to admit that the advantages of a debugger when compared to logging instrumentation may be limited in scope. At least, that’s how I see things with typical debugging workflows.

Tools like rr open up new debugging workflows, one of which I delved into above. I plan to write more in the future about other such new workflows, such as ones provided by services like pernos.co. More generally, I want to spend some time thinking about the nature of debugging: how it links in with how we develop features, how we communicate our ideas, and how we validate our models.

Now I want to show you the design document I developed, which digs more into how I thinkI cannot emphasize enough that nothing here is set in stone. These are ideas I am circulating with you precisely so that you can tell me what needs to change. the system should be architected.

I will warn you up-front: This is a long post. Eight pages printed in my PDF preview; ten if you include the appendices. And yet it is largely just a transcription of another doc. But the thing it is transcribing is going to evolve with time, and I like the idea of taking a snapshot of its state today. I also think the ideas in it are fun. A lot of it still remains to be designed. Much in here falls into the category of software that you can produce with a ton of dedicated programming on every level of the software stack; I want to figure out how to avoid that.

My most important unresolved question is, how can we deliver all of this functionality with minimal investment on the part of our end developer who is themselves a newcomer to Async Rust programming.

Everything that follows was taken verbatimOkay, not verbatim. I am again spicing things up a little by interleaving another favorie song alongside the text of the document, at the risk of turning this post into a Mark Z. Danielewski novel. from the document I presented to the other members of the AWS Rust team. This document is an attemptThere was an earlier iteration of this document, with a slightly different design. I am not going to write a blog post about the earlier document (I figure it is better to jump straight to the superior version), but it is openly available, just like the document I have transcribed into this blog post. to describe a software architecture for the first, most important tool: An “Async Monitor” that models the behavior of one’s Rust async runtime, alongside an “Async Console” that presents slices of information from that model. As with parts one and two, I will be adding side-commentary on the right-hand margin.

TurboWish Development Plan: The Async Monitor and Console

⊕ (This time I did not repeat my “mistake” of crowd-sourcing the choice of source material. If I were a proper magician, I would have employed a force, but I had a bit too much confidence in my ability to find appropriate quotes, even from ABBA for goodness sakes!)

Overhead the albatross

Hangs motionless upon the air

And deep beneath the rolling waves

In labyrinths of coral caves

TurboWish Overview

TurboWish is a suite of tools that give Rust developers insight into performance issues“Performance issues” raises questions immediately. What does “issues” mean? Is this a profiler? A debugger? My current answer: I had not appreciated this until recently, but I think there is a spectrum between profiling and debugging. After all, they both come down to understanding the behavior a program. Providing insight into that is what TurboWish is all about. in their code.

The first TurboWish deliverable is the Async Monitor and Console, which answers a developer’s questions about how their code’s async runtimeI had been using the word “async executor” here, but Carl pointed out to me that the executor is just one small piece of the overall framework provided by a typical async environment. So now it says “runtime.” I am curious what other terms might work here as well. is behaving as their program runs.

The Async Console provides a summary of an async runtime’s behavior, using concepts shared across the async rust ecosystem (such as tasks and resources), and metrics that are applicable to any async program (such as the average time each task spends waiting in a ready state).

When a developer asks: “Why is my task stuck?” or “Why is my app slow?”, the Async Console is the first, and in some cases, the only tool they need to reach for. It will allow the developer to quickly see how tasks are scheduled (to learn how much time is spent in the developer’s own code versus waiting to run), identify tasks that are starving or blocked and what resources they are waiting on, and identify tasks responsible for replenishing a scarce resource.

We plan to expand the TurboWish tool suite with other toolsYou may have wondered when reading the user stories in part two how a single tool was going to satisfy all those different narratives. The phrase “other tools” here hopefully answers that questions: currently, I am not planning to have a single tool do all of the stuff described in those stores. dedicated to other investigations, such heap profiling or sampling-based CPU profiling. This design document is dedicated to the Async Monitor and Console; in tandem, they are the tool that will drive the top-down investigation of an async developer’s performance issues.

The echo of a distant time

Comes willowing across the sand

And everything is green and submarine

Document overview

This document describes the development plan for the first TurboWish tool: the Async Monitor and Console. It opens with the original goals followed by (proposed, unfinalized) tenets for the TurboWish suite itself. It then presents a description of the customer experience using the Async Monitor and Console. Then it sketches an implementation plan. A section follows describing metrics for evaluating whether the tools are achieving their goal, and finally a section touching on security concerns. Appendices follow the conclusion of the document, including an appendix with the project schedule.

And no one showed us to the land

And no one knows the where’s or why’s

But something stirs and something tries

And starts to climb toward the light

Goals

Profile Production Code: Incorporating the TurboWish Framework is low-overhead: it can be incorporated into production codeYou might ask: Does “production code” mean the same thing as “deployed code running live”? And you would be right to ask that. In hindsight, I should have included an entry about this in the “Minimum Viable Product” section where I described long-term concerns versus short-term sacrifices. My attitude here, in any case, is that there are other infrastructures for logging behavior of live services. Integration with those infrastructures might make sense eventually, but I do not regard it as a pre-requisite for launching this product. without producing an undue maintenance burden and without incurring significant performance overhead.

Domain-specific Feedback: Frameworks and applications can provide data for specialized metrics, specific to their internal architecture.

Understand Hidden Costs and Connections: Frameworks like tokio ease writing asynchronous code because they hide a number of details behind abstractions (such as generator code produced by the Rust compiler, or task queues managed by the tokio runtime). TurboWish exposes those hidden details, allowing developers to correlate them with other program events. It also exposes connections that humans usually have to reconstruct by hand (such as future to resource to future chains that can yield deadlock), allowing one to directly see from Rust’s ownership model how resources are being held in the object graph.

Framework Agnostic: Many of Rust’s customers use tokio, but not all of them. async-std and fuschia_async are other frameworks for asynchronous programming. TurboWish can provide value to any such framework (though it may also provide framework-specific functionality when warranted). For our initial releases, we can focus on tokio alone, but expect integration with others if use with tokio proves successful.

EC2 Instance Type Agnostic: If we make use of any OS specific features (e.g. dtrace probes), they will be available on all EC2 AL2 instances, regardless of instance-type. (Specifically, we cannot require access to CPU performance counters for core functionality. We may offer extra features that utilize them.)

⊕ This use of the word “tenets” is part of Amazon parlance. When you see it, you can think “Guiding Principles”.

Tenets of TurboWish Tool Suite

The Rust community at large, both within and outside of AWS, is the customer for TurboWish.

Injected instrumentation must not block application progress.

Timing-measurement noise induced by client instrumentation should be minimized.

Present diagnoses in terms of customer-centric concepts, such as resource and task.

Minimize coupling between components to encourage concurrent community development.

The transition from development to production deserves as much performance tooling as production monitoring itself. (More specifically: We see customers struggling to get their software operating well enough to push to release; therefore, some tools can be specialized to the development use-case.) The Async Monitor is one such tool: It is aimed at programs under development, not in release.

Strangers passing in the street

By chance, two separate glances meet

And I am you and what I see is me

The Async Console Experience

When a developer wants to deploy the Async Monitor on their program, they will need to hook it in by adding it as an upstream dependency (in the Cargo.toml file) and also writing a few initialization lines at the start of their source code:

1
2
3
4
5
6
7
8

use turbowish_async_monitor as tw_monitor;

#[tokio::main]
async fn main() {
    tw_monitor::builder().port(8080).init();

    // Rest of the app
}

With that initialization code in place, the service will operate in the same fashion as before, but will now also run the Async Monitor, which observes events and then, based on those observations, builds an internal model of the program and the async executor. External programs, such as the Async Console can now connect and present the state of the Async Monitor by connecting to the port indicated above..

When the developer first connects the Async Console, they get a presentation similar to the UNIX top command, showing a brief summary of the executors (with metrics like the number of tasks running or sleeping, average times spent waiting to run, average Future::poll runtimes), and below that, a list of the current tasks, each on its own line with with an id number, name, current run state (Polling, Ready to poll, and Waiting), and a list of task attributes that include developer-specified metadata.

⊕ Mock up of the console UI was provided by Carl Lerche, a lead Tokio developer.

This view will dynamically update (just like top) as the application runs.

The async console presents data primarily oriented around tasks and resources. The aim is for a model as straight-forward to understand as the Resource Allocation GraphsI will be honest: These task/resource graphs are the part of the system I personally am most excited about. I do not yet know how we will deliver on the developer experience that I envisage, but I want it to be as beautiful as the graphs I presented in part two. described by Ric Holt in 1972: We wish to explain as much as we can in terms of tasks waiting for or using resources (either by locking them if they are exclusive resources, or by consuming them if they are cumulative resources such as message channels), or resources waiting for tasks to make them available (either by unlocking them for exclusive resources, or by refueling them if they are cumulative resources).

The async console monitors for known performance pitfalls, and includes a highlighted alert at the top of the screen if the program is currently exhibiting one of those pitfalls. The alert provides a link to a problem-oriented view, that explains the pitfall and provides guidance as to how one can resolve it.

From the async console, the developer can do three main things. First, they can dig into an individual record of a task or resource, by traversing hyperlinks for that task or resource. (A “record” in this context is a screen summarizing the event history and current status for that task or resource.) Second, they can pause a waiting task (or request that a specific non-waiting task be paused when it next evaluates a .await); such paused tasks can be subsequently inspected by a debugger attaching to the running process, and then resumed when the developer is done with the attached debugger. Third, they can rollback history, in order to inspect past states for tasks or resources.

And do I take you by the hand

And lead you through the land

And help me understand the best I can?

Performance Pitfall Alerts

The performance pitfall alerts handle a collection of known cases where our customers are asking themselves: “why is my task stuck?”

Example of problems that the Async Monitor can detect include: deadlock cycles, excessive polling times, and buggy implementation of Future that fail to register a waker.

As a concrete example: When a deadlock cycle occurs, the async console displays an alert saying “deadlock cycle detected.” The developer follows the hyperlink for the alert, and this brings up a “problem view” that shows an interleaving list of tasks and resourcesHere I made a concession to reality: users do not want graph diagrams shoved in their faces all the time. Lists are sometimes an entirely reasonable presenation medium, especially when it comes to describing a deadlock cycle. , corresponding to the chain of dependencies that forms the cycle.

Resource Records

A resource record shows information about an individual resource.

Some resources are exclusive: they can be held by at most N tasks at a time (where N is often 1, such as in the case of a mutex lock). The resource record for an exclusive resource will include a list of tasks that currently hold it. It will also include a separate list of tasks that are currently blocked while trying to acquire the resource.

Some resources are cumulative: they hold some number of resource units (work items in a queue, messages on a channel, et cetera). The resource record for a cumulative resource will include a list of tasks that are currently blocked on the resource’s current state (for example, for a channel, a receiver will block on an empty channel; a sender will block on a bounded channel at full capacity).The fact that a (bounded) channel can cause a sender or receiver to block is an insight that is simultaneously obvious but at the same time raises important modelling questions. In short: the arcs representing blocking dependencies in the task/resource graph may go in either direction, depending on the state of the channel. My mind keeps coming back to this point, and I want to make sure we do not forget it.

(In Holt 1972, the terms “reusable” and “consumable” roughly correspond to our usage of “exclusive” and “cumulative”. The correspondence is imperfect though; e.g. Holt refers to individual consumable resource units, while for us, a “cumulative resource” describes a collection of such consumable units, such as a channel.)

In addition, the resource record will include lists of tasks that have signaled intent to interact with the resource. This way, the resource record for a multi-producer single-consumer channel can include a list of all of the sending tasks that hold the channel, as well as the receiver task associated with the channel, regardless of whether the channel’s message buffer is empty, partially-full, or full.

Listing the tasks that intend to interact with the resource is how we can provide the developer with enough information to resolve unexpected blocking behavior in their code. For example, after seeing that a sender task is blocked because a bounded channel is at full capacity, it is simple for the developer to follow hyperlinks to go from the sender task to the channel’s resource record, and from there through another hyperlink to the receiver task, and then work on figuring out how to make the receiver task process messages more efficiently.

Task Records

If a task is blocked, the task record will show what specific .await expression it is blocked on, by showing the span (file:line and column range) and the expression itself (my_channel.send(value).await). If the blockage is due to a specific resource, then a hyperlink to its resource record will be provided as well.

More generally, a task record shows information about an individual task, including lists of resources associated with the task.

If a task currently holds exclusive access to resources, all such resources will be listed.

As mentioned earlier, tasks can signal intent to interact with a resource. There are two kinds of intention that can be signaled: conditional intent and unconditional intent.I need to more research to see what related work there is here. My naive notion of “conditional” and “unconditional” intent is probably not novel. What I really want to know is whether it is a useful distinction to make.

A task signals conditional intent when the interaction is predicated on some external condition holding; i.e., it will occur only on some of the task’s non-error control-flow paths. (One could imagine approximating conditional intent by just listing all the resources that are reachable from the task; this is a topic we should explore during implementation.)

A task signals unconditional intent as a way to state “if I can make sufficient forward progress, I will interact in this manner with that resource.” It is essentially another way of saying: “if you can figure out how to get me unblocked, these are the actions I will take”, which can be crucial in resolving certain kinds of resource starvation issues.

The task record shows two separate lists corresponding to the two kinds of intention. This way, developers are informed about which tasks to look at first when trying to understand the interactions between tasks and resources in their program.

During execution, resources may move from the conditional list to the unconditional list, or they may move off the conditional list entirely, all according to what a task indicates as its code runs. When a task has actually performed its intended operation to completion (e.g. if it is entirely done sending messages on a given channel, and signals such to the async monitor) then the resource will likewise be removed from the task’s resource lists.

And no one calls us to move on

And no one forces down our eyes

No one speaks and no one tries

No one flies around the sun

Pausing a Task

Sometimes the information provided in the TurboWish Async Monitor’s records will not give enough detail, and the developer will want to inspect the actual state of the task’s memory on the live process in a debugger.

To pause a task, the developer can go to its Task record page, or just select it in the Async Console overview, and then issue the “pause” command. The executor pauses tasks by opting not to schedule them, instead leaving them suspended where they evaluated .await.

As a convenience, the Async Console provides gdb and lldb commands as helpers that pause the task, then spawn the corresponding debugger processes and attach them appropriately to the paused task.

Rolling Back the Event Log

Sometimes humans just don’t understand how they got into some mess.

For those situations, the Async Monitor offers rollback functionality. Developers can step backwardsI have spent a long time advocating for the use of the rr debugger, and I suspect this section is a small reflection of that. (This document is describing a completely different operational model to what rr does, though. I am emphatically not suggesting the event log contain enough state to actually replay program code.) through the event history and see how such steps affect the task and resource records.

For example, a select! expression in tokio will run a collection of futures concurrently. It will proceed with the value provided by whichever future completes first, canceling the other futures that lost the race. If code is not written to anticipate such cancellation, it can lead to data corruption, or tasks appearing to be blocked indefinitely (aka “hung”).

As a concrete example adapted from a blog post, consider this code:

1
2
3
4
5
6
7
8
9
10

let mut file = ...;
let mut channel = ...;
loop {
    futures::select! {
        _ => read_send(&mut file, &mut channel) => {},
        some_data => socket.read_packet() => {
            // ...
        }
    }
}

Here, read_send is constructing a future on every iteration through the loop. If the socket.read_packet() invocation ever wins the select!-race, then the read_send future is dropped, which means the data may have been extracted from the file but not yet relayed on the channel. (The details are spelled out on the blog post.)

If a task appears to be hung, the Async Monitor allows the developer to gain insight into why: They can visit the task view for the blocked task. The task view will indicate where it is stuck (in the select!). Since select! is known to be a source of problems for developers, the view will also include a hyperlink that will “rewind time” to the invocation of select! on the previous iteration of this loop, so that one can observe what side-effects it had. Upon rewinding time, the task view indicates that the socket.read_packet() future won the race, and that the future returned from read_send was dropped.

(At this point, we are relying on the developer having an “Aha” moment about the effect of dropping a future when it is in the middle of its computation. It would be good to explore ways to help people get the insight sooner. For example, maybe having the view here indicate where the read_send future was in its own computation at the time it was dropped would help someone see that we need to keep that same future running.)

Constraining event log overhead

As part of the configuration options for the Async Monitor, one can adjust the upper-bound on the size of the event log, in order to ensure its memory usage does not induce memory exhaustion, virtual memory swapping, or other problems that could excessively perturb performance evaluation.

Concluding the Experience

For many problems, the Async Monitor will provide the developer with the insight they need to resolve their questions about their programs interactions with the async executor. However, some performance problems will require further inspection. The Async Monitor can help with some of that, such as when it allows a developer to pause a task. But sometimes a developer will need insight into other aspects of their program, such as where memory or CPU time is going. Other tools will be needed for such cases.

An appendix of this document shows a diagram of the expected developer work flow. The diagram presents steps that are described above; but it also shows decision points where a user might need to employ another tool.

Now that we have finished explaining the experience of someone using the tool, we will now discuss how to deliver that experience to our customers.

Cloudless everyday

You fall upon my waking eyes

Inviting and inciting me to rise

Implementation Plan

As mentioned above, there are two main components: the Async Monitor, and the Async Console.

The 2021 release of TurboWish will need to provide value without having any support from the Rust compiler or standard library. Therefore, the initial release will work by: 1. leveraging instrumentation added directly to the async executor, 2. encouraging service developers to add corresponding instrumentation, and 3. capturing stack backtraces at key executor events.

Instrumentation

The instrumentation will be responsible for documenting how the relationships between tasks and resources change over time. More specifically, the instrumentation will capture: task state transitions (running, ready, waiting), task acquisition of exclusive resource (e.g. locking a mutex), task modifications of resource state (e.g. sending on a channel or receiving from it), and task intent to interact with resources.

As mentioned in the Task Records section, there are two kinds of intent: conditional and unconditional. My long-term hope is to leverage some sort of heap-ownership tracing system to infer conditional intent, because signalling it via manual instrumentation will be arduous and error-prone. (Heap ownership tracking alone cannot infer unconditional intent, but it may be possible to leverage compiler analysis to perform such inference of unconditional intent.)

Component Separation

There are four short-term deliverables: 1. the Async Monitor, 2. the Async Console, 3. a specification of what instrumentation events the Async Monitor understands (which may come from the client code, the async executor, or any related libraries (e.g. Rayon or other crates that offer a thread pool)), and 4. the instrumentation of Tokio (following the aforementioned specification) to support the Async Monitor.

Diagram of Async Monitor Architecture

This is a rendering of the component separation. There are two running programs: The program being developed, and the Async Console. Within the program being developed, instrumentation events are generated and travel on the event bus, where they are observed by the Async Monitor running as a thread in that process. The Async Monitor communicates with the Async Console, presenting portions of the model according to the requests that arrive from the console.

Development Decoupling

In order to enable concurrent development of the four deliverables, we will follow these steps:

Deliverable: Event specification

First, we must pick some format for specifying the grammar of the instrumentation events. Anything with an existing off-the-shelf parser-generator available as a Rust crate will serve. JSON might be a reasonable choice here, at least for initial prototyping; but we should also evaluate whether other event description formats incur less encoding overhead than JSON.

(Another option would be a dedicated enum type for events. That would then need to be exposed in its own crate that downstream clients would need to add as a dependency. Perhaps more importantly: we are very likely to use tracing as the event bus, which takes string-encoded messages as the basis for its encoding.)

Second, we must make an initial specification of events. Just saying “it’s JSON” is not sufficient; we need to state how each desired event is actually encoded in the chosen grammar. This will evolve over time. It must cover everything listed in Instrumentation.

With these two pieces in place, we can spin off a few parallel tracks of development.

Deliverable: Instrumenting tokio

For instrumenting tokio. I recommend that we start by developing an event-validating async monitor (or more simply, “event-validator”). The event-validator is responsible for ensuring the incoming event stream is coherent; it might build up a minimal model as part of its coherence check, but it is not responsible for gathering all the metrics that are expected of the Async Monitor product itself. With an event-validator in place, one can test the correctness of tokio instrumentation as it is added. (We will also need to do integration tests once the async monitor itself is constructed.)

Deliverable: Async Monitor

For the Async Monitor, I recommend the following baby-steps toward implementation.

First, either make a new mock async executor, or select a pre-existing simple reference async executor (several exist on github), and using this as the mock executor. Use the mock executor for initial development of the Async Monitor: that is, the in-development Async Monitor would be exercised by building an internal model of the mock executor’s state.

The reason for employing a mock executor is two-fold: 1. it will be quicker to add (and modify when inevitably needed) the necessary instrumentation as the Async Monitor itself is developed, and 2. using a mock rather than tokio directly will help prevent inadvertent coupling between tokio itself and the async monitor; such coupling would make it hard to add support to other async executors like async-std in the future.

In addition to a mock executor, I also recommend making an mock event stream that will simulate both ends connected to the Async Monitor: it will generate the events that we expect to be signaled by an instrumented application sitting atop some instrumented executor, and it will also simulate a console attached to the other side of the Async Monitor. I recommend having a mock event streamer because this is the easiest way to generate a deterministic series of instrumentation events interleaved with console events. Making such event generation deterministic will be crucial for testing the async monitor.

Deliverable: Async Console

Finally, for the console, I recommend that we make a mock async monitor. It will simulate the Async Monitor’s side of the communication protocol used by any Console connecting to the monitor’s port.

(Defining the communication protocol between the Async Monitor and the Console is a prerequisite here. However, this protocol is only exposed to the Monitor and Console, and so it can be revised without having to worry about updating downstream clients.)

Deliverable Development Decoupling Diagram

The three decoupled tracks listed above are repeated below in a visual diagram, showing the dependencies between the pieces.

Long-term concerns; short-cuts for the Minimum Viable Product

See Appendix: Sacrifices for Minimum Viable Product

And through the window in the wall

Come streaming in on sunlight wings

A million bright ambassadors of morning

Metrics

The goal of these tools is to provide users with insight into performance pitfalls. How can we know if the Async Monitor and Console are achieving their goal?

There are two ways I can imagine approaching this: telemetry from the Async Console program, or evaluating out-of-band signals (such as sentiment analysis on social media). I will focus here on the telemetry option, under the assumption that any potential telemetry would be an opt-in choice presented to the customer when they first use the Async Console tool.

Telemetry: Success Metrics

For evaluating whether the tool is successfully providing users with insight, the most obvious answer is we could ask our users. When the console detects a problem and the user shifts to the dedicated problem view describing the problem, the console could also proactively ask the “Yes”/“No” question of whether the information it presents is helping the user solve their problem (or perhaps request a user happiness rating on a 1-5 to scale), and then ship those responses back to a service that collects such feedback.

Alternatively: we already plan to have the tool link to websites that provide documentation of various known performance pitfalls. Rather than building telemetry into the console, the linked websites could ask the visitor whether the Async Monitor and Console are working for them. (However, this approach runs the risk of omitting the experiences of users who do not follow the links.)

Telemetry: Failure Metrics

On the flip side of things, we also want to ensure that the instrumentation from TurboWish is not injecting too much overhead into our customer’s applications.

While it is probably not reasonable to try to measure the time spent issuing each individual instrumentation event, it is entirely reasonable to measure how many messages are being sent to the Async Monitor, how large they are, and which component (namely, the aync executor, or user code) issued them.

My recommendation is to monitor how much load the instrumentation is putting onto the event bus, according to the event density over a sliding window of time (let’s say 100 ms long). If the event density from user code exceeds some predetermined threshold in any given 100ms window, then the Async Console signals a problem report to the developer, telling them that their program’s instrumentation may be slowing down the program. If the event density from the async executor (lets say tokio) exceeds a similar predetermined threshold, then the Async Console reports that up to a service associated with the tokio project. (Or, if the tokio project does not see value in getting such traffic, then the Async Console could suggest that the user file a bug on the tokio github.)

Security Concerns

The instrumentation added to the tokio runtime and client application code may expose details of internal operation that customers do not exposed to the world at large.

We should check that there are controls in place that ensure either: 1. code deployed to production does not have such instrumentation turned on, or 2. the async monitor is not initiated on production systems, or 3. the port associated with the async monitor is blocked by a firewall guarding the customer’s host machine or intranet.

Conclusion

The Async Monitor and Console answer a developer’s questions about how their code’s async executor is behaving as their program runs. They provide a summary of the executor’s behavior, with metrics about the tasks and resources their program is using. They allow the developer to identify scheduling decisions and see how task and resources depend on each other.

Furthermore, they serve as the foundation of TurboWish. So, in effect: We aim in 2021 to deliver the foundation for 2022 and beyond. We will work with the Rust community to lay this groundwork, and our community will be enabled to make even more amazing tools atop this foundation.

And no one sings me lullabies

And no one makes me close my eyes

So I throw the windows wide

And call to you across the sky

FAQ

• Q: Why build the Async Monitor into the application binary? Couldn’t that be part of the Async Console instead, and have all the events stream to the Async Console?
  • A: The primary reason that the Async Monitor is part of the application binary is that we believe streaming all the events that the monitor needs to observe to an external program will inject too much overhead. A secondary reason for building the Async Monitor into the application binary is simplicity. If the monitor were a separate program, then one cannot ensure that the full stream of events reaches the monitor. At best one could strive for a suffix of the event stream, which means that the monitor has to be designed to still produce a useful model given such a suffix. (A previously considered architecture handled the suffix problem by allowing the monitor to pose queries to the instrumented application, in order to get information such as “what is the current set of tasks?”)

Appendices

Appendix A: Sacrifices for Minimum Viable Product

Minimal Viable Product: Shims for resource instrumentation

Long-term concern: We want developers to be able to deploy the Async Monitor with minimal changes to their code. A few lines of initialization code is acceptable, but broad changes to many files is not a good customer experience.

Ways to accomplish this are under active discussion, but any solution with such limited source code modification will require one or more of: 1. changes to the Rust standard library, 2. changes to the rust compiler, or 3. use of low-level features such as dtrace probes or eBPF. We have not yet decided on which of these options is appropriate.

Short term sacrifice: In addition to the initialization code described at the start of the description of the developer experience, initial versions of TurboWish also require the developer to swap in instrumented versions of common modules; a provided lint will guide developers in this process and help ensure no instances are overlooked.

1

use turbowish::sync::Mutex; // was previously `use std::sync::Mutex;`

(As already stated above, longer-term, we hope to leverage other facilities to get these instrumentation events.)

Minimal Viable Product: Tokio-specific

Long-term concern: We want the Async Monitor and Console to work with any popular async executor: tokio and async-std are obvious choices here. For the Async Monitor to be useful on an async program, one must use an async executor that has appropriate TurboWish instrumentation added to it.

Short term sacrifice: We will deploy a prototype with tokio, but try to keep in mind any differences with other async executors as we design the protocol used for the async executor to communicate with the async monitor.

Minimum Viable Product: Require client instrumentation for full-features

Long-term concern: Client application code can benefit from adding extra instrumentation to their code. However, developers should be able to use and benefit from TurboWish without going to extremes adding new instrumentation beyond what the executor has out-of-the-box.

Some components of the Async Monitor will work with zero client instrumentation. In particular: the initial console output that shows the list of tasks and how their time is spent between polling, ready, and waiting does not require any client instrumentation.

However, other features of the Async Monitor, such as tasks listing resources with which they intend to interact, require either Rust compiler support or client instrumentation, or maybe both.

Short term sacrifice: We will prototype under the assumption that clients are willing to add instrumentation. The Async Monitor will differentiate between instrumentation that is “trusted”: cases where instrumentation bugs will make the Async Monitor and Console produce misleading results (e.g. if transitions between polling and waiting are omitted or forged), and “untrusted”: cases where instrumention bugs, by design of the monitor, will at most lead to confusion, but not outright lies (e.g. if incorrect attributes are attached to a task or resource, the console output showing those attributes will be likewise incorrect, but it need not disrupt other parts of the Async Monitor or Console).

Appendix B: TurboWish User Work Flow

Appendix C: Instrumentation notes (brainstorming)

(This section assumes that a set of related wakers is a reasonable and necessary proxy for “resource”. This assumption will be revisited during development; the executor controls creation of wakers, so it is a natural point to instrument; but the wakers may not have sufficient context available at their creation points to support describing their associated resource.)

The most basic functionality for the task/resource graph user story requires the executor to emit events whenever:

• a task is spawned,
• a task is dropped,
• a waker is created,
• a waker is cloned,
• a waker is dropped, or
• a waker is transferred from one task to another.

Supporting other user stories will likely require tracking other information as well (such as how many pending futures have had wake called and are awaiting a call to poll). We may need to add additional hooks to the async executor, analogous to the support for “pause”, that the Async Monitor can invoke to turn on tracing of such information.

The emitted events should include unique identifiers (UID) for any referenced task/wakers.

• For values that are themselves boxed or own a heap-allocated value, we should be able to use a memory address as a UID, as long as we also include some sort of timestamp with the events (and the Event Collector will infer when memory is being reused and update its internal model accordingly).
• (If we need to track values that do not have a uniquely associated heap values, then we may need to add some sort of unique-id generation for them. So far I haven’t seen a need in tokio’s types.)

The emitted events should also include some notion of the calling context for the event. This calling context should be meaningful from the viewpoint of the Client App Code.

• For example, when <TimerFuture as Future>::poll calls cx.waker().clone(), we want the waker construction event to include (at minimum) that a waker was created from TimerFuture, so that one can properly tell what kind of resource that waker is associated with.
• (It would be even better to include enough information in the event for the Event Collector to know which specific resource is associated with the waker, rather than just its type.

These events may include program-internal details such as (partial) stack traces that will include memory addresses of program code

• (We cannot change existing APIs to thread through details like file/line-number info in the style of #[track_caller], so in general this is the only way I expect to be able to infer calling context without putting undue burden on client code.)
• More specifically: Based on my understanding of the API’s available, providing contextual info about the calling context of cx.waker().clone() will require either 1. client instrumentation that sets some thread- or task-local state, or 2. backtracing through the stack to find instruction addresses that the Async Monitor can, via debuginfo, map back to the calling context.

As I walk, I think about a new way to walk

In my previous post, I described the observations that led us to select performance tooling as a focus, as well as the goals document I wrote to guide the next steps of the project. Now I want to show you the User Stories I wrote, to show what I think our customers want (and can reasonably expect) out of performance tools.

Everything that follows was taken verbatimOkay, not verbatim. As promised, I am spicing things up a little, this time via embedding of outright quotations (at an offset), rather than section headings. from the document I presented to the other members of the AWS Rust team. This document is an attempt to describe what I envisage as awesome developer experiences doing performance investigation. As with part 1, I will be adding side-commentary on the right-hand margin.

As I think, I’m using up the time left to think

My next post will present the design document that arose from these and other stories, as well as discussions about the realities of what we can expect to implement.

Note: These are meant to be read in order: The first describes some up-front investment that the later stories either 1.) assume other developers already put in, or 2.) in the case of user “Barry”, state explicitly that such investment of effort is deliberately skipped by the developer.

And this train keeps rolling off the track

Trying to act like something else

Trying to go where it’s been uninvited

⊕ I jumped right in and started making up a story about a newcomer to async Rust. I did not actually look at examples of user stories as they are typically used in Agile development, or even really how it is done at AWS. It has been a long time since I had read essays describing “how” to do Agile development; looking at that now, I see that typical user stories are one or two sentences. (I’m not sure I can properly convey what I want these tools to do in one or two sentences. I mean, it would be a place to start, but I wanted to spell out the experience. Maybe the reality is that the step I’m doing is not a user story. But it does seem like it at least falls into the “Working Backwards” style that is promoted here at AWS. It also matches the “shiny future” story writing approach that Rust’s Async Foundations working group has been trail-blazing (apart from the fact that I skipped the step of writing a description of the “status quo” for each of the stories here).

User Story 1: Abigail

Abigail is getting started with async Rust development. She’s worked her way through the Async Book and transcribed some example code. Now she’s trying to write her first medium sized program, a terminal-based chess program, and it doesn’t produce any output after she makes her first move and is waiting for the opponent to move.

After asking for advice about what to do on the Tokio Discord chat, she enables the turbowish feature in her Cargo.toml, and recompiles. She receives warnings from the compiler saying that there are calls to task::spawn but no contextual labels anywhere in her crate.

Abigail reads the compiler’s suggestions for the kinds of declarations to add, and adds annotations accordingly, labeling her tasks. (Her program is roughly structured according to an actor model, so she has labels like “game AI”, “move validator”, and “terminal UI”).

Abigail reattempts a build, and now notices some of the previous warnings were not addressed in her change and are emited again: These warnings are about missing labels on resources, namely the channels used for communication between the tasks, e.g. “user input”, “player move”, “opponent move”, etc. She adds labels to the channels, and the program now compiles with no diagnostic warning. She runs her Chess program.

The Chess program starts up, and this time it includes, as part of its console output, a socket to connect to, which is serviced by a dedicated Turbowish thread. She runs a separate tokio-console program, connecting it to the advertised socket on the Chess program. Events begin streaming from the TurboWish thread to the tokio-console. The tokio-console output notes that it has a webserver mode available; she opts into that, and the tokio-console spits out a local URL to connect to. Abigail starts up a web browser and connects it to the URL. The rendered page lists the three tasks.

The Chess program again unexpectedly blocks (or, more precisely, livelocks) after she inputs her first move. Abigail looks at the rendered page, and notices a button that says “task dependencies.” She clicks it, and the page renders a picture showing a directed graph,This is the core part of my vision for performance tooling that I strived to preserve when I ported this story into Barbara Makes A Wish on the Async Vision document. linking tasks and resources (or more generally, wakers), which depicts both 1.) what resources a task is waiting on, and 2.) what tasks are responsible for making those resources available for use.

The resulting rendering shows a cycle in the graph that looks like this, where circular nodes are tasks and rectangular nodes are resources shared by the tasks, like channels.

⊕ I am breaking my rule about presenting the verbatim text here. In the document I shared with my colleagues, the underlying file didn’t support mermaid or any other markdown-oriented diagramming tools. So there I resorted to something like (terminal UI) -> [board update] -> (move validator) -> [player move] -> (terminal UI) in that document, where parentheses and square brackets represent circle vs square node shapes, differentiating tasks from resources. But I could not bring myself to subject my readers here to that, when its so easy to put in a mermaid diagram.

Notably, there are no paths in the rendered graph from [board move]I must have meant for this to say [board update]. I had not caught that typo until just now. to (game AI).

Abigail wonders why the move validator task is waiting for a player move, when she would expect it to be waiting an opponent move from the AI.

She realizes there must be a error in her logic for how either the “move validator” or the “board update” are set up, since she would expect the “game AI” task to be somewhere in the path above.

Moving her mouse over the graph, she notices that the graph edges link to points in her source code. She starts digging in, using the provided links as direction for where to look to figure out what went wrong in her logic.

And the rain falls down without my help I’m afraid

⊕ When I wrote these stories, I felt it would be useful for all of the characters to have names where their first letters corresponded to the letters of the Alphabet: Abigail, Barry, Chris, Daphne. Some feedback I got back almost immediately from Niko Matsakis was “Why didn’t you re-use the characters from the Async Vision cast?” I don’t have any great answers to that. (The obvious answer was a sheepish “I haven’t read the Async Vision document yet…”; I have corrected that oversight since then.)

User Story 2: Barry

Barry is an experienced tokio developer. He has been asked to help out with an AWS cloud service using tokio. The developers of the service are seeing higher latencies than expected for relatively small service requests.

Barry enables tokio’s turbowish feature. He opts not to take the compiler’s diagnostic suggestions regarding missing contextual labels and downgrades the diagnostic to #![allow] across the crate, knowing that TurboWish will infer labels and attach them automatically based upon crate and module names (and a fixed number of stack frames, when debuginfo is available).

Barry connects tokio-console to the cloud service’s socket, and chooses its terminal user-interface instead of the webserver mode.

Barry first asks for a live feed of tasks transition between (running, waiting, ready). The feed produces too much output to the terminal, so Barry sends it to a file instead, and then inspects it in a text editor, trying to see trends by eye. It is a lot of data though, too much for Barry to interpret without more tooling.

Barry has dealt with parsing this output before, though.

Barry has a home-grown script to extract a CSV file with numeric data from the output. Barry imports the CSV file into a spreadsheet and then constructs a histogram with buckets of ranges of individual wait-times, and how many tasks’ transition points are in each bucket.

From this, Barry identifies a small set of tasks that spent an unusually long time in the waiting state. Going back to the original feed, Barry finds the the records for those long waits, which include a link to the point in the source code where the long waits occurred.

Barry looks, and sees a call to std::thread::sleep instead of tokio’s sleep function. [TODO: I’d be happy to put in some other example here of blocking code that should be replaced with non-blocking code.]This TODO was in here during the team document review. The basic response was “eh, std sleep is good enough as an example. It is a real instance of a much more general mistake that people do make, and it gets the idea across.”

Barry replaces the call std::thread::sleep with tokio’s sleep function, and returns to the histogram to see what other instances of unusually long wait times he can find.

And my lawn gets wet though I’ve withheld my consent

⊕ With the Chris story, we have a sudden shift from async-oriented performance issues to compiler developer performance issues. what can I say other than “write what you know”. ⊕ (Interestingly, all the essays I can find on that trope indicate that it is meant to be about knowledge of inner emotions, not about facts or experience.)

User Story 3: Chris

Chris is a contributor to the Rust compiler, rustc. Chris is well-versed with Rust, at least for batch programs like the compiler, but they do not do any async Rust development. Chris wants to gain better understanding of memory usage internal to rustc itself.

After asking around in the #compiler-team Zulip stream, Chris enables the build.heap_inspector setting in rustc’s build configuration file, and rebuilds the compiler.

Chris also fires up the TurboWish web frontend. This invocation of TurboWish uses a specialized config file developed and maintained by the Rust compiler team that handles interpreting the types that are special to the compiler.

Chris uses environment variables to indicate which points in the compiler’s control flow are of interest to them. More specifically, Chris wants to inspect the heap as it stands immediately after the borrow-checker runs.

The compiler obliges: right after the borrow-checker runs, it starts from a set of known roots (tagged as such ahead of time by the rustc developer) and traverses the ownership graph from those roots,You can see my garbage-collection background leaking through here. I guess this is an instance of “what what you knew…” emitting a description of the arcs it encounters during its traversal. For each heap-allocated value the traversal encounters, the traversal code also includes information needed to reconstruct its “payload size”; i.e., enum values will include their discriminant, vectors and string include their length, etc.

The TurboWish web front end receives this stream of graph traversal data. It uses this to provide a table describing the number of objects of each type, and how much internal fragmentation it detects based on the reported payload sizes. Chris inspects this table and determines that that there is an opportunity to reduce fragmentation by changing one of the enums to use a Box. Chris tries it out and sees the memory consumption of the compiler go down, and files a pull request with the change.

(Note: This user story may or may not not provide value over other tools like dhat, depending on 1. whether it allows tracking allocations at a finer grain than malloc calls (e.g. sub-allocations within an arena) and/or 2. how much value one puts on being able to inspect portions of the heap rooted at certain owners. See also the appendix, which includes a more ambitious “dream” proposal for Chris, but also one that I personally am not as sure actually pays off in terms of customer value versus mere “coolness” factor.)

When this grey world crumbles like a cake

User Story 4: Daphne

⊕ With Daphne’s story, we return to async-oriented performance analysis. Async is a hot topic, and I figured it was worth exploring other stuff we could deliver. In this case, the idea is to turn the internal logging data into a rendered message sequence chart. (I actually do not know if people use message sequence charts on real logs in practice. I have seen them used very often for presenting small snippets that explain a protocol, but are they actually useful for traversing a complex series of interactions? I can imagine it could be, especially in an interactive presentation that can highlight links and automatically re-center on a selected end of a given arc in a graph. Daphne is maintaining an existing a web service built atop tokio. She gets reports of poor performance for certain input queries that match the “E” command for her service, which already has TurboWish integrated. In response, she attaches tokio-console to her service, and tells it to trace the flow of requests matching the “E” command (more concretely, [{req.path="/Q"}]=trace), and then fires up the TurboWish web front-end.

Daphne chooses a tab that says “Show Scheduler.” The resulting page looks something like Message Sequence Chart: Each of tokio’s executor threads has a vertical line, and as futures are polled, they have corresponding boxes on the corresponding thread. Since Daphne has limited the output to just events related to the “E” command, all the futures she sees being scheduled are part of that query.

From skimming the resulting charts, Daphne sees that when the “E” futures are being polled, they seem to yield again promptly. She does see some evidence the future is migrating to different threads, and she makes a note to try to investigate whether there is a way to ask tokio to avoid such thrashing (and also, whether there are metrics should could gather about whether doing so could help).

There are also points in the program flow when no future related to the “E” query is scheduled. The event stream does not include information about which futures are polled during those times. Those portions of the executor threads vertical lines are colored dark green: they may be doing useful work, but its not work directly related to the events that have been filtered in.

Daphne realizes that she needs to find out whether the “E” futures are being left unexecuted because they’re still waiting for their corresponding wakers to be invoked, or if the “E” futures are all ready and there is some other issue in the scheduler (or in the other tasks that causes them to starve the executing threads). Daphne goes to tokio-console and issues the command requesting the feed of all tasks transitioning between (running, waiting, ready). With that, the message sequence chart now shows that her “E” futures are indeed ready much of the time. She also sees that there are large blocks of time where all of the threads in the pool are running without any interrupt. Daphne hovers the mouse over those rendered blocks of time, and a pop-up window shows a description of the futures that are being polled at that time. Daphne goes off to investigate that piece of code, and discovers a for-loop with some semi-expensive body that never yields to the executor.

Appendix

I’ll be hanging from the hope

That I’ll never see that recipe again

User Story 3b: Chris’s Dream

Chris wants to inspect the heap as it stands immediately after the borrow-checker runs.

The compiler obliges: right after the borrow-checker runs, it pauses and prompts for the user to connect TurboWish. Chris connects the rustc-console app and interactively asks what root objects are available to inspect. One of the named objects in the result is the “MirBorrowckCtxt” (i.e. the borrow-checking context). Still working at the terminal rustc-console, Chris first asks for how much memory is solely owned, potentially indirectly by the “MirBorrowckCtxt” object. rustc-console spits out a number. Chris then asks for how much memory is owned, potentially shared with others (e.g. via Rc), by the object. rustc-console spits out another, much larger number.

Chris connects the TurboWish web front-end, and then queries for a ownership tree describing the values owned by the “MirBorrowckCtxt” object.

TurboWish does a graph traversal over the owned heap allocated objects, starting from the borrow-checking context (MirBorrowckCtxt) as the root, and emits a description of the arcs it encounters during its traversal, as well as well as the data payload for any fields of types that have been previously marked by the rustc developers as of interest.

The TurboWish web front end renders the resulting graph. But in addition to the direct ownership relatinships implied by the Rust language semantics the graph, it also includes arcs for any &/&mut references in the graph, and it also includes some domain-specific dotted arcs for keys into tables that are meant to be interpreted as references. (This of course is only possible due to the rustc-specialized config file that was included when TurboWish was loaded up.)

⊕ I think this whole appendix was just an excuse to try to squeeze in a reference to Hyperbolic Trees. Just over twenty years ago, I spent a summer (or was it a year) hacking together code to render a graph via such trees as part of a project called Footprints; the linked paper still has screenshots of it.

However, since the object graph is massive, the rendering does not fit on the window. Chris first toggles an option to change the rendering to use a hyperbolic tree (see https://en.wikipedia.org/wiki/Hyperbolic_tree), where the current object is presented centrally, and immediate children are surrounding smaller nodes, and grand-children are still smaller, et cetera. The hyperbolic tree presentation compacts the view significantly, at the cost of obscuring details of distant descendants and ancestors, as well as cousins.

There is also an option on the presentation UI to have the circles for each node be different sizes depending on how much space they individually occupy on the heap (not including their linked children); Chris enables this setting, just to get a rough visual feeling for where memory space is going in the ownership tree.

And that’s all the user stories. That’s the whole document.

For the sequel, I will be presenting the design document that I think best represents the plan going forward.

We are still figuring things out, to some extent, but I continue to dream about what we can do for our customers, the Rust developers of the world.

When the word comes down “Never more will be around”

Thought I’ll wish we were there, I was less than we could bear

And I’m not the only dust my mother raised

I am not the only dust my mother raised

TurboWish, in a nutshell, is our umbrella term for a planned suite of tools for understanding your Rust program’s dynamic behavior. We want the tools to especially focus on providing insights into the performance characteristics of your program.

Let me take a moment to tell you what has happened so far: How did I get here?

(if you want skip the history/motivations and jump straight to the doc, you can follow that link; you may find yourself in another part of the world.)

Into the blue again

About six months ago, I left Mozilla and joined Amazon Web Services (AWS). I am now part of a new team: the AWS Rust team.

Our team charter is simple: “The AWS Rust team works to make Rust performant, reliable, and productive for all its users.”

Details of what that means are spelled out via our team’s tenets. One of those tenets, one that is incredibly important to me, is that “we work in the open.” This blog post is itself my own attempt to follow through on that promise: I want to be more open about what I’ve been writing down and circulating amongst a relatively small group of people, and try to be better about putting my draft ideas out into the open from the start going forward.

Same as it ever was

From my perspective, I still have the same personal career goals: make the Rust programming language an awesome option for developers, by improving the code of the Rust compiler, the design of the Rust language, and the organization of the Rust community.Our community is awesome. I would never want to claim that I am trying to “improve” the community itself. But… I also would not claim that we have any silver-bullet when it comes to self-organization.

Those were my goals when I was at Mozilla, and they are still my goals now.

You may ask yourself

What do performance tools have to do with those goals?

Our team’s inspiration for this focus was based on a few observations.

Part of Rust’s promise, the reason people are trying Rust out at all, is that it says it can give you the kind of high-performance that in the past was normally the realm of low-level languages like C or C++.Or FORTRAN, I suppose, though I do not know if Rust is doing much today to target the scientific computing communities that have historically used FORTRAN.

At a customer-obsessed company like Amazon, that promise raises an important question. Are our team’s customers,Some people might be confused by this use of the word “customer.” I admit confusion too, since I do not expect to get any compensation from Rust developers themselves. But I still like the word here; I like that it challenges preconceptions about what a customer is. (Also, I have been using the word “customers” for projects written in Rust, such as Servo, since at least 2016.) the developers using Rust, actually seeing such performance wins in practice?

There is water at the bottom of the ocean

Based on some discussions and informal interviews with fellow Rust developers, I saw three groups of people with differing answers to that question.

The first group says “Rust is great! Once I got it compiling and my unit tests passing, the code works and meets my performance expectations.” These customers always made me happy; they are usually happy to then point out what pet features they want to see in the future.

The second group says: “I got it compiling, but its not actually as fast as I had hoped. What should I do now?”

Okay, to be fair, I’m making a massive over-generalization there. The second group doesn’t always say that. Some of them do extensive analysis to identify awesome ways to make things faster. Some also document exactly what they did and how you can do it too.

But the point remains: If Rust isn’t meeting your performance goals, its not always obvious what to do. Some people with a lot of in-depth experience with compilers or systems analysis have tools in their utility belt that work well for C/C++ and … they sort of work for Rust.

But those tools do not work as well as I would like, and I’m not sure they are the right answer for the bulk of our community. I said five years ago that I want Rust to be Systems Programming for Everybody, and part of that story is that we need tools that everybody can use and understand.

How do I work this?

The third group says: “I struggle to figure out what design to use. I struggle to get my service compiling. You are asking me what I think about performance, but I’m debugging deadlocks from my async code.”

This third group raises an entirely different line of thinking when it comes to tooling. When I first starting thinking about performance monitoring tools, I was thinking solely in terms of gathering and presenting metrics, such as CPU cycles, memory utilization. But this third group represents a broad set of customers for whom such focus is premature: They are hitting road blocks that prevent their project from ever leaving the prototype phase. One big source of problems was async code; there’s a good chance that async-specific tooling will provide the biggest return on investment here.

The concerns of the second and third groups what led us to the TurboWish project: tools that will help our customers make Rust deliver on its promises: performant, reliable, productive.

Letting the days go by

After performing a set of informal interviews with various customers both within and outside of AWS, I felt confident that there are real problems here to solve. We want Rust developers to have good tools ⊕Ideally, the tools would leverage features of Rust itself: the type system, the ownership model, the value rendering system provided in std::fmt, the virtual method dispatch tables, et cetera. But that is not a strict requirement; it is just “nice-to-have.” on hand that will answer their questions.

So, I jumped into writing a goals document, so that our team could explain to other teams at AWS, and ourselves, what we want to make. I shared it with the rest of the AWS Rust team in the middle of February 2021; they had lots of feedback, but I am not including that here (mostly to avoid having to coordinate authorship of this blog post).

I am ending this blog post with that goals document, warts and all, along with meta-commentary in the right-hand margin notes. In follow-on blog posts this week, I will share some of the next steps that followed after I wrote this.

doc: Opening Line

⊕ You can see in the opening line itself the focus on async/await, for better or for worse. TurboWish is a framework for profiling Rust programs, focused on illuminating the performance and resource usage of task-oriented code written with async/await.

doc: Goals

⊕ I played a bit of a semantic shell game here, with my use of the term “production code.” That term could be interpreted as “code deployed as a live service.” Or it could be interpreted as “code compiled in release mode, but still running on development boxes.” I plan to talk more about this distinction in later posts, but the short version is: I think we can provide great value today to developers working on their development boxes, without trying to concern ourselves with making a tool that is sufficiently low-overhead and security risk-free that it could be part of a deployed system. Profile Production Code: Incorporating the TurboWish Framework is low-overhead: it can be incorporated into production code without producing an undue maintenance burden and without incurring significant performance overhead.

Domain-specific Feedback: Frameworks and applications can provide data for specialized metrics, specific to their internal architecture.

Understand Hidden Costs and Connections: Frameworks like tokio ease writing asynchronous code because they hide a number of details behind abstractions (such as generator code produced by the Rust compiler, or task queues managed by the tokio runtime). TurboWish exposes those hidden details, allowing developers to correlate them with other program events. It also exposes connections that humans usually have to reconstruct by hand (such as future to resource to future chains that can yield deadlock), allowing one to directly see from Rust’s ownership model how resources are being held in the object graph.

Framework Agnostic: Many of Rust’s customers use tokio, but not all of them. async-std and fuschia_async are other frameworks for asynchronous programming. TurboWish can provide value to any such framework (though it may also provide framework-specific functionality when warranted). For our initial releases, we can focus on tokio alone, but expect integration with others if tokio proves successful.

⊕ I included this goal specifically because I had done a bunch of investigation into using rr, and discovered during that time that some of the cloud development machines hosted on EC2 do not support the performance counters that you need for rr to function. EC2 Instance Type Agnostic: If we make use of any OS specific features (e.g. dtrace probes), they will be available on all EC2 AL2 instances, regardless of instance-type. (Specifically, we cannot require access to CPU performance counters.)

⊕ It was an interesting exercise writing this schedule. There are a number of constraints that I was trying to meet that are not represented in this table. The biggest one was that the Rust release schedule itself follows a six-week cadence; if TurboWish needs any support from rustc itself, and we want it to be available in the stable version of the compiler at the end of October, then that means any such support needs to land in the nightly version of rustc before July 29th.

doc: Milestones

doc: Milestone explanation

Development Tracks: Some features of TurboWish will provide the most value to customers if they are developed in concert with additions to other components of the Rust ecosystem, such as the Rust compiler. However, we are not the sole owners of the Rust ecosystem nor those components. We need to identify target components of interest early (and I am assuming that part of that identification will require actual prototyping, which is why I am allocating a month for such prototyping in parallel with drafting the PR/FAQI was unaware of the PR/FAQ format before joining AWS, but it is apparently used in many customer-centric and agile development models; see also this medium post. ), so that we can properly prioritize such development. In each case where we do not own the component, we must also establish the backup plan if our desired changes will not land in time for use in the product this year.

Launch Partners: Customers within Amazon are willing to evaluate pre-release versions of the product. We should strategically select three such customers to partner with us; these are the Launch Partners. We will give each such launch partner 1.) demonstrations of the alpha proof of concept, 2.) access to the beta minimum viable product, and 3.) dedicated engineering time for integrating the beta into their service. In exchange, the launch partner will give us feedback on the alpha and beta versions of the product (which will inform each subsequent development sprint).

⊕ For some reason I was especially proud of the distinction being drawn in this paragraph. I have a somewhat superficial understanding of project management and Agile development methods, so I was not really thinking about whether demo’s are common products of a sprint. (And a post by Johanna Rothman makes the great point that even a product that is “only” demo-able has still demonstrated integration amongst the team.) From my perspective, the demo/release distinction had an entirely different motivation: I simply did not see time in the schedule for the year for two full release plus integration plus customer-feedback cycles. Alpha Demo versus Beta Release: We want to move quickly, develop a minimum viable product and then iterate on it until we have something that delights our customers. We also want to work with a set of dedicated launch partners to evaluate an early version of the product (the beta). However, a product like this is unlikely to be a tool that can be trivially integrated: we expect there to be some amount of development effort associated with linking TurboWish into a given code base. Therefore, we do not expect our launch partners to be able to participate in multiple iterations of evaluating the product, simply due to the amount of development effort each integration is likely to take. So, we propose using different evaluation methodologies for different iteration cycles: For the alpha version, we will integrate TurboWish into code bases that we choose ourselves, give demos of the integrated result to our Launch Partners, and use their feedback in subsequent development of the alpha. For the beta version, we will work with our Launch Partners to integrate TurboWish into their code bases, and then at the end of the integration period, we will use the feedback they provide to make the final changes to the product.

doc: Risks, Mitigations

Risk: Time from “development tracks identified” to “PR/FAQ published” is only a week.This risk was accurate. It took quite a while longer for me to make a suitable design document. That document will be the subject of a later post.

Mitigation: We need to develop the PR/FAQ in parallel with doing the feasibility studies that identify the development tracks. (But I do not want to make them independent milestones; I want to be confident that the features in the PR/FAQ can be constructed before I try to enlist Launch Partners.)

Risk: Rust compiler leadership/maintenance will distract pnkfelix.Another way to look at this: “the TurboWish work will distract pnkfelix from their compiler development efforts.” And indeed, both can be true.

Mitigation 1. Get buy-in from others and spread development effort

Mitigation 2. Compiler Team focus for 2021 is rustc contributor experience, especially w.r.t. performance. Hopefully synergies will emerge from that.

Risk: Not much time remaining in February to establish user stories. Felix’s personal focus for short term are memory usage issues, and so he has been contributing stories related to that. But many customers express concern related to async/await, especially about understanding why their tasks fail to progress (i.e. sources of deadlock).

(No mitigation documented.)I did not come up with a mitigation for this risk. I just hoped I would have time to write something reasonable.

Risk: Some features may depend on some amount of GC style tracing, potentially starting from local owned variables on the stack as roots, and in any case traversing values on the heap. (For example, automatically dissecting ownership chains between futures and resources in order to identify causes of deadlock could make use of such tracing.) pnkFelix has experience in this area and believes it to be a solvable problem (especially given the profiling goal, as opposed to correct integration with arbitrary 3rd party GC tech), but it is not a settled problem.

Mitigation: Leverage ownership of relevant frameworks where possible. E.g. you don’t need to trace the local stack if you know your “root set” of interest is the set of tokio tasks. And you don’t need to make an general-purpose value-tracing system when a make-shift trait and associated derive will suffice for the specific problem at hand.

Time isn’t holding us / Time isn’t after us

And that’s the goals doc, as it stood in mid-February

Like I said above, there was a bit of a journey to get to this point. And even with this document in hand, we do not have enough to start making code: I wouldn’t be able to hand this to a programmer and say “do this.”

But I had goals. And I had a schedule. What was next on the schedule? User Stories., the subject of the next blog post.

The problem

Driving deallocation of state via ref-counting under our current atomic API means that you cannot avoid having a &AtomicUsize variable at the same time that you are deallocating the block of memory that contains the referenced atomic usize state.

The proposal

The idea is simple.

Experienced Rustaceans know that in the general case, &T doesn’t actually mean immutable; it merely signals that unsynchronized mutation through aliases is disallowed. Interior mutability is still an option, via Cell or AtomicUsize.

All such sources of interior mutability are tracked in the type system. Namely, they all have an UnsafeCell<U> somewhere in the structure layout that tells the compiler that such mutation is permitted on the U inside the unsafe cell. Thus the compiler knows it cannot make optimizations that rely on that U remaining invariant.

So, here’s the suggested change: if code needs to allow for concurrent actors (or even aliasing accesses on the current thread) to mutate some state, that same code must allow that those accesses might deallocate that state.

⊕ That is, “may be modified or deallocated” unless (of course) the compiler/author has some extra proof or established invariant that the memory M cannot be deallocated. Such invariants are the very basis of ref-counting, for example. In other words: in addition to saying that the content of an &UnsafeCell<U> might be modified via an alias, we also say that the memory holding the unsafe cell may even be outright deallocated, despite any extant &-references to that unsafe cell.

These statements about “what may happen” are talking about what reasoning you can employ locally, based on the type declarations.

(Background) The Issue: Side-effects of atomic operations

The status quo: I will present a strawman atomic-reference counted type (analogous to std::rc::Arc).

It needs a heap-allocated payload, along with the reference count.

As an added part of this strawman presentation, I’m going to have the ref-counted object also carry a single-character label. The label will be used to illustrate potential program transformations (i.e. compiler optimizations) as we progress. (Furthermore, I will try to catch use-after-free bugs resulting from these transformations by reallocating memory and scribbling on it before the label accesses.)

1
2
3
4
5
6
7
8
9
10
11
12
13
14

use std::sync::atomic::{self, AtomicUsize, Ordering};

type Data = [u8; 128];

struct LabelledPayload {
    label: char,
    #[allow(dead_code)]
    data: Data,
}

struct Inner {
    ref_count: AtomicUsize,
    payload: LabelledPayload,
}

fn dealloc is a small shim around Box-based deallocation to keep the examples cleaner; fn scribble is a routine that tries to catch us if we have a use-after-free. by making a fresh allocation and overwriting its state.

1
2
3
4
5
6
7
8
9
10
11
12
13

unsafe fn dealloc(ptr: *const Inner) {
    drop(Box::from_raw(ptr as *mut Inner));
}

fn scribble() {
    Box::new(Inner { ref_count: AtomicUsize::new(0), payload: LabelledPayload { label: 'X', data: [0; 128], } });
}
macro_rules! scribble_then_println {
    ($template: expr, $label:expr) => {
        scribble();
        println!($template, $label);
    }
}

(In reality the Payload type would usually a generic type T, but this is detail we can side-step for this discussion. The same issues with atomics still arise, and side-stepping generics simplifies the presentation in various ways with respect to issues of variance, PhantomData and ?Sized.)

You will also need the handles that have (fractional) ownership of each heap-allocated Inner:

1
2
3

struct Handle {
    ptr: *const Inner,
}

(In practice the type of the ptr field would actually be core::ptr::NonNull<Inner>, but that is a niche optimization we can sidestep in this presentation.)

The allocation code and ref-count maintenance code would look something like this

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

impl Handle {
    pub fn new(label: char, data: Data) -> Handle {
        let x = Box::new(Inner {
            ref_count: atomic::AtomicUsize::new(1),
            payload: LabelledPayload { label, data },
        });
        Self { ptr: Box::leak(x) as *const Inner }
    }
}

impl Clone for Handle {
    fn clone(&self) -> Handle {
        unsafe { (*self.ptr).ref_count.fetch_add(1, Ordering::Relaxed); }
        // (we'll skip preventing overflow; see Arc code for those details.)
        Handle { ptr: self.ptr }
    }
}

The troubles arrive with the destructor code that decrements the ref count and deallocates the heap-allocated value if the resulting ref-count is zero.

Version 1: “inner-ref handled inline”

One “obvious way” to do it is with something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

#[cfg(feature = "version1")]
impl Drop for Handle {
    fn drop(&mut self) {
        let inner: &Inner = unsafe { &(*self.ptr) };
        let label: char = inner.payload.label;

        // reminder: concurrent actors may also be running drop, decrementing the ref-count...
        let pre_decr = inner.ref_count.fetch_sub(1, Ordering::Release);

        // ... so at this point in control flow, if `pre_decr > 1`,
        // then someone else might be freeing `*inner` (or have
        // already freed it). Now the only time it is safe to
        // access `*inner` is if we can prove we are sole remaining
        // owner (i.e. that `pre_decr == 1`).

        if pre_decr == 1 {
            unsafe { dealloc(inner as *const Inner); }
        }

        scribble_then_println!("version1 dropped handle to {}", label);
    }
}

This code is designed to deallocate the heap-allocated Inner on some paths. But at the same it does that, the local variable inner: &Inner is still in (lexical) scope.

Is this okay?

In general within this document, when I write “is this okay”, it is meant as a rough short-hand for the following more elaborate text:

Does allowing a deallocation like this break any of:

⊕ (Of course items (b.) and (c.) should in principle be derived from (a.); but we do not have a specification on hand, while we do have concrete examples of compiler optimizations and unsafe code that people write in practice, so that is why I try to spell them out as explicit concerns to keep in mind.) (a.) the memory model we hope to have for the language itself, (b.) compiler optimizations today or tomorrow, or (c.) unsafe code people write in practice today or tomorrow?

What is an example of a code transformation one might consider (for either compiler optimization or unsafe code authors)? Well: in version 1, we have code that looks like:

1
2
3
4
5
6

{
  let inner: &Inner = [...];
  let label: char = inner.payload.label;
  [...]
  println!("dropped handle to {}", label);
}

A reasonable person might say “clearly *inner has to be valid and immutable for its whole scope, and that label has only one use. So we should be able to transform it to this:

1
2
3
4
5

{
  let inner: &Inner = [...];
  [...]
  println!("dropped handle to {}", inner.payload.label);
}

⊕ I tried to use a semi-novel name, specific to the example, because I don’t want to get bogged down debating what optimization we’re discussing, where it lives, and how it justifies its actions. So instead I’m choosing something relatively self-contained, easy-to-explain, and potentially incorrect. Let’s call this the “Label Code Motion” transformation.

If version 1 of the impl Drop for Handle code is okay, then Label Code Motion cannot be applied here. Is that expected?

(Perhaps so; perhaps such code motion must be justified with deeper analysis of the other code I’ve elided here, rather than be justifed by the types alone.)

(As you might have guessed, the whole reason I put in the scribble_then_println! macro was to illustrate that things do go wrong if you apply the Label Code Motion transformation to version 1. So something is wrong; later we will try to establish whether version 1 itself is to blame or the Label Code Motion transformation.)

Nonetheless, let us assume that version 1 is okay, at least going into the discussion of version 2.

(We will revisit the question of the correctness of each variant at the end of the document.)

Version 2: “inner-ref handled out-of-line”

Consider this different factoring of the code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

#[cfg(feature = "version2")]
unsafe fn may_drop_inner<'a>(inner: &'a Inner) {
    let pre_decr = inner.ref_count.fetch_sub(1, Ordering::Release);
    // (at this point, `pre_decr > 1` ==> someone else might be freeing `*inner`)
    if pre_decr == 1 {
        unsafe { dealloc(inner as *const Inner); }
    }
}

#[cfg(feature = "version2")]
impl Drop for Handle {
    fn drop(&mut self) {
        unsafe {
            let inner: &Inner = &*self.ptr;
            let label: char = inner.payload.label;
            may_drop_inner(inner);
            scribble_then_println!("version2 dropped handle to {}", label);
        }
    }
}

Now we have fn may_drop_inner which takes a &'a Inner as a function parameter.

My own intuition about function arguments in such cases is that “obviously” the value *inner has to live as long as the lifetime 'a, and that lifetime “obviously” has to be longer than the invocation of may_drop_inner(inner), right?

The intuition in the previous paragraph implies that Version 2 (“inner-ref handled out-of-line”) is not correct.

So: Is my intuition wrong? (Maybe; we’ll revisit that in a moment.)

If my intuition is right (and thus version 2 is not okay), then: Is the refactoring transformation from version 1 to version 2 wrong? Or does non-okay-ness for version 2 imply that version 1 is also not okay?

Version 3: “atomic-ref handled inline”

Let us consider this variant on version 1.

1
2
3
4
5
6
7
8
9
10
11
12
13

#[cfg(feature = "version3")]
impl Drop for Handle {
    fn drop(&mut self) {
        let ref_count: &AtomicUsize = unsafe { &(*self.ptr).ref_count };
        let label: char = unsafe { (*self.ptr).payload.label };
        let pre_decr = ref_count.fetch_sub(1, Ordering::Release);
        // (at this point, `pre_decr > 1` ==> someone else might be freeing `*self.ptr`)
        if pre_decr == 1 {
            unsafe { dealloc(self.ptr); }
        }
        scribble_then_println!("version3 dropped handle to {}", label);
    }
}

Now, instead of holding a whole &Inner reference, we are holding “just” an &AtomicUsize reference. It still is a pointer to storage that is deallocated on some control-flow paths, but we again are careful to never dereference it after the decrement (because after the decrement, concurrent actors could deallocate *self.ptr).

Clearly if version 1 is okay, then version 3 should be too, right?

But what about the other direction: can version 3 be okay even if version 1 is not okay? Well, notably, now one cannot apply the Label Code Motion optimization: at the point where println! is called, there is no &Inner in lexical scope to pull the label out of. So that may be evidence in favor of permitting version 3 while disallowing versions 1 and 2.

We have one more variant to consider in our matrix.

Version 4: “atomic-ref handled out-of-line”

Now let us take the refactoring we saw when we made version 2, and apply that refactoring to version 3.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

#[cfg(feature = "version4")]
unsafe fn may_drop_owner<'a>(ref_count: &'a AtomicUsize) -> usize {
    let pre_decr = ref_count.fetch_sub(1, Ordering::Release);
    // (at this point, `pre_decr > 1` ==> someone else might be freeing owner of `*ref_count`)
    return pre_decr;
}

#[cfg(feature = "version4")]
impl Drop for Handle {
    fn drop(&mut self) {
        let label: char;
        unsafe {
            let ref_count: &AtomicUsize = &(*self.ptr).ref_count;
            label = (*self.ptr).payload.label;
            let pre_decr = may_drop_owner(ref_count);
            if pre_decr == 1 {
                dealloc(self.ptr);
            }
        }
        scribble_then_println!("version4 dropped handle to {}", label);
    }
}

In short, we took the ref-count decrement and moved it into its own local function.

And we might take the same intuition about function arguments from version 2 and apply it to fn may_drop_owner in version 4: “doesn’t *ref_count have to outlive the lifetime 'a? How can we allow a concurrent actor to free *ref_count here?”

An important insight about version 4: the fn may_drop_owner is nearly a trivial wrapper around AtomicUsize::fetch_sub. So if fn may_drop_owner ends up classified as “not okay”, then under what mental model is AtomicUsize::fetch_sub “okay”? (In other words, how can one write any helper functions if something this simple is not okay.)

Version 5: “out-of-band ref-count”

In the discussion on UCG#252, Ralf pointed out other corner cases one might consider. In particular, so far the cases I’ve shown here have ref-count on the allocated memory block itself. But one can also imagine scenarios where the ref-count is stored elsewhere.

However: I argue that an out-of-band ref-count does not hit the particular problem I am concerned with. Since the ref-count is not stored with the heap-allocated block, we are simply not going to encouter the issue described here.

(Such a system may encounter similar problems when it designs its system for deallocating the out-of-band ref-counts themselves, but without more concrete details about the proposal, it is hard to make suggestions about how to handle it.)

See also: https://github.com/rust-lang/unsafe-code-guidelines/issues/88

Summary of versions

Here’s a matrix summarizing the first four variants I posted above.

What options might we consider

Option 1. Outlaw them all

Preface: This space held my preferred solution when I first started seriously looking at this months ago (but don’t worry, I end up arguing against these “outlaw them all” variants).

When I first looked at this problem, I thought the right answer would be to outlaw all four versions. That is, I thought my intuition about function arguments should extend to all lexical scopes for all &T, and therefore it was simply always wrong to let the ref-count decrement on something else in a &T (whether T is Inner or AtomicUsize) drive the deallocations shown above.

At that time, the “clear” answer to my point of view was that all four cases should instead have been manipulating either a *const Inner or a *const AtomicUsize at all times. (This would trivially disallow the Label Code Motion transformation, since no &Inner would be in scope at the point where label is printed.)

A critical problem with this point of view is that we don’t offer any way for someone to decrement an atomic direectly on a *const AtomicX pointer. Our atomics API for e.g. AtomicUsize::fetch_sub forces the developer to pass in a &AtomicX.

So we have to refine this option in some way. Here are two variants I considered.

1a. Outlaw them all and stick with current atomic API

Since the API forces the developer to pass an &AtomicUsize to fetch_sub: Maybe the developer handles *const AtomicX in their code, but casts it via as back to &AtomicX at the fetch_sub calls? But then can the developer similarly cast the *const Inner to a &Inner and trust that the lifetimes will work out in their favor? (No, that doesn’t work; one ends up in the same morass again.)

In short, I don’t think this option is reasonable. Even if we could come up with a semantics for what people are supposed to do, I think it will be too confusing for people to understand what the ref-to-ptr and ptr-to-ref casts in each spot convey.

1b. Outlaw them all and put in alternative atomic API

Why not have variants of AtomicUsize::fetch_sub and similar atomic methods that take a *const AtomicUsize as their formal parameter? If the standard library provided that, then the developer could be advised to only manipulate *const Inner or *const AtomicUsize in code like this, and they would be able to pass *const AtomicUsize to the relevant methods.

There’s a host of answers as to why we wouldn’t want to do this:

• *const-pointers are hard to work with (e.g. we don’t support *const self-methods);
• it would force us to duplicate a large portion of the API surface and maintain it,
• we would have to bikeshed whether these methods take *const or *mut, and
• perhaps most importantly, it would be a ongoing source for confusion with users as to when they should use the &AtomicX methods versus when they should use the *const AtomicX methods, (note in particular that we almost certainly wouldn’t want to deprecate the &AtomicX variants in favor of the *const AtomicX variants).

So: Let us at least consider other options.

Option 2. Allow them all

I wasn’t going to talk about this option when I started writing, but after taking the time to develop the matrix of variants, I figure its worth at least talking about it.

If we were to allow all four variants, then the my own mental model about &T would be incorrect. The Label Code Motion optimization could not be applied via type-based reasoning; thus, people (or compilers) who used such reasoning to justify such transformations would be performing an incorrect transformation.

When I first started writing this section, I wanted to include this assertion:

I suspect very little Rust code used in production today would actually fall into that category of “unsound because incorrect reasoning was used to justify this code”

But I don’t know if I believe that assertion anymore. Use-after-free bugs happen. Code like the below (the result from applying Label Code Motion) is going to happen (and, as you might guess, you get a scribbled label due to the use-after-free here). But there’s nothing in the type structure of how Inner holds its label field that would give you a hint that inner.payload.label could be invalid there.

1
2
3
4
5
6
7
8
9
10
11

#[cfg(feature = "locomotion")]
impl Drop for Handle {
    fn drop(&mut self) {
        let inner: &Inner = unsafe { &(*self.ptr) };
        let pre_decr = inner.ref_count.fetch_sub(1, Ordering::Release);
        if pre_decr == 1 {
            unsafe { dealloc(inner as *const Inner); }
        }
        scribble_then_println!("locomotion dropped handle to {}", inner.payload.label);
    }
}

I do assert that the mental model for what &T means becomes “quite subtle” (at best) if we allow all four variants.

I would like to do better.

Option 3. Function boundaries are special

I put this in here because I figured there was some reason that UCG#252 was focused on function arguments in particular. That is also why I have the in-line vs out-of-line distiction.

I think the heart of this would be something saying that &'a T where 'a is a lifetime parameter to the function (or any lifetime that outlives the function, really) gets treated specially: The compiler gets to assume that in such cases, the T will not be deallocated via an alias.

For the matrix, here’s the outcome of this option

This isn’t really satisfying, because it isn’t actually coherent.

As discussed in the text below the presentation of version 4, we need to have some way to at least designate version4 as “okay”, even if its via some opt-in attribute or special casing in the compiler. If we cannot do that, then we might as well go back to option 1 (outlawing them all).

In short: There’s a reason UCG#252 is not trivial to resolve: in practice, we just cannot rule out all of the out-of-line column.

Option 4. Put deallocation on same footing as mutation

This option comes from a UCG#252 comment. As RalfJung put it:

this is the key new property that unsafe code would get with your proposal: if you are allowed to use a pointer for writes to some memory, you are also allowed to deallocate that memory.

This is the proposal I put at the outset of this document.

But what does it imply for our examples?

interpretation 4A: focus on Atomics alone

The proposal might be interpreted to mean:

if you hold a &A, where A is UnsafeCell<U> or some simple (e.g. #[repr(transparent)]) wrapper around UnsafeCell<U> that merely adds alignment constraints (like AtomicUsize) then you must assume that aliasing accesses could deallocate the UnsafeCell<U>.

The main goal of this interpretation is to allow people to usually use their existing mental model of &T: if you have a &T, then in almost all cases you know its dereferencable for the entirety of its lexical scope. The only exception is when T itself is, in spirit, itself an UnsafeCell. Such cases are already a warning flag for developers: “here be dragons”, and we would just be adding a new wrinkle to the dragon’s scaly skin.

For the matrix, here’s the outcome of this interpetation

(again, “not okay” means it is unsound to write deallocation code in this manner.)

interpretation 4B: Blast radius includes Buddy fields

The proposal might be interpreted to mean:

if you hold an &T where T has any UnsafeCell<U> within it, then you must assume that aliasing accesses could deallocate the whole T.

In other words, sibling fields like .label in our example above would be deemed as potential casualties of a potential deallocation, and thus the Label Code Motion optimization

I think this ends up meaning the same thing as “Option 2: Allow them all”, except that we could at least say that my naive mental model about &T is applicable when T holds no UnsafeCode<U>

For the matrix, here’s the outcome of this interpetation

(This wouldn’t be the end of the world, and in fact, it might be our only real choice, depending on how much code in the wild we might expect to be broken by interpretation A (“focus on Atomics”). In particular, Interpretation 4B would classify the Arc code in the standard library as “okay”.)

Option 5. Add marker to indicate that struct may be deallocated in a volatile manner.

This is a refinement of option 4.

Essentially: If there is too much fallout from following option 4 (variants 4A or 4B, either applies), then we could narrow the focus of the change by having type definitions declare that they have this behavior.

Under this option, a developer of our ref-counted type would be expected to state that this type is deallocated via a protocol that can fire at times unpredictable to the compiler.

This opt-in could take the form of an attribute on the struct Inner definition. Or perhaps it would be a trait that Inner implements. Or maybe its a wrapper type, similar to UnsafeCell<T>, that would wrap around the AtomicUsize.

In any case, that would then be used as a marker telling the compiler whether the “Label Code Motion” optimization applies.

(Another way of seeing this: instead of inferring from the presence of UnsafeCell that such deallocation might happen, we would be adding an alternative marker that developers would have to know about and use correctly if they are implementing a memory-management scheme like this.)

Conclusion

This is all a bunch of notes I wrote down to prepare for a language team meeting. And at some point I wanted them to have executable code. (I use a literate programming system, tango, to write blog posts like this.)

And at some point after that, I realized this document was too long for the language team meeting.

So it became a blog post.

It deliberately does not have a conclusion, at this point. I thought I would come to one over the course of writing it, but all I’ve determined so far is that I need to come up with a more succinct write-up of the problem.

Appendix

Below is some code to just try driving the above things, to test them out.

1
2
3
4
5
6
7
8

fn main() {
    let mut payload = [0u8; 128];
    let source = "Hello World".as_bytes();
    (&mut payload[..source.len()]).copy_from_slice(source);
    let h1 = Handle::new('h', payload);
    let _h2 = h1.clone();
    scribble_then_println!("h1 ref_count: {}", unsafe { (*h1.ptr).ref_count.load(Ordering::SeqCst) });
}

Hey there again!

I have been pretty busy with Rust compiler tasks, so no there has not been a blog post for a few months. But now, here we are!

While I was working some particularly nasty bugs recently, I put a lot of effort into bug minimization: the process of taking a large test case and finding ways to remove code from the test while preserving the test’s ability to expose the same bug.

As I worked, I realized that I was following a somewhat mechanical process. I was using regular patterns for code transformation. At some point, I said “you know, I’m not sure if everyone is aware of these patterns. I only remember some of them while I am in the middle of a bug minimization session.”

Since the Rust compiler team always wants to help newcomers learn ways they can contribute to the project, I realized this could be the ideal subject for a blog post.

And, to try to keep things concrete, I’m going to try to drive the presentation by showing an actual bug being minimized. (Or rather, I’m going to recreate the minimization that I already did. So if it seems like I am rather conveniently picking transformations that always seem to work: You’re right! I’m cheating and not telling you about all the false paths I went down during my first minimization odyssey on this bug.

The Rust specifics

Oh, and one other thing: A lot of the ideas here are applicable to many languages, and so its possible readers have already heard about them or discovered them independently.

However, a few of the techniques leverage features that are not universal to languages outside of Rust.

Specifically:

• “cfgmenting” makes use of Rust’s attribute system to remove items in a lightway fashion.

• “loopification” makes use of the fact that the Rust allows the divergent expression loop { } to be assigned any type.

• “loopification” via pretty-printer leverages the compiler’s (unstable) ability to inject loop { } for all function bodies.

• Bisecting the module tree makes use of Rust’s support for a mix of inline and out-of-line modules to allow one to quickly swap code in and out.

So, without further ado: here is the odyssey of my minimization of rust-lang/rust#65774

Philosophical meandering

What does “minimal” mean anyway?

The objective of a “minimal” test case could mean minimize lines of code; or the number of characters. It is also good to minimize the number of source files: one file, cut-and-pastable into play.rust-lang.org, is especially desirable. ⊕ One could even argue that the number of nodes in the abstract syntax tree is a better metric to use than text-oriented metrics when minimizing.

Minimizing the source test in this way can yield a good candidate for a regression test to add to the compiler test suite when the bug is (hopefully) eventually fixed.

But those syntactic metrics of minimality overlook something: My own end goal when minimizing the code for a bug is a better understanding of the bug: a better understanding for myself, and for other developers who are reading the test case.

⊕ I am writing this post from the viewpoint of a rustc developer. So I may have a slightly skewed view on what is useful or minimal. And for such understanding, there are other metrics to keep in mind:

• Minimize the amount of time the compiler runs before it hits the bug.

  Minimizing the compiler’s execution time before failure serves two purposes:

  1. It makes every future run of this test faster, which can accelerate your pace of minimization.

  2. When the rustc developers themselves examine the behavior of the compiler on the test, they will be grateful to have a shorter execution trace of the compiler to analyze.

  (Such time reduction often occurs anyway when you remove lines of code and/or dependencies; I just want to point out that it has value in its own right.)

  ⊕ As a concrete example of how reducing imports may help expose a bug’s root cause: some people will have a bug that occurs with some combination of #[derive(Debug)] and a call to format! or write, and the test showing the problem may be only a few lines long. But it hides a lot of complexity behind those uses of derive, macros, and std::io trait machinery; a longer test that defines its own trait, a local impl of that trait, and a small function illustrating the same bug, may make the bug more immediately apparent to a rustc developer.

• Minimize dependencies: reduce the number of language features in use and the number of imports your test uses from other crates (including the std library!).

  This can help expose the essential cause of the bug, potentially making it immediately apparent what is occurring.

• Minimize your jumping around in the source code.

  If you can fit all the code needed to recreate the bug onto your monitor screen at once, by any means necessary, that is a serious accomplishment. Less time spent scrolling through a file or switching editor windows is more time you can spend thinking about the bug itself.

To be clear: Often a minimum amount of code needed for understanding does correlate with a minimum amount of code needed for reproduction. (This explains why using lines-of-code or the size of the syntax tree as a metric can be useful when reporting a bug.)

The over-arching goal in minification is to remove all distractions: to reduce the problematic input (in this case, Rust source code) to its essence: a minimal amount necessary to understand the problem.

Why not build it up from scratch?

Its worth pointing out that at some point, maybe even at the outset, you may have a sufficiently rich understanding of the bug that you can go straight to building up a minimal example from scratch. And that’s great , go for it!

However, this post is dedicated to the problem of what you can do when you haven’t hit that level of understanding. When you’re looking at a set of files that make up over 90,000 lines of code, you want a set of semi-mechanical techniques to take that test input and strip it down to its essence.

To be honest, the most effective methodology is going to use a blend of build-up and tear-down. As you are tearing down the 90,000 lines of code, there should be a voice in the back of your head asking “can we now try what we’ve learned to try to build up a minimal example?”

Assumptions and Conventions

⊕ Some of the techniques may also be applicable to cases where the compiler is accepting code that it should be rejecting; but I am little wary of advertising these tools for use in that context, which is why you’re reading this in the margin and not the main text. The tests I am talking about minimizing in this post are cases where the compiler itself fails in some way: an ICE, a link failure, or rejecting code that it should be accepting. Bugs that are witnessed by actually running the generated code are, for the most part, not covered by the patterns here. In particular: many of the patterns presented here rely on making semantic changes to the input: changing values, or replacing complex expressions with something trivial like loop { }.

I named each transformation, usually with absurd made-up words like “unusedification”. They are all in explicit quotation marks to try to make it clear that I am speaking nonsense, deliberately.

⊕ I had originally planned to structure this post so that all transformations for a given theme would be present together, so that you’d see all the transformations for that theme at once. But as I wrote, it became clear over the course of actually doing a reduction, then we often bounce around between transformations, and it usually does not end up being nicely grouped with all transformations for one theme colocated in time. So rather than using that hierarchical presentation, I am instead just going to try to mention the grouping by marking each as transformation being part of a “theme”. Several of the transformations serve similar goals, like “delete the unnecessary”. I have attempted to categorize the different tranformations according to what purpose they serve in the minimization process. Over the course of documenting these transformations, I identified the following themes:

• Simplify Workflow: Make your own life easier.
  • Enable Incremental Steps
• Delete the Unnecessary: Remove items not related to bug!
• Identify the Unnecessary: Eliminate accidental necessity.
• Trivialize Content: Turn complex expressions to trivial ones.
• Eliminate Coupling: Break links between items.

Notes on Notation

In this post, I follow typical notation by using ... as a placeholder for (possibly relevant) code that will be kept approximately the same (modulo mechanical updates like alpha-renaming) by a transformation. However, I also use the non-standard notation of ---- for irrelevant code that removed via a transformation. This is meant to draw attention to the distinct kinds of code, so that you can more easily tell which code is being removed by a particular transformation.

Sometimes I will show an explicit regexp that I am feeding to a tool to do the transformation, but usually I will stick to informal patterns with the aforementioned ... and ---- placeholders.

When a given item or expression can appear within the context of a middle of a sequence (e.g. consider { ... THING ... }), I often use the standard shorthand of just writing { THING ... } or { ... THING }, to simplify the textual presentation and focus attention on the transformation itself.

⊕ theme: Simplify Workflow

Record your steps

Before you do any reduction, move the test case into its own local git repository (or whatever versioning tool you prefer: SVN, RCS, etc). Being able to backtrack through previous reduction steps is an incredibly important option when doing this kind of of work.

⊕ theme: Simplify Workflow

Continuously test the test.

Finally: A crucial part of reduction is to continuously double-check that the bug still reproduces after every reduction step. I’ll show the command line invocation on occasion, but not every command line build invocation (the output is usually the same on every run, so it would just clog up the flow of the presentation here). But even though I don’t show every invocation, you can believe that I was doing the runs. (And in the cases where I tried to skip doing a run, I usually regretted it and had to backtrace my steps to the state before an attempted reduction.)

Make it easy to do your runs. Use your IDE. I use emacs, so I make a M-x compile invocation that runs the compiler with the right arguments, and then you can hit g in the *compilation* buffer to re-run the compile.

The test case

As I mentioned at the start: The presentation here will be driven by referencing a concrete test case that I reduced recently: we have been given a crate graph, and we can observe a bug when we build as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

% ( cd tock/boards/arty-e21/ && \
    RUSTFLAGS="-C link-arg=-Tlayout.ld" \
    cargo build --target riscv32imac-unknown-none-elf )
   Compiling tock-registers v0.4.0 (/Users/felixklock/Dev/Mozilla/issue65774/demo-minimization/tock/libraries/tock-register-interface)
   Compiling tock-cells v0.1.0 (/Users/felixklock/Dev/Mozilla/issue65774/demo-minimization/tock/libraries/tock-cells)
   Compiling tock_rt0 v0.1.0 (/Users/felixklock/Dev/Mozilla/issue65774/demo-minimization/tock/libraries/tock-rt0)
   Compiling enum_primitive v0.1.0 (/Users/felixklock/Dev/Mozilla/issue65774/demo-minimization/tock/libraries/enum_primitive)
   Compiling kernel v0.1.0 (/Users/felixklock/Dev/Mozilla/issue65774/demo-minimization/tock/kernel)
   Compiling riscv-csr v0.1.0 (/Users/felixklock/Dev/Mozilla/issue65774/demo-minimization/tock/libraries/riscv-csr)
   Compiling rv32i v0.1.0 (/Users/felixklock/Dev/Mozilla/issue65774/demo-minimization/tock/arch/rv32i)
   Compiling capsules v0.1.0 (/Users/felixklock/Dev/Mozilla/issue65774/demo-minimization/tock/capsules)
   Compiling sifive v0.1.0 (/Users/felixklock/Dev/Mozilla/issue65774/demo-minimization/tock/chips/sifive)
   Compiling arty_e21 v0.1.0 (/Users/felixklock/Dev/Mozilla/issue65774/demo-minimization/tock/chips/arty_e21)
   Compiling components v0.1.0 (/Users/felixklock/Dev/Mozilla/issue65774/demo-minimization/tock/boards/components)
   Compiling arty-e21 v0.1.0 (/Users/felixklock/Dev/Mozilla/issue65774/demo-minimization/tock/boards/arty-e21)
error: internal compiler error: src/librustc/traits/codegen/mod.rs:127: Encountered errors `[FulfillmentError(Obligation(predicate=Binder(TraitPredicate(<() as core::fmt::Display>)), depth=1),Unimplemented)]` resolving bounds after type-checking

In reality the command line was a little more complicated:

1
2
3
4
5

% ( cd tock/boards/arty-e21/ && \
    RUSTFLAGS="-C link-arg=-Tlayout.ld -C linker=rust-lld -C linker-flavor=ld.lld -C relocation-model=dynamic-no-pic -C link-arg=-zmax-page-size=512" \
    cargo build --target riscv32imac-unknown-none-elf   )
...
error: internal compiler error: src/librustc/traits/codegen/mod.rs:127: Encountered errors `[FulfillmentError(Obligation(predicate=Binder(TraitPredicate(<() as core::fmt::Display>)), depth=1),Unimplemented)]` resolving bounds after type-checking

⊕ An eagle-eyed reader might look at that ICE message and immediately consider grepping the source code, such as uses of the trait bound core::fmt::Display. A fine idea, but intuitive jumping to an answer is not what I’m focusing on here.

but for the purposes of the experiments presented in this blog post I will use the simplified invocation. (The difference only matters once we hit cases where the bug goes away.)

Remember that 90,000 lines of code number that I mentioned a few paragraphs ago? It didn’t come out of nowhere:

1
2
3

% find tock/ -name *.rs | xargs wc
...
   91958  317229 3150451 total

So this is an excellent example of an input that we really want to reduce down to something smaller.

⊕ theme: Simplify Workflow

Tactic: Reduce the driving code first

We have many input files, and they make up a crate graph with at least 13 crates in it.

We know from our command line invocation that it is the build of boards/arty-e21 that is exposing the ICE. It would be good to remove unnecessary crates from the crate graph for boards/arty-e21. But to do that most effectively, we need to first discard as many imports as possible from boards/arty-e21, and then work our way backwards through the dependency chain.

So, will start by reducing the boards/arty-e21 crate to the minimum necessary to reproduce the bug.

This single crate is a much easier thing to work with:

1
2
3
4
5
6

% find tock/boards/arty-e21/ -name '*.rs' | xargs wc
       4       6     124 tock/boards/arty-e21//build.rs
      27      68     565 tock/boards/arty-e21//src/timer_test.rs
      40     101     959 tock/boards/arty-e21//src/io.rs
     279     667    9135 tock/boards/arty-e21//src/main.rs
     350     842   10783 total

However, we still want to reduce it as much as we can: Every import we can remove from arty-e21 represents swaths of code that we might be able to remove in the rest of the crate graph.

⊕ theme: Simplify Workflow

Technique: “mod-inlining”

This technique may surprise some: I like to reduce the number of files involved early in the game, if I can. An easy way to do this is to replace the out-of-line mod foo; item with its inline counterpart: mod foo { ... }. Here the out-of-line module file content has been cut-and-pasted into an inline mod item.

Strictly speaking, moving from mod foo; to mod foo { ... }does not reduce the input: You still have all the same content that you started with, the compiler has to do the same amount of work, et cetera.

However, I still prefer to do it, because I find that it helps with later reduction steps if I can do my transformations on a single file.

There two techniques I use for “mod-inlining”:

1. Manually cut-and-paste: take each instance of mod foo; in the root file, and find that module’s contents. Then replace the ; with { and }, and finally paste the contents in between the curly brackets you just added. You don’t even have to re-indent ⊕Its not like this is Python! it if you don’t want to.

   For a small module tree, such as we find in boards/arty-e21, manually cut-and-pasting is entirely reasonable. But if you have a large module tree, with many directories and files, it can become tedious and error-prone.

   ⊕ Warning: this will not only expand the module tree: it will also expand all the macro invocations, including #[derive] attributes. This can be a bit overwhelming. (I am continually tempted to add a new unstable --pretty variant that just expands the module tree but does not expand the macros otherwise.)

2. Alternative: use rustc to expand the module-tree. You can add --verbose to the cargo build invocation to see the actual rustc command line invocation cargo is using, and then you can add -Z unstable-options --pretty=expanded to that rustc invocation, piping the output to a new rust source

I will show a concrete example of using rustc in this way later in the blog post. But for now I will just manually cut-and-paste the contents of boards/arty-e21/src/timer_test.rs and boards/arty-e21/src/io.rs into main.rs

After doing the inline, make sure the bug reproduces.

It is up to you whether you want to delete the out-of-line module files. Today, Rust does not treat their presence as some sort of conflict with the inline definition ⊕Checking for unused out-of-line module files might be a nice lint for someone to add to rustc . You will see later in the post cases where keeping them around on the file-system can be useful. But for the case of boards/arty-e21, I will go ahead and delete them.

1
2
3
4

% find tock/boards/arty-e21/ -name *.rs | xargs wc
       4       6     124 tock/boards/arty-e21//build.rs
     348     840   10665 tock/boards/arty-e21//src/main.rs
     352     846   10789 total

As expected, this didn’t reduce the line count. But it did make it so we can work with a single file at the leaf. That simplifies my own workflow.

⊕ theme: Delete the Unnecessary

Tactic: “Decommentification”

This may be obvious, but it is a step that I often forget to do at the outset of test reduction, even though it is easy and pays off handsomely.

Comments in code are typically there to explain or justify some detail of the implementation. When diagnosing compiler bugs, the purpose of the original code is usually not relevant. You are better off increasing the number of lines of actual code that you can fit on your screen at once.

In short, the transformation looks like this:

1

^        //...$

to <empty-string>

In this case, I have used ^ and $ as markers for the beginning and end of lines (just like in regexps).

Or, as an Emacs M-x query-replace-regexp input: ^ *//.*^J* ⊕The ^J there is a carriage-return inserted via M-x quoted-insert (aka C-q) in Emacs. (and empty string as replacement text).

⊕ Why I use M-x query-replace-regexp: it previews the matches, I verify a few by eye, and then hit ! to do all the remaining replacements.

Another related transformation: get rid of the other blank lines that are not preceded by comments. I leave that regexp as an exercise for the reader.

In the case of boards/arty-e21, this got rid of 53 lines of comments (and blank lines succeeding them):

1
2

% wc tock/boards/arty-e21/src/main.rs
     295     575    8551 tock/boards/arty-e21/src/main.rs

⊕ theme: Trivialize Content

Tactic: “Body Loopification”

⊕ see also: “none-defaulting” “Loopification” removes the body of a given function or method item, replacing it with loop { }.

Change:

1

fn foo(...) -> ReturnType { ---- }

to:

1

fn foo(...) -> ReturnType { loop { } }

⊕ You might choose to use alternatives like unimplemented!(). I personally like loop { } because its something you almost never see in other people’s code, so its easy to search for and be pretty certain that it was introduced as part of reduction. Also loop { } relies on less machinery from the core library than unimplemented! does. The use of loop { } is deliberate: since loop { } is known by the compiler to diverge, it can be assigned any type at all. So this transformation can be performed blindly.

• Note that this does not work for const fn; the compiler currently rejects const fn foo() { loop { } } as invaild syntax.

• Also, it generally will not work for impl Trait in return position: the compiler needs a concrete type to assign to the impl Trait, and loop { } will not suffice for that unless you ascribed it a non-! type in some manner.

When it comes to replacing a function body with loop { }, you may be able to get help from your IDE. ⊕ More specifically, I have used Emacs to define a keyboard macro (via C-x () that: 1. searches for the next occurrence of fn, 2. searches forward from there for the first {, which often (but not always) corresponds to the start of the function’s body, 3. runs M-x kill-sexp to delete the { ---- } and 4. types in { loop { } }, replacing the method body. This is incredibly satisfying to watch in action. In my own case, I have often used Emacs M-x kill-sexp to delete the { ---- } block in fn foo { ---- }

If you want to take a risk, you can ask rustc to do the replacement of fn bodies with loop { } for you as part of pretty-printing via the -Z unpretty=everybody_loops flag. I’ll speak more on that later.

Once the body has been removed, you can optionally replace all of the formal parameters with _:

Change:

1

fn foo(a: A, b: B, ...) -> ReturnType { loop { } }

to:

1

fn foo(_: A, _: B, ...) -> ReturnType { loop { } }

⊕ These explicit marks can be useful as hints for future reduction steps; see “genertrification”) This is essentially a special case of “unusedification”; we do not delete the parameters outright yet (see “param-elimination” for that), but we mark them as completely useless by writing them as _: ParamType.

In the case of board/argy-e21/main.rs, there was one const fn, and you cannot “loopify” that. For the other fn’s, I was able to replace all but the last fn body with loop { }. (Replacing the last one made the bug go away.)

• In practice, the ICE diangostic often gives me some hint about which fn is the problem, and therefore I can blindly replace the other fn bodies with loop { }.

• But even without such a hint, once can often bisect over the set of fn’s to try to identify a maximal subset that can have their bodies replaced with loop { }. We will talk more about how to do such bisection later.

1
2

% wc tock/boards/arty-e21/src/main.rs
     262     523    7584 tock/boards/arty-e21/src/main.rs

(Okay, only removing 33 lines of code might not be very impressive. The technique will pay off more as we move further along.)

⊕ theme: Trivialize Content

Tactic: “Expr-elimination”

Even if you were not able to replace a whole fn body with loop { }, you can usually simplify the fn bodies that remain now.

⊕ themes: Trivialize Content, Identify the Unnecessary

Technique: Suffix-bisection

In this case, I recommend a form of bisection where you first comment out the bottom half of the original fn-body, and see if the problem reproduces.

• Why comment out the latter half? Well, if you comment out the top half, you’re almost certainly going to comment out let-bindings that the bottom half relies on. The top half very rarely relies on items in the bottom half.

• If the fn in question has a return type, then you can usually use a loop { } as a tail-expression at the end of the fn-body to satisfy the return-type; a natural variant of “loopification”.

If it does reproduce without the bottom half, then you can delete that half, and recursively process the top half.

If it doesn’t reproduce, then the bug relies on something in the bottom half. If you’re lucky, it only relies on stuff in the latter half, and you can try deleting the top half now. But even if you’re “unlucky” and the bottom half relies on stuff in the top half, you leave the top half in place (at least for now) and recursively process the bottom half, narrowing the code until you can identify a single statement whose presence or absence is the line between triggering the ICE or not.

If the fn is too large for fit on one screen, then I use M-x forward-sexp to identify the start and end of the fn-body. Then I jump to a line in the middle, and search around for the start of a statment in the body, and then comment out everything from that statement forward in /* ... */

In the case of boards/arty-e21/src/main.rs, this meant commenting out from line 191 through 263.

1
2
3
4
5
6

/*
    let gpio_pins = static_init!(
    ...

    board_kernel.kernel_loop(&artye21, chip, None, &main_loop_cap);
*/

And, “darn”: It didn’t work. The bug disappeared. But: This is okay! We still have learned something: Something in the latter half is causing the bug. So bisect that latter half (again, favoring commenting out second halves).

Eventually, by recursively processing the suffix statements, we identify that the bug does not reproduce with this code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

{
    ...
/*
    kernel::procs::load_processes(
        board_kernel,
        chip,
        &_sapps as *const u8,
        &mut APP_MEMORY,
        &mut PROCESSES,
        FAULT_RESPONSE,
        &process_mgmt_cap,
    );

    board_kernel.kernel_loop(&artye21, chip, None, &main_loop_cap);
*/
}

but does reproduce with this code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

{
    ...
    kernel::procs::load_processes(
        board_kernel,
        chip,
        &_sapps as *const u8,
        &mut APP_MEMORY,
        &mut PROCESSES,
        FAULT_RESPONSE,
        &process_mgmt_cap,
    );

/*
    board_kernel.kernel_loop(&artye21, chip, None, &main_loop_cap);
*/
}

So, now we have identified a function call to load_processes that seems to be intimately related to the bug.

Unfortunately, load_processes is an item defined in an external crate. (We will deal with that eventually.)

Now that we’ve identified this function call to load_processes as part of the cause of the bug, the new goal is to simplify the earlier part of the function to the bare minimum necessary to support this function call.

⊕ In all of these cases, if an otherwise unused statement or expression influences the type-inference for the body, you may need to keep it around, or some variant of it.

• Get rid of all non-binding statements. (We should not need any side-effecting computations to reproduce the bug.)

• Initialize things to their default values (see “Defaultification” below).

• Eliminate unused lets (see “Unusedification” below).

After applying those steps repeatedly, and further “decommentification”, we are left with this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

#[no_mangle]
pub unsafe fn reset_handler() {
    let chip = static_init!(arty_e21::chip::ArtyExx, arty_e21::chip::ArtyExx::new());
    let process_mgmt_cap = create_capability!(capabilities::ProcessManagementCapability);
    let board_kernel = static_init!(kernel::Kernel, kernel::Kernel::new(&PROCESSES));

    kernel::procs::load_processes(
        board_kernel,
        chip,
        &0u8 as *const u8,
        &mut [0; 8192], // APP_MEMORY,
        &mut PROCESSES,
        kernel::procs::FaultResponse::Panic,
        &process_mgmt_cap,
    );

}

Just checking: the bug still reproduces, and we now have:

1
2

% wc tock/boards/arty-e21/src/main.rs
     111     289    2807 tock/boards/arty-e21/src/main.rs

Its not 100% minimized yet, but its about as far as we can go in changes to fn reset_handler without making changes to crates upstream in dependency graph.

⊕ theme: Delete the Unnecessary

Tactic: “Unusedification”

Another way to remove distractions: remove them.

1

non_pub_locally_unused_item(----) { ---- }

to <empty-string>

At this point, after our initial round of “loopification”, we should have a lot of unused stuff: variables, struct fields, imports, etc. And even better, the compiler is probably already telling you about all of them!

In the specific case of arty-e21, I am currently seeing 25 warnings: 13 unused import warnings, 4 unused variable warnings, 2 static item is never used warnings, 1 constant item is never used warning, and 5 field is never used warnings.

If you’re doing the compilation runs in your IDE, then you should be able to just jump to each unused item and remove it in some way.

• In the case of fn parameters, you can replace it with _ as previously discussed.
• ADT contents (struct and enum fields) may require more subtlety; see “ADT-reduction” below. For now, you can prefix their names with _.
• Warning: Items with attributes like #[no_mangle] or #[link_section] may also want special treatment: you may be able to remove them without causing the bug to go away, but once the bug does go away, their absence may cause a confusing linker error.

Make sure to check periodically that the bug still reproduces (Sometimes supposedly unused things still matter)!

This got boards/arty-e21 down to 95 lines:

1
2

% wc tock/boards/arty-e21/src/main.rs
      95     254    2382 tock/boards/arty-e21/src/main.rs

Again, not so impressive yet. But as with optimizations, sometimes the effect of these techniques only becomes apparent when they are combined together.

⊕ theme: Delete the Unnecessary

Technique: “Cfgments”

As a note: As a test run, you can easilly preview the effects of “unusedification” and “demodulification” (discussed below) transformations without actually deleting code. (After all, you may quickly discover that you need to put it back.) One classic approach for this is a comment block:

1
2
3

/*
non_pub_locally_unused_item(----) { ---- }
*/

But it not always easy to toggle such comments on and off, since you need add and remove the /* and matching */ each time you want to toggle it. Some IDEs help with this, but I still prefer to use more local changed if I can.

A less used option that is more specific to Rust (and that I use all the time), is to use a #[cfg] attribute to temporarily remove the code:

Change:

1

unused_or_little_used_item { ---- }

to:

1
2

#[cfg(not_now)]
unused_or_little_used_item { ---- }

⊕ No, I do not know how to pronounce “cfgment”; I have only typed it, never uttered it. Iä! Sheol-Nugganoth! I call this “cfgmenting” out code (as opposed to “commenting out code”).

Whether or not you choose to use comments, “cfgments”, or just delete code outright is up to you, (though I do recommend you eventually delete the lines in question, as discussed in “decommentification”).

⊕ theme: Delete the Unnecessary

Tactic: “Demodulification”

In addition to “unusedifying” each identified unused item, you might try this: You may also be lucky enough to be able to just remove whole modules from this crate at this point. If the compiler is telling you that a lot of the items in a given module are unused, maybe you can get rid of the whole module. Go ahead and try it!

There can be a bit of guesswork involved here. For various reasons the compiler’s lints do not identify some definitions as unused even though it may seem obvious that they are not used.

• This can be a consequence of the impls in the source; see “Deimplification” below.

• Also some cases are only identified after doing “depublification” of the contents of such modules, which is also discussed below.

In the case of boards/arty-e21 I was able to identify mod timer_test as entirely unsed.

After “cfgmenting” it out and further “decommentification”, we have this:

1
2

% wc tock/boards/arty-e21/src/main.rs
      73     191    1967 tock/boards/arty-e21/src/main.rs

(For now we have to keep the mod io { ... }; it defines a panic-handler, and we won’t be able to get rid of that until we do more reduction elsewhere.)

⊕ theme: Delete the Unnecessary

Technique: “ADT-reduction”

What more is there to reduce here? Well, there is a struct ArtyE21 all of whose fields are unused: