Tim's journal

I got rustpkg to use workcache to enough of a degree that one of the existing tests fails because it isn't rebuilding a file when it's not necessary to :-) The old test used "touch" to make a dependency look as if it had changed, then ran rustpkg and checked datestamps. Since workcache works based on a hash of the file contents, the test fails even though it works correctly.

So that's exciting! Despite Graydon explaining it to me, I hadn't quite grokked how workcache works till now. It's a quite neat idea: most build systems (including) make are declarative. You write a whole other program to explain how to build your program (albeit in a very weird programming language, usually). Workcache, and a system called fbuild that it's based on, are imperative. You express your dependencies in your usual programming language, specifying how to determine if a particular target needs to be rebuilt. That made more sense in my head than it did when I tried to write it down just now.

In any case, the missing piece that I filled in today was the freshness map, which is part of the state that you have to pass around for workcache to work. The freshness map is a hash table that maps "kinds" (just strings) onto functions. The "kinds" in this case are "source file", "executable or library file", "URL", and maybe a few other things. The function takes a name (for example, a file path) and a value (for example, a hash of the file's contents) and returns a boolean indicating whether the given name and hash are up-to-date. The "given" name and hash come from the workcache database (which is persistent) and the freshness function is the code that (in this case) reads in the contents of the file, hashes it, and compares the two hashes for equality.

Most of the time that took was understanding the code well enough to write the previous paragraph; once I understood that, the code was quite simple.

Next step is to change the test cases to reflect that rustpkg won't just rebuild if the timestamp changes[*], and make sure they change the contents of the file if they want to test behavior-if-a-file-changes. But I feel like things are moving now.

[*] Or maybe it should, I'm not sure, but in that case the freshness function would have to be changed so that the hash is a hash of both the file's contents and its metadata.

Today, I looked at the metabug for rustpkg and noticed that all of the items marked as "necessary to call rustpkg 'ready for use'" are finished. I added one more that was overlooked, which is workcache support, and after discussion with Graydon, added several more issues. Still, most of those should be pretty easy to fix, so we're very close to the point where we'll want to say to the community "hey, start using this and test!" Some people, of course, already have, such as cmr with rustdoc_ng. Which I'm currently trying to build on my system with rustpkg but am making rustc segfault. Must be Wednesday.

If you drive a car:

• Obey all traffic laws, including posted speed limits, even if you're on a freeway.
  
• If you're near a crosswalk, you should be driving slow enough to stop even for a pedestrian who "jumps out in front of you".
  
• Do not honk at cyclists or pedestrians. In many places, it's illegal to honk where there is no imminent danger. If you are causing the imminent danger, you should stop causing it instead.
  
• Do not shout death threats at cyclists or pedestrians.
  
• Stop at all stop signs.
  
• Always use turn signals before turning or changing lanes.
  
• Know how much space you're legally obligated to give cyclists, and give it to them.
  
• If a cyclist is taking the lane, don't try to run the cyclist off the road. They are doing that because they are safer riding that way and because it's their legal right. Treat a cyclist like any other slow-moving vehicle: pass the cyclist if and when you can do so safely, but otherwise stay behind and leave lots of room.
  
• Before you open your door while parked in a parking space, look outside to make sure no cyclists are passing by.
  
• Don't park in bike lanes.
  
• Don't drive drunk or while under the influence of other drugs that impair coordination.
  
• Don't text while driving.
  
• Don't talk on the phone while driving, not even using a hands-free device; in person, a conversation partner will be aware of your surroundings and can react to them by stopping or slowing down the conversation, whereas someone on the phone can't do that and thus is much more of a distraction.
  
• Before making a right turn (or left turn if you're in a country where you drive on the left), check to make sure you wouldn't be hitting a cyclist to your right (or left) who is riding straight through the intersection and has the right of way.
  

Whether or not you see cyclists riding without helmets, not coming to a complete stop at stop signs, or doing anything else you disapprove of, with power comes responsibility and it's your responsibility as an operator of a heavy, dangerous machine to prevent accidents. If everybody followed the above advice, we'd see a lot fewer fatal accidents involving pedestrians and cyclists.

Just as most advice about preventing rape seems to be targeted at women -- the group more likely to be a victim and less likely to be a perpetrator -- there seems to be vastly more advice about traffic safety targeted at cyclists, along with the assumption that motor vehicle operators know everything and can do no wrong. And just as conventional rape prevention advice perpetuates rape culture, the emphasis on cyclists perpetuates a culture where hit-and-run accidents are common and law enforcement (at least against white drivers who break the law) is rare.

The reality is that "safety" tips aimed at cyclists don't make anyone safer -- they just discourage cycling by making it seem dangerous, as well as empowering motorists to terrorize cyclists ("it's her fault I ran her over, she wasn't wearing a helmet"). And discouraging cycling actually does make it more dangerous to be a cyclist -- the fewer cyclists are on the road, the less likely it is that motorists will know what to do when they encounter one. Vehicular cycling (that is, riding a bike like a car and obeying traffic laws) does make cycling safer, but such "tips" rarely promote it since it sometimes involves taking the lane where it is legal to do so, and that would inconvenience motorists.

In any case, unless you always follow every item on the above list when you're driving a car, you really have no business criticizing cyclists' behavior. You should be focusing on your own obligation to use your vehicle in a responsible way.

The extern-mod patch finally landed! The last bug: it turns out that something on Mac OS X wasn't right in our run code (that shells out to create a new process), and if you supplied environment variables, it was extending the environment and not overwriting it. On Linux, the code was correct, which meant shell commands failed because among other things, the PATH was getting set to "". I changed my test code that was calling the run code to explicitly fetch and extend the existing environment, and that was the last fix.

The next big thing to finish will be using workcache. I'm realizing that the workcache library itself needs some work before rustpkg can use it. The first thing to do was to change it so that it actually saves and reloads its database to disk, so that you can see changes in between runs of rustpkg, and not just within the same run. I did that, and now it's complaining about something missing from the file when it reads it back in, which is at least progress.

When running up-to-date master on my local machine, I was noticing a segfault while building the docs that obviously didn't happen on the bots, but it seems to be due to the new scheduler and the handling of stack overflows (I git-bisected and saw that the first bad commit was the "Turn on the new runtime" commit). There's a patch in the queue to increase the stack size, so I'm going to go home and hope it'll magically be fixed in the morning.

Finished the big pull request for extending what extern mod can name, and other stuff! That was... way more work than I expected, even when taking into account Hofstadter's Law.

I'd say more, but I want to get out of the office. Next priority is workcache (you know, so rustpkg can actually be a non-ridiculous replacement for make...); I should also be able to get #6409 out of the way in the foreseeable future.

No wait, one more thing! The very next thing I do will to be to get rid of the local_path / remote_path hack. See, right now, in rustpkg, package IDs look something like:

struct PackageId {
    short_name: ~str,
    local_path: LocalPath,
    remote_path: RemotePath,
    version: Version
}

LocalPath and RemotePath are both just newtype'd Paths (where Path is a path on the filesystem -- basically a newtype'd string with some extra operations):

struct LocalPath (Path);
struct RemotePath (Path);

which in turn is special syntax Rust has for something like:

struct LocalPath { localpath: Path }

In any case, I implemented this because a lot of package names have dashes (hyphens) in them, but a Rust identifier can't contain a hyphen. Before #8176, because of extern mod directives, a package name had to be a legal Rust identifier. I implemented a hack in rustpkg that turns '-'s into '_'s, but, so it can still find the original directory on disk, keeps the original name with '-'s around as the RemotePath. Now that extern mods can have a path attached, though, I can just get rid of this special case. Hurrah!

The pull request I landed on Friday, to make more kinds of refspecs work, landed.

I got the whole test suite to pass with my branch-that-took-forever that extends the extern mod syntax! But as I often do, I decided I should address the many NOTEs[*] I'd added to the code before submitting a pull request. There's always the tension between getting it right and getting it in the codebase, but this time I felt like there were too many embarrassing things, tests passing or not.

One of the things is that I changed the PkgId constructor (shockingly, that's for the type that represents package IDs) to take the current directory as an argument, in case you give it something like foo/bar/baz and that turns out to be a git repository and you want to get the version. Well, that was super silly of me, because as with everything in rustpkg, we want to search the RUST_PATH rather than searching in the current directory in some ad hoc way. So I fixed that (I always enjoy getting paid to undo my own work) and now all but one of the rustpkg tests work, but I'll leave that for tomorrow so I can have a victory early in the day.

Submitted a pull request in the waning hours of the day, which wraps up support for tags. rustpkg now knows about tags that aren't versions. I refactored some of the git-fetching code while I was at it, and changed it so that the build output for foo goes in build/foo and not directly in build (this is more consistent, since rustpkg was already doing that for packages with compound names).

I worked more on extended extern mod names, but didn't finish that. Hacking on metadata::{filesearch, loader}: always a good time.

I also found this delightful bug to do with calling run process functions and specifying an environment, though I'm still not convinced it's not due to something weird happening on my machine. So if someone wanted to try reproducing it, that would be awesome.

I fixed the dependency problem, which was trivial to fix:

diff --git a/mk/tests.mk b/mk/tests.mk
index 770e728..c3659e7 100644
--- a/mk/tests.mk
+++ b/mk/tests.mk
@@ -345,7 +345,8 @@ $(3)/stage$(1)/test/rustctest-$(2)$$(X_$(2)):                                       \
 
 $(3)/stage$(1)/test/rustpkgtest-$(2)$$(X_$(2)):                                        \
                $$(RUSTPKG_LIB) $$(RUSTPKG_INPUTS)              \
-               $$(TLIB$(1)_T_$(2)_H_$(3))/$$(CFG_LIBRUSTC_$(2))
+               $$(TLIB$(1)_T_$(2)_H_$(3))/$$(CFG_LIBRUSTC_$(2)) \
+               $$(TBIN$(1)_T_$(2)_H_$(3))/rustpkg$$(X_$(2))
        @$$(call E, compile_and_link: $$@)

See, totally trivial! Um. Workcache changes haven't landed yet, so I didn't work on that. Rather, I spent the rest of the day futzing with directories, again, because I haven't done enough of that, as part of making-general-extern-mods work. Admittedly some of that had to do with the above change -- making the test suite able to run rustpkg from the target directory instead of the host directly. (Cross-compilation thing. Makes perfect sense. Yeah.)

Another recompile is running, but it's time to go home.

I started trying to make rustpkg use workcache, but ran into a problem. So, workcache is a library for caching the results of functions. This is obviously a useful thing if you're trying to avoid recompiling outputs whose inputs haven't changed. To make this work, you pass the closure whose result you want to cache into a special wrapper function, called exec. Since if you really want to avoid re-doing work, you need to save the dependencies you've discovered persistently, exec also takes a "context" argument that includes a pointer to a database, among other things (which is just a file on disk). Also, exec takes a ~ closure, meaning that the closure's environment only has sendable things. You can think of sendable as meaning "no @-boxes", because @-boxed things are only task-local. But if the closure wants to invoke itself recursively, it needs to capture the workcache context inside it. And the workcache context isn't sendable; it contains @-boxes.

rustpkg discovers dependencies as it goes along, doing parsing, so you could start installing one package A, find another package B that needs to be installed, go ahead and install B, then go back to building A. So the natural way to rewrite this code using workcache involves passing around the workcache database to recursive calls. But that's not allowed, since it's not sendable and since the workcache wrapper requires a sendable closure.

Graydon agreed it would be better to rewrite the workcache code so that the context is sendable, so now I'm blocked on him doing that.

In the meantime, though, I also figured out the duplicate package problem from yesterday. It was really silly; I was adding in both the .rust subdirectory of the home directory and of the current working directory -- and all ancestors of the CWD. But what if the home directory is an ancestor of the current working directory (as one would normally expect it to be)? sadtrombone.wav

And in the meantime, finishing extern mod is blocked on fixing the duplicate-package bug, which is easier now that I've found it.

Submitted a pull request today! It's been a while. This one eliminates the dependency some of the rustpkg tests had on remote URLs, specifically github. Since Github tends to go down a lot, that was causing spurious test failures.

To fix it, I made rustpkg handle fetching from a local git repository, and changed the tests to test that instead. If you supply rustpkg with a package ID referring to a github repo that's outside the RUST_PATH, it will clone that into a workspace in the RUST_PATH and build from there.

I can now go home feeling a bit more accomplished.