Tim's journal

I more-or-less finished external crate inference, at least in the local-crate case. I spent most of the time on that fighting with the borrow checker. To add the right directories into the session's list of additional library search path, I wrote some visitor code that looks at all the view_items in the crate. There are two types of view_items: extern mod directives and use directives. For this, I only care about extern mod directives.

The problem is that the visitor wants an @-closure, which can't use borrowed pointers, so accumulating a mutable vector of found extern mod thing seems... hard. I eventually gave up and worked around it with massive amounts of copying.

Once I get some tidy stuff cleaned up, and figure out the copy issue, I'll be ready to check that in and close #5681.

If I can finish #6407 (allowing extern mod directives to name remote packages) and #5683 (the test runner for rustpkg, which I've already started a rudimentary version of) tomorrow, then I'll be on-schedule so far. It might happen.

Back once again from another (shorter) absence -- I had to go down to Olympia to say goodbye to my 11-year-old rabbit, who'd had a stroke and needed to be euthanized. This spring, I'm telling you...

This morning, Ron Garcia came over to the office from UBC to visit us and hear about Rust. It's always good to get a bit of external validation, and it seems like he and our team have at least some research interests in common.

After lunch, I got down to hacking, having previously gotten somewhat derailed by adding a few more tests to #6418, the URL-like package-ID pull request. I hit the "computing fictitious type" ICE that a lot of other people had hit, and so I figured it was time for me to fix it, since I knew it probably related to my recent changes to do with derived errors. Turned out the ICE has to do with erroneous matches on patterns of the form (foo, bar) where the scrutinee doesn't have a tuple type -- this is a common error because of, among other reasons, argument patterns where it should really be &(foo, bar). So I fixed that, but that was all over the weekend.

Today, that patch having landed, I went back to finishing #6418. Partly this required resolving merge conflicts, due to the intervening make-vectors-not-implicitly-copyable change which made PkgIds non-implicitly copyable. Partly, I fixed the ugly hack I was using where output file names would be of the form foo-0-1 instead of foo-0.1, where 0.1 is the version number. I was doing that so that rustc wouldn't drop the portion after the last dot in order to make the output .o filename. But it turns out the correct output file to pass is just foo, since for libraries, rustc pulls the version number out of the metadata (not the filename) anyway.

And so I can get this checked in before I leave the office tonight, I hope (which is a bit of a race against time, since all the bathrooms in the office are out of order).

After taking three days off to fly to Europe, attend a funeral, and fly back, then some time to catch up on the inevitable email and bug-triage backlog, I'm back to working on remote package IDs for rustpkg.

On the plane over the weekend, I was working on getting rustpkg to accept package IDs that resemble URLs. For example: rustpkg build github.com/catamorphism/test-pkg. So, not really a URL, but a URL fragment. rustpkg should do the obvious thing and use git to clone the remote repo into a local directory, then build the local cached copy as usual.

You'd think this wouldn't be hard, but I ran into issues because internally, rustpkg normalizes hyphens into underscores so that all package IDs can be legal Rust identifiers. (Rust identifiers can't contain dashes, but can contain underscores.) But for a remote package, you still need to keep track of the original package ID since that dictates the URL you need to fetch from. I solved this by adding two fields, a local path and a remote path, into rustpkg's internal representation of package IDs, but I'm still not sure this is the right solution. It beats calling a normalize function in an ad hoc way everywhere, as I was doing before, though.

Then the second thing is being consistent about which build artifacts contain the version number as part of their name (for example, the library name does, but the installed executable doesn't...) though I'm not sure why this wasn't coming up before.

And, doubtless, more than that. Details, details...

The past couple days, I've been unsurprisingly not very focused on work; you'll know why if you read my most recent blog post that doesn't have the "research" tag.

But life does go on. Having landed the file searching patch that I talked about in my previous Rust post, I went on to working on rustpkg install. Like a good little software developer, I wrote the tests first, making the actual implementation pretty painless. Right now my tests are failing because apparently, I'm creating directories with the wrong permissions. Although lsing the directories, they look fine, so the Rust permission-checking code I wrote was probably wrong.

That (and a triage meeting) was about all I could manage today.

With rustpkg I'm trying as much as I can to stick to a tests-first/docs-first approach (I suppose strictly speaking, tests-first/docs-second), so this is my pull request for today: adding a draft of a manual (without much actual content, but starting to write things down) and rearranging the test scenarios again to have the directory structure that we agreed on.

I figure when I don't know exactly what I'm doing, writing things down as I go along (rather than waiting until the end to write the docs) helps me see if there's an area I'm still not clear on. I don't know why it's so hard for me to remember that that's the case for me :-)

I'd given feedback on a rustc pull request saying "try to get rid of that copy" in a pull request, but when asked how, I looked at the code and realized it wasn't easy to rewrite without a copy. So that was an hour gone!

Then, back to rustpkg; I started testing it on the test cases I wrote yesterday (just manually, for now, though writing a test runner needs to happen soon) and noticed that I wasn't naming the output executables or libraries correctly. So I added code to synthesize the link crate attributes with the inferred name and version, so you don't have to write it explicitly in the package file. Now, 2 out of 3 of my test cases work.

And finally, to round the day off, removing a few more bad copies.

Next:

• Error out in the case where no lib.rs, main.rs, bench.rs, or test.rs fine is found
  
• Split tests into pass and fail subdirectories (for tests expected to build correctly, and tests expected to fail)
  
• Add a test case to test that a package without a lib, main, bench, or test file fails.
  
• Submit a new pull request.
  

My pull request from yesterday landed, so I'm on to further cleanup. First, I submitted another pull request to use a condition for error handling instead of making a function return an Option. Conditions are supposed to be for rare errors; this particular case was the situation where creating the build directory in the package directory to put the build output in fails. I figured that would be pretty rare. I hadn't really written code with conditions before, but they're about as simple as it gets. Sort of like exceptions, but simpler and implemented in terms of Rust macros.

For my next trick, I wanted to make sure that rustpkg could compile a package that doesn't have a package script. For realism, I decided to see which of the Servo submodules had a relatively simple Makefile, meaning that rustpkg should be able to build it as-is; I picked rust-core-foundation, and am working on getting it to build now.

My titles are un-creative lately.

So remember when I said I'd gotten rust-sdl building with rustpkg? I really... didn't. Or else, I broke a whole lot of code while rebasing. Either way, I spent today and yesterday (when not paying attention to the work week proceedings) getting it to work again. And, just now, I finally did, after learning a valuable lesson about Unix command line argument processing that doubtless I knew in 2004 but didn't know yesterday.

Next, continue cleaning up code so I'm not embarrassed by "oops, how did that get in there?" moments when I submit a pull request.

I said yesterday that I got LLVM to build without a Makefile, but I didn't, really. The Android-related message that I thought was a warning was actually an error, but it took me most of the day to figure out for sure. The cause of the problem? There was code in our Rust wrapper for the LLVM library that was unconditionally initializing the command-line arguments on every call to LLVMRustWriteOutputFile. Since rustpkg may call this function twice -- once to write the executable for the package script, and one to write the executable or library (library, in this case) we're compiling, the error resulted. Cryptic for several hours, to crystal clear within a minute. This wasn't a problem in previous iterations of rustpkg since the change to add some Android-related flags was made recently.

Fixing it was easy, since apparently in LLVM, the flags that say whether a particular command-line flag is toggled are global state, and I submitted a pull request.

But I'm still kicking myself that it took me hours to track this down. Software engineering, folks.

Next, cleaning up the changes I made to rustpkg and rust-sdl (in the latter case, getting it to build with incoming Rust) so I can actually submit pull requests!

The release work continues apace, with just 3 open issues left! I worked on documentation a bit and then when it seemed like there was nothing blocking the release that I could easily finish, went back to one of the bugs that got deferred, #4850. I had done some work on this last week but got over-ambitious in trying to restructure trans, when the urgent thing was to fix the bug.

Once I started over, with the knowledge I gleaned from last week's debugging, it was pretty obvious that the issue was that trans code was assuming that the self argument to a trait reference is always passed by reference. But Rust allows traits to be declared with their self argument passed by value; depending on what type self actually is (and this is usually, but not always, known at compile time, since Rust compiles polymorphism by monomorphization), that means either moving it into the callee, or copying it. So there was some code assuming self was passed by reference, other code assuming it was passed by value, so in some cases with by-value self, the result was generated code that segfaulted.

It seems like the trans code could be better structured as far as how many pieces of code know things about self. But I'm trying to make minimal changes right now, and am just patching up those places as they appear to take into account that the self argument to a method might not always be a pointer. However, I'm worried that the fact that the size of the self argument depends on the monomorphized code for the method is going to cause problems. It hasn't yet, but I'm afraid it will, and that this will mean bigger restructuring is needed for trans.

I'm setting this aside in any case once we tag the release, though, and moving on to focus on rustpkg for a while, so I don't get totally lost in bug-fixing.