tim | Recent Entries

With rustpkg I'm trying as much as I can to stick to a tests-first/docs-first approach (I suppose strictly speaking, tests-first/docs-second), so this is my pull request for today: adding a draft of a manual (without much actual content, but starting to write things down) and rearranging the test scenarios again to have the directory structure that we agreed on.

I figure when I don't know exactly what I'm doing, writing things down as I go along (rather than waiting until the end to write the docs) helps me see if there's an area I'm still not clear on. I don't know why it's so hard for me to remember that that's the case for me :-)

I've got a new blog post up at geekfeminism.org about structureless organizations and whether or not they are good for people experiencing marginalization in the tech industry. You can also read my past posts on geekfeminism.org.

Today's work on rustpkg is all in my pending pull request. I wrote out a schedule containing all the significant pieces-of-work remaining on rustpkg that Graydon and I could think of right now, and estimated times for each task (like all software engineers I'm terribly bad at estimating time for things). I'm hoping having everything laid out in a linear fashion will help me stay on-track, though.

I'm also glad to have spent some time writing down hypothetical-rust-package-commands-that-should-work (in a text file in the pull request), even though we can't do automatic testing yet. It's good to know what you're doing before you do it...

As an end-of-the-day small task, I made the typechecker able to survive more errors; generally the typechecker should be able to emit a type error and then keep going (since most people like to see as many errors as possible so they can try to fix them in one go, rather than one at a time) but there are a few cases here and there where it's tricky to set up the right data in various environments for the typechecker to keep going. So there's two less of those cases now.

I'd given feedback on a rustc pull request saying "try to get rid of that copy" in a pull request, but when asked how, I looked at the code and realized it wasn't easy to rewrite without a copy. So that was an hour gone!

Then, back to rustpkg; I started testing it on the test cases I wrote yesterday (just manually, for now, though writing a test runner needs to happen soon) and noticed that I wasn't naming the output executables or libraries correctly. So I added code to synthesize the link crate attributes with the inferred name and version, so you don't have to write it explicitly in the package file. Now, 2 out of 3 of my test cases work.

And finally, to round the day off, removing a few more bad copies.

Next:

Error out in the case where no lib.rs, main.rs, bench.rs, or test.rs fine is found
Split tests into pass and fail subdirectories (for tests expected to build correctly, and tests expected to fail)
Add a test case to test that a package without a lib, main, bench, or test file fails.
Submit a new pull request.

I landed this commit from the end of last week, cleaning up tests and error handling in rustpkg, and I submitted this pull request that allows rustpkg to build executables.

When talking to Graydon about how to do regression tests for rustpkg, I realized I'd been doin' it wrong, at least a bit -- the new policy for rustpkg is that you're going to have to name your crate files main.rs if you want to get an executable and lib.rs if you want to get a library (both can coexist), with additional options for building tests and benchmarks. I'd been allowing an additional option, which is naming your main crate foo.rs for a package named foo, and inferring whether or not it was a library based on the presence or absence of a main function. This was all wrong, so I'll have to undo that work tomorrow. I'm trying to tell myself I'm "exhibiting flexibility and willingness to admit mistakes" rather than "spending all my time writing code that I then delete".

For an easy end-of-the-day task, I decided I'd try to kill off a few bad copies, but that involved rebasing a branch, which involved rebuilding LLVM, which is evidently going to involve deleting my build/ directory and starting from scratch. Which, amusingly, emphasizes the need for a functional rustpkg!

My pull request from yesterday landed, so I'm on to further cleanup. First, I submitted another pull request to use a condition for error handling instead of making a function return an Option. Conditions are supposed to be for rare errors; this particular case was the situation where creating the build directory in the package directory to put the build output in fails. I figured that would be pretty rare. I hadn't really written code with conditions before, but they're about as simple as it gets. Sort of like exceptions, but simpler and implemented in terms of Rust macros.

For my next trick, I wanted to make sure that rustpkg could compile a package that doesn't have a package script. For realism, I decided to see which of the Servo submodules had a relatively simple Makefile, meaning that rustpkg should be able to build it as-is; I picked rust-core-foundation, and am working on getting it to build now.

So I finally submitted a pull request with my initial work on rustpkg. I'm embarrassed at how little this patch actually does; I've tested it with one command, build, on one package, my fork of rust-sdl. The difference now is that as per #5679, rustpkg doesn't require a package to explicitly declare its ID and version number; instead, it just looks for a directory with the name you give it. So if you write rustpkg build rust-sdl, rustpkg will expect the current directory to contain a directory called rust-sdl that contains the package. Likewise, if it was rustpkg build rust-sdl-0.1, it would infer the version to be 0.1 (though I don't think I actually implemented that part yet).

I still have to figure out how to write unit tests for this. Also, rather than trying to do everything at once, I made several of the commands other than build fail with "not yet implemented". rustpkg wasn't in a great state anyway, so I don't think this will make anyone too unhappy. I think it's better to explicitly fail than quietly do something not-quite-right.

Hopefully I can proceed faster now that I'm past the awkward initial "what am I doing?" stage of the project.

My titles are un-creative lately.

So remember when I said I'd gotten rust-sdl building with rustpkg? I really... didn't. Or else, I broke a whole lot of code while rebasing. Either way, I spent today and yesterday (when not paying attention to the work week proceedings) getting it to work again. And, just now, I finally did, after learning a valuable lesson about Unix command line argument processing that doubtless I knew in 2004 but didn't know yesterday.

Next, continue cleaning up code so I'm not embarrassed by "oops, how did that get in there?" moments when I submit a pull request.

Today was the beginning of the research work week here in Vancouver, and I don't know about you, but I can't follow a discussion for 8 hours straight -- even with a break for lunch in the middle. So I tried to listen as well as I could to the parts that overlapped more with my knowledge, and during the rest, hack on stuff and listen with one ear.

I started cleaning up my rustpkg changes in preparation for submitting a pull request. Part of that was adding more documentation and such, part was removing bits of code that aren't needed now that we're using package IDs and not requiring a package script. I'm not sure whether to break existing code that uses rustpkg (not that there's a lot of it yet) or try to be backwards-compatible.

After that, it'll be submitting a pull request for the changes I made to rust-sdl, making the clean command work (since it's annoying to have to manually delete files every time to make sure my changes still worked), and making rust-http-client build as my second test project. And then, of course, making all the other commands work. That's just the beginning.

Figuring out how to write unit tests for rustpkg will be interesting as well. Necessary, but I'm not sure how.

I said yesterday that I got LLVM to build without a Makefile, but I didn't, really. The Android-related message that I thought was a warning was actually an error, but it took me most of the day to figure out for sure. The cause of the problem? There was code in our Rust wrapper for the LLVM library that was unconditionally initializing the command-line arguments on every call to LLVMRustWriteOutputFile. Since rustpkg may call this function twice -- once to write the executable for the package script, and one to write the executable or library (library, in this case) we're compiling, the error resulted. Cryptic for several hours, to crystal clear within a minute. This wasn't a problem in previous iterations of rustpkg since the change to add some Android-related flags was made recently.

Fixing it was easy, since apparently in LLVM, the flags that say whether a particular command-line flag is toggled are global state, and I submitted a pull request.

But I'm still kicking myself that it took me hours to track this down. Software engineering, folks.

Next, cleaning up the changes I made to rustpkg and rust-sdl (in the latter case, getting it to build with incoming Rust) so I can actually submit pull requests!

Rust 0.6 happened!

I set aside the trans bug I was working on before, just like I said I would do, because trans badly needs the Hatchet of Refactoring, but now is not the time.

Instead, I went back to working on rustpkg; the work that needs to be done is now better outlined (by Graydon). In the meantime, though, I'd wanted to make rustpkg able to build an actual, existing package and infer the package name and version from the path name. I succeeded in doing this today, with rust-sdl, except that I got some warnings at the end about the Android back-end that I need to investigate. The notable thing is that rust-sdl previously had a Makefile that renamed a symbol in the existing SDL library before building; I ported that logic into a package script -- a Rust file -- instead, so now it builds without a Makefile.

Of course, one of the things rustpkg needs to do is build packages that don't have a pkg.rs file -- in the current trunk version, it always requires one -- so perhaps I didn't pick the best example package to start with.

The release work continues apace, with just 3 open issues left! I worked on documentation a bit and then when it seemed like there was nothing blocking the release that I could easily finish, went back to one of the bugs that got deferred, #4850. I had done some work on this last week but got over-ambitious in trying to restructure trans, when the urgent thing was to fix the bug.

Once I started over, with the knowledge I gleaned from last week's debugging, it was pretty obvious that the issue was that trans code was assuming that the self argument to a trait reference is always passed by reference. But Rust allows traits to be declared with their self argument passed by value; depending on what type self actually is (and this is usually, but not always, known at compile time, since Rust compiles polymorphism by monomorphization), that means either moving it into the callee, or copying it. So there was some code assuming self was passed by reference, other code assuming it was passed by value, so in some cases with by-value self, the result was generated code that segfaulted.

It seems like the trans code could be better structured as far as how many pieces of code know things about self. But I'm trying to make minimal changes right now, and am just patching up those places as they appear to take into account that the self argument to a method might not always be a pointer. However, I'm worried that the fact that the size of the self argument depends on the monomorphized code for the method is going to cause problems. It hasn't yet, but I'm afraid it will, and that this will mean bigger restructuring is needed for trans.

I'm setting this aside in any case once we tag the release, though, and moving on to focus on rustpkg for a while, so I don't get totally lost in bug-fixing.

How this works:
You comment, I give you an age (please tell me how old you currently are - I don't know all of your ages unfortunately) and you fill out the meme questions with what applied to you back then, and now.

wordweaverlynn gave me 9.

I lived in:

A studio apartment on the 19th floor of an apartment complex on the Boston side of the Boston/Cambridge border, near the Science Museum, with my mother. My idea of a proper dwelling for an adult has always been an apartment in a high-rise building, and I've never really realized that vision given that I moved to the West Coast when I grew up, and that's less of a thing (unless you're into the condo-buying class, which I'm not; not too long after that, we got booted out of our subsidized apartment because the developer decided to flip Boston off and break his promise to maintain affordable housing, instead converting it all to condos).

I drove:

Nothing, because I was too old for my Smurf Big Wheels and hadn't yet learned how to ride a bike without training wheels.

I was in a relationship with:

The Boston Public Library, West End branch.

I feared:

People. Not knowing what to say. Embarrassment. Boredom. (This is not *too* different from now.)

I worked at:

My dissociation skills.

I wanted to be:

A librarian *and* a bookstore owner (just one or the other wasn't enough).

Down to 35 open issues for the release. I'm still working on #4439 and trying to untangle the mysteries of owned self. Since it started looking like we're not releasing 0.6 this week, I decided (quite probably unwisely) to rewrite a lot of the trans code to use the Datum type, to make the error easier to track down. The older trans code passes around raw LLVM values, whereas some of the newer code uses a Datum, which is an LLVM value bundled with a Rust type t and an annotation as to whether this value is a value of type t, or a pointer to t. Modernizing things is hard and tedious and I don't really know whether it's more or less time-consuming than actually tracking down the bug without modernizing the code. But that's how it goes.

I also added a span (location in source code that tells you the line and column number of a location in some source file) field to the Datum type, which I don't know if reviewers will object to when I try to check it in, but it sure makes debugging easier!

I feel like I should be doing something more urgent for the release, but I don't know what it is.

We're down to 46 open issues for the release! I spent most of my time today working on #4439, which may be behind a few other open bugs. Bugs that cause the compiler to generate seg-faulting code are bad, so I figured I would see if I could fix it before the release. Turns out trait methods that take their self by-move... are implemented strangely. As usual, many bits of undocumented code in trans (the Rust-to-LLVM pass) combine to create confusion.

I thought the best way to go forward would be to convert the data structure representing the self object in trans to use a Datum -- a type that wraps up an LLVM value with data about whether it's meant to be passed by reference or by value -- instead of a raw value. I still think it is, but getting everything consistent means debugging many LLVM assertion failures.

Today I thought I would focus on closing remaining release blockers; we're trying to punt most of the bugs marked 0.6 to some point in the future, as I mentioned last week. I didn't have a super-productive time, but we are down to 72 open issues milestoned for 0.6. I think I isolated the mysterious bit of code -- in trans::base::copy_arg_to_allocas -- that's causing by-value self not to work (as manifested in segfaults or bad frees), for example in #4850 and #4439. But I don't know exactly why the code is there or how to fix it.

Otherwise, I tried fixing a few bugs only to realize the work had already been done and the bug just needed to be closed.

Today, landed the patch I've been working on for a month to make the typechecker's handling of the error and _|_ types more sensible. I've already written about this; I'm just glad to have it done now, and know that the Rust typechecker won't complain about derived errors anymore! I think this is important for usability; I know from experience that too much noise (error messages that don't matter) can be demoralizing when you're trying to get your code compiling.

Except... then I went back and ran the test for this issue and still got a derived error. Oh no! It turned out I didn't edit the code in the pattern-match-checking part of the typechecker, and it was still calling span_fatal (a function that, as you might guess, unconditionally prints out an error message and then kills the whole compiler) rather than report_type_error (a function I wrote that checks for errors involving error types and suppresses them). So I started fixing that, but ran out of time.

In the meantime, I also finished triaging my allotted quota of randomly selected 0.6-milestone bugs, with the goal of getting everything either closed or un-milestoned unless it's SUPER IMPORTANT; the plan is to try to release 0.6 next week. We're going to shift towards priorities rather than milestones in the bug tracker, since it's become clear that labelling bugs with milestones doesn't help that much when doing time-based releases.

Tomorrow, finishing the pattern-checking stuff, doing my quota of doc work for the release, and going back to rustpkg.

I got my week off to a good start by restoring a "new" laptop from my TimeMachine backup, for the second time in about ten days. (After last week's 193 million memory errors.)

After the restore (which took about an hour, fortuitously coinciding with the weekly all-hands meeting) I started my three Rust workspaces building, and after some strategic sledgehammer (removing-the-build-directory) applications, that actually succeeded. So I went back to alternating between work on rustpkg and the typechecker refactor, as build times permitted.

I actually made some progress on rustpkg: last week I'd started trying to port an example of a custom Makefile into a package script instead. The example package I'm using, rust-sdl, currently has a Makefile with this rule (which only fires when building on Macs):

$(SDLXMAIN): $(SDL_PREFIX)/libSDLmain.a
        $(CP) $< $@
        $(AR) -x $@ SDLMain.o || $(RM) -f $@
        $(LD) -r SDLMain.o -o SDLXMain.o -alias _main _SDLX_main \
             -unexported_symbol main || $(RM) -f $@
        $(MV) SDLXMain.o SDLMain.o || $(RM) -f $@
        $(CHMOD) u+w $@ || $(RM) -f $@
        $(AR) -r $@ SDLMain.o || $(RM) -f $@

It's not for me to ask why; anyway, being fairly simple, this made a good example for me. I transliterated it into Rust like so:

const SDL_PREFIX : &static/str = "/usr/local/lib";

#[pkg_do(post_build)]
fn post_build() {
 // todo: delete new_sdl_lib_name if any commands fail
    let existing_sdl_lib_name = path(SDL_PREFIX).push("libSDLmain.a");
    let new_sdl_lib_name = path(SDL_PREFIX).push("libSDLXmain.a");
    let sdl_obj_file = ~"SDLMain.o";
    let sdlx_obj_file = ~"SDLXmain.o";
    copy_file(existing_sdl_lib_name, new_sdl_lib_name);
    archive_file(Extract, new_sdl_lib_name, sdl_obj_file);
    load_file(sdl_obj_file, sdlx_obj_file, ~[(~"_main", ~"_SDLX_main")],
              ~[(~"_main")]);
    move_file(sdlx_obj_file, sdl_obj_file);
    change_file_permissions((User, Writable), new_sdl_lib_name);
    archive_file(Extract, sdl_obj_file);
}

This is still sketchy and I haven't actually implemented most of the functions above yet. But the basic idea for what we're trying to do in rustpkg is that as much functionality as possible should be in the Rust build system; you shouldn't need to write a Makefile even to do something fancy.

So I spent that part of today trying to make this work. It doesn't yet, but I did get up to the point where rustpkg fails with a resolve error -- when calling out to compile the code in the package script -- because the functions I didn't implement yet aren't defined. Most of that time was spent defining a new version of rustpkg's PackageScript structure that has a PkgId field instead of a name field and a version field. PkgIds just bundle together the name and version of the package, both of which now will be inferred from the directory name (or version control info if present) if not declared. (The existing version of rustpkg that's up on github requires you to declare your package's name and version explicitly.)

The typechecker situation is sadder; I fixed my bad merges and ran tests, only to find that one of the run-fail tests was failing with a nondeterministic error:

Assertion failed: ((Ty->isIntegerTy() || Ty->isPointerTy()) && "Invalid cast"), \
    function CreatePointerCast, \
    file /Users/tchevalier/rust3/src/llvm/lib/VMCore/Instructions.cpp, \
     line 2385.

error: internal compiler error: Asked to compute contents of fictitious type

rustc(29409,0x107cca000) malloc: *** error for object 0x7fdd9ba19f80: pointer \
    being freed was not allocated
*** set a breakpoint in malloc_error_break to debug

Assertion failed: (box->td != NULL), function free, \
   file /Users/tchevalier/rust3/src/rt/boxed_region.cpp, line 75.

These are all different error messages that I got on different runs of the same test. Because this behavior looked so similar to what was happening on the other, broken laptop (though with a different test), I thought maybe this one was borked too. What are the chances of that? It's like in the joke that goes "I always bring a bomb with me when I fly, because what are the chances of there being two bombs on the airplane?" Running valgrind, though, assuaged that fear, since it actually worked (and reported a metric crap-ton of invalid reads), as opposed to declaring its inability to execute rustc (like on the broken laptop). The errors suggest some sort of memory corruption having to do with types, which isn't totally surprising since rustc represents types in a totally unsafe way, and it would be easy to get something wrong. It's still puzzling me that I can get so many tests to work with only one failing if I introduced a bug like that, though. Usually, these kinds of failures are catastrophic.

Anyway, progress, in that I closed the day with a laptop that isn't obviously broken, unlike last Friday...

I spent too much of today dealing with an apparent hardware failure on -- irritatingly enough -- the laptop I'd borrowed from IT to replace the laptop I was using for the past almost-a-year, which just before I left for Vancouver decided it didn't have a wifi card anymore. Most of yesterday I was trying to debug an issue where I was getting non-deterministic segfaulting and also occasional not-seeing-the-bug-I-was-looking for. Graydon suggested that I run memtest86, and so I did... and it reported 193 million errors before I decided that was enough. So then I sshed into a buildbot that Graydon decommissioned temporarily, and moved my working directories onto it.

After which, I found the bug! I'm not happy with my fix, though. So you may recall that a while ago, I introduced a ty_err type to represent a value that can't be typed and allow the typechecker to typecheck other parts of the program before finally reporting all the errors. The fix I was working on was to never represent a type like () -> ty_err -- a function type where the result type is erroneous -- because this should just normalize to ty_err; a type with a component that was erroneous is erroneous itself. The problem is that a type might look like () -> α, where α is a type variable, and if α gets unified with ty_err at some later point in time, the way the Rust typechecker works makes it hard to see how that information gets propagated back to anyone who might have a copy of () -> α sitting around.

My solution was just to call a function that resolves all type variables in the type at the point where we've just checked the body of a closure, and write the resolved type back in for the closure. But this is unsatisfactory to me because I haven't really thought about whether this is the only point in the typechecker where that needs to happen.

The whole approach is probably wrong and misguided in some way I don't quite understand, and then I decided it was a good idea to rebase my branch for some reason, which resulted in conflicts and quite possibly a bad merge by me. I think that means it's time to leave for the weekend.

I succeeded at moving to Vancouver! A bit of culture shock. (Feel free to come up with a joke about moving out of a mutable reference or something.) I'm only here for less than three months, so I'm trying to absorb as much of Graydon's knowledge as possible in the meantime.

Continuing to work on the typechecker refactoring (yes, still); running up against very frustrating sporadic segfaults that I'm pretty sure are not to do with my changes, and that's about the most frustrating thing that can happen.

I also started working on rustpkg to finish up the work that our valiant volunteer Zack started. So far, trying to get rustpkg to build rust-sdl. Since rust-sdl requires some custom build logic -- currently implemented in a Makefile -- to rename the library file on Macs, I figured that might be a good starting point for porting that into build logic implemented in Rust that rustpkg can run. Or not; I still don't really know what I'm doing yet.

Figured I'd prioritize resuming regular posting ahead of making informative postings ;-)

S	M	T	W	T	F	S
			1	2	3	4
5	6	7	8	9	10	11
12	13	14	15	16	17	18
19	20	21	22	23	24	25
26	27	28	29	30	31

Tim's journal

TMI: Documentation: what a concept

Because I do write about topics other than Rust hacking occasionally...

TMI: More rustpkg work

TMI: Today in rustpkg

TMI: rustpkg marches on

TMI: rustpkg: landed!

TMI: First rustpkg pull request!

TMI: rustpkg progress

TMI: More rustpkg progress

TMI: rustpkg progress

TMI: rustpkg

TMI: The finger pointing at the moon is not the moon

Age 9

TMI: Yak-shaving

TMI: Countdown to 0.6

TMI: Release week

TMI: The end of an error

TMI: Two bombs on the airplane

TMI: When it rains...

TMI: rustpkg and moving

Profile

May 2013

Links

Syndicate

Most Popular Tags

Page Summary

Active Entries

Style Credit

Expand Cut Tags