codegrunt.co.uk

The Wisdom of Fearless Stupidity

2012-01-29T00:00:00+00:00

A problem I’ve had for a long while has been the fear of looking stupid. It’s held back projects, study, blog posts (you can see this from the sparseness of this blog), and more generally things I’ve wanted to do which risk looking stupid - translated: worthwhile things.

Ironically this fear has often been self-fulfilling as I’ve ran through the familiar cycle of publicly promising to do something → fearing looking stupid since this thing is tough and I’ve hit up on a brick wall → engaging in endless procrastination → end up looking stupid for not having made progress on said thing.

This is all highly applicable to programming. Computers are so ridiculously complicated that there will always be something to defeat you, the field so wide that there will always be something you don’t know about, or even if you do know something relatively deeply there is always something (actually many things) which will crop up to surprise you (C sharpies should read Eric Lippert’s blog to bring this point home).

A curious aspect of our craft is how quickly you can find yourself in a situation where your particular problem is a unique enough combination of factors for you to be out there on your own at least in some regard or another. That is enough to make the most hardened, experienced developer insecure and is yet another factor in my own battle with the stupid.

Another consequence of this fear is that, since it’s really impossible to improve at something without questioning whether you have the best approach to the problem at hand (and thus risking looking stupid for having got something wrong), you end up less great than you could potentially be.

In terms of programming this fear stands in the way of better code. Without considering whether your code might be suboptimal (and therefore you possibly looking stupid), you end up almost definitely ensuring your code will be suboptimal. This is not unconnected to the only valid measure of code quality.

Clearly a big factor in all of this is pride, especially when you consider future employability, and especially when you have a name which is globally unique over the whole of the internet (the intersection of people with forename of Lorenzo and people with surname of Stoakes, rather than the more common Stokes, is small).

Fuck pride. Fuck not doing things you want to do because you might end up looking stupid, fuck getting stuck on a problem and not bringing the awesome power of crowd sourcing to bear on it.

Expect some stupidity going forward, but also intelligence as I work hard on fixing my stupidity, little-by-little. Sucking a little less each year.

Rulemotion

2012-01-28T00:00:00+00:00

For those who didn’t know, I have quit my job at a hedge fund to work at a startup - Rulemotion (website under construction). Exciting! Scary! Awesome! All at the same time :-)

I don’t move until early March, at which I point I am sure I shall blog some more about what I’m getting up to there/what we are all about.

Already looking to be an interesting year, this!

New Year

2011-12-31T00:00:00+00:00

It’s been a long time since I’ve updated this blog, and a completely crazy past few months, so I hope to rectify the former by using the latter as the basis for this post :)

I turned 30 this September, and the force of switching from the deferring-friendly 20’s (‘I can achieve these things, don’t worry, I’m still young’) to the ‘oh shit I’m an adult now’ 30’s has hit harder than I expected (in fact, to be honest, I didn’t expect it at all).

So what’s happened? Well in my personal life, I finally did something about my weight issue and lost ~3.5 stone (~50lb, ~23kg).

In my intellectual/geek life I committed to the open Stanford AI course and completed it with a 98.7% average*, and wrote notes for the purposes of actual applying the course to real things (the notes are not yet fully complete, however I do plan to finish that later), which has really helped with my recently very low intellectual confidence.

Two fundamental things have changed - firstly I’ve got organised about things (for weight loss - food diary, for study - committed, realistic work schedule), secondly and far more importantly, I’ve stopped beating myself up quite as badly about everything. Beating yourself up like that makes you incapable of doing anything since you feel constantly bad about whatever it is you’re doing, and thus reticent to do it, which means you don’t improve/change anything, which means you beat yourself up more, etc. - a vicious, vicious cycle.

Another thing to note here is the amazing usefulness of timeboxing. The past week I have got more hacking done in a week than I have for hundreds of weeks prior.

So what about resolutions? Over the coming year I plan to contribute considerably more to go, and build weak into a fully-fledged chess engine (though I can make no guarantees to its eventual strength) and I intend to work on improving my algorithmic and computer science fundamentals knowledge. On the final bit - I plan to enter the google code jam and blog about the experience, though I don’t expect or predict any sort of performance, it’s more of a motivation and point of focus.

Happy New Year!

* I’m not boasting here. I fell considerably below the performance of many other participants and wish I’d been more careful with many of the answers. Also I realised how rusty I am at this stuff. Also I couldn’t stand to carry on attending the meetup I was attending given how erudite, intelligent and informed the other people were compared to me.

Brain Thaw

2011-10-30T00:00:00+01:00

After writing my last post, I again worked my arse off this week to study both the Stanford AI and ML classes in a less crunchy fashion than the week before. Unfortunately the crunch avoidance did not come to pass and I was up until 3am this morning (well, 2am if you take into account the clocks going back :-) cramming on AI.

Since I actually want to learn about AI (and apply it to weak) I am not finding this cramming aspect of the study particularly beneficial. In my experience, cramming simply wears you out and gets you ready for a given short-term task (e.g. homework, later exams) before the information you’ve learned largely plops out of your brain.

For that reason, and for the sake of keeping this doable, I’ve decided to drop the machine learning course and focus fully on AI. There is an irony in that - I have found the AI work to be considerably more involved and challenging than machine learning (that’s not a comment on the relative quality of one course vs. the other by the way), however a. AI is of more relevance to what I want to do, and b. focusing on one thing reduces the stressful feeling of distraction that maintaing another course which requires homework, etc. brings.

I am still very interested in machine learning, so I might very well go over the machine learning course material at a later date. Additionally, the AI course contains some cross-over material so I won’t do entirely without.

Another aspect is that I would like to maintain my notes to a better standard than I have (and which more time would allow me to achieve), rather than having to rush them out because of time constraints. There are almost certainly better notes for the course out there, however there is nothing quite like having a set of notes to refer to you which you’ve written in your own style and to your own tastes. Not invented here syndrome, perhaps!

Meta-Blog Tediousness

On an entirely different subject, you may have noticed I’ve deleted a number of posts. This is because I felt they were of little value and often quite negative. This blog is meant to be a diary of where I’m at in my tech life, and yes that involves both positive and negative, however there is nothing more tedious than wading through either rather mediocre technical posts, or moany personal ones. I’d like to maximise the chances of me (and to some degree others of course :-) actually enjoying reading this blog.

I can only apologise to those few whose comments have consequently been deleted from this site. Hopefully you can understand why I’ve done it.

AI/ML Brain Melt

2011-10-23T00:00:00+01:00

Am currently attempting to do both the Stanford AI and Machine Learning courses at once. My brain is melting.

I’m keeping my ongoing rough notes on github, I’m writing them in markdown so they render relatively nicely there.

I have realised that, if I am actually going to see both courses through to the end, I am going to have to put everything else on hold until they’re done.

Luckily, they both pertain to greater or lesser degrees to weak so its not entirely a pause on other projects. Also, I might choose to hack on weak from time-to-time since I am chomping at the bit to get it moving.

The Myth of the Genius Programmer

2011-05-16T00:00:00+01:00

Recently found an amazing talk focusing on the social/interpersonal aspects of development from Google I/O (2009 vintage), the point on letting yourself be vulnerable particularly chimed with me.

Forward-Referenced Recursive Types in Go

2011-05-04T00:00:00+01:00

Patch Accepted

I recently received some good news - after much code review (and much cleaning up of my messy first attempt :-) my go patch addressing issue 667 has been accepted into the mainline. Yay!

Recursive Types

The issue is related to invalid recursive types, so let’s cover that first - what exactly is a recursive type, and when is it invalid?

As to the first question - a recursive type is simply a type which, at some point or another references itself in its definition.

A simple example:-

type List struct {
        Value interface{}
        Next *List
}

A convoluted example (note the recursive types here are T2,T5-7) :-

type T1 struct { T2; T3 }
type T2 struct { T5; T7 }
type T3 T4
type T4 struct {}
type T5 T6
type T6 T7
type T7 struct { *T2 }

Note that both of these examples describe valid recursive types. So what is an invalid recursive type?

type Foo struct {
        foo Foo
}

What makes this invalid?

Simply put, determining the size required for storage of this type would (assuming we don’t check for this in advance) result in an infinite loop - there is a cycle in the type’s definition.

There are many ways in which you can achieve this in code, it doesn’t have to be as overt as the previous example, e.g.:-

type Foo struct {
        bar Bar
}
type Bar Foo

This second example can be made valid by making the bar field a pointer type - that way we have no problem determining the size of the type - the field’s size is the system pointer size, e.g.:-

type Foo struct {
        bar *Bar
}
type Bar Foo

The Bug

Now we get to the bug - the code above wrongly resulted in an ‘invalid recursive type’ error.

More generally - types which were validly recursive via a forward reference were wrongly reported as invalid recursive types.

Note that the following compiled just fine:-

type Bar Foo
type Foo struct {
        bar *Bar
}

The problem comes down to the way the compiler constructs its internal representation of declared types - it constructs underlying types first (assuming they haven’t been constructed already) before constructing the type itself.

A problem with this approach arises when you have a declaration like:-

type Bar Foo

In essence this says - ‘copy Foo into a new type called Bar’.

This is fine most of the time, but if the type you are copying is not constructed yet (remember I am talking about the construction of the internal compiler representation of the type here), then you get whatever the compiler defaults the type to before finishing its construction.

It turns out that the compiler defaults to classifying all types (for those that are interested in the internals - the etype field) as forward types (etype == TFORW), before setting their actual type once the underlying types are constructed. Later, the compiler checks to see whether the type is still classified this way, and if so an ‘invalid recursive type’ error occurs (this isn’t the only way invalid recursive types are picked up, incidentally).

This makes sense for situations like:-

type Foo Foo

Or:-

type Foo Bar
type Bar Foo

And is, in fact, a nice solution in these instances, however the fact that the language allows a pointer type to refer to a forward reference is the cause of our bug.

The Solution

The solution I came up with was to specifically look for these circumstances and defer the copying of type information until after the base type has been constructed.

So, at the point where we would otherwise simply go ahead and check the type, specifically check to see whether there is a forward-type in its underlying type graph. If so, defer the type copy.

We have to be careful to actively check for cycles in the underlying type list to prevent the compiler from disappearing into an infinite loop when we encounter:-

type Foo Foo

-type scenarios. I used ‘Floyd’s Cycle-Finding Algorithm’ which is O(n) - described here.

The Best Bit

My favourite part of it all is the fact I could refer to the solution as a ‘dance’:-

// a dance to handle forward-declared recursive pointer types.

The go team often uses the term which is quite appropriate for some of the things which are necessary to make sure stuff works :-)

$ cd $GOROOT/src
$ find . -iname *.c | xargs grep -i 'dance' | wc -l
17

There are obviously more details going on here, but I don’t want to go into unnecessary detail here.

Also, yeah - I know - the patch is far from perfect, and could well be improved, but in any case I’m proud to have been able to fix the problem and contribute to a project I believe in.

Go Open Source!

Another Go Patch

2011-04-04T00:00:00+01:00

Finally after a lot of work I have been able to submit a patch to go which fixes issue 667.

I will write a more detailed post about what was involved later; suffice it to say I’m happy I’ve got it done, though I suspect it might some further tweaking (prompted by the comments of the other gopher devs of course) to be entirely up to snuff.

Go, err…, Go! ;-)

Go Go Gadget Patch!

2011-01-14T00:00:00+00:00

One of my core projects right now is to attempt to contribute something to the go language project. In this post I describe the experience of tackling my first ever open source bug.

Note: All links to the source code, etc. were correct at the time of writing, however being an open source project there’s every chance things will change and code I’m referencing will almost certainly sit on lines other than the ones I link to. Hopefully there’s enough context in the post to make clear what I’m talking about :-)

Why (Attempt to) Contribute?

I’m doing this because -

I really like go and want to give something back,
I really want to contribute to an established open source project with smart contributors and a tough code review policy for the sakes of improving myself and, of course,
I am very interested in compiler writing and language design and thus this is a great OSS project for me.

My Very First Open Source Bug Fix

So, one of the first problems I tried to tackle was bug 1324 - though it seemed like it might end up being too much, it seemed interesting and would be, succeed or fail, a good introduction to the codebase.

The bug occurs when compiling the following code:-

package main

type T struct{}
type P *T

func (t *T) Meth()  {}
func (t T)  Meth2() {}

func main() {
	t := &T{}
	p := P(t)
	p.Meth()
	p.Meth2()
}

Let’s run through this piece-by-piece. Firstly we declare two types - a struct type, T, and a pointer type, P, with base type T:-

type T struct{}
type P *T

Let’s see what the spec’s (illustrative and thus not necessarily exactly as implemented) grammar says. The grammar for type declaration, TypeDecl, is as follows:-

TypeDecl     = "type" ( TypeSpec | "(" { TypeSpec ";" } ")" ) .
TypeSpec     = identifier Type .

We are declaring type P *T, so ‘P’ is our identifier:-

identifier = letter { letter | unicode_digit } .

And *T is our Type:-

Type      = TypeName | TypeLit | "(" Type ")" .
TypeName  = QualifiedIdent .
TypeLit   = ArrayType | StructType | PointerType | FunctionType | InterfaceType |
	    SliceType | MapType | ChannelType .

So it looks like P is a type literal, since a pointer type seems the best fit for our declaration. Assuming our PointerType is *T:-

PointerType = "*" BaseType .

And our BaseType is T:-

BaseType = Type .

Okay, so it’s definitely a pointer type (or the grammar would be non-deterministic, which in theory it could be, but don’t worry, I checked, it definitely is a pointer type :-)

And now things get recursive :). Glancing back to the Type grammar, it is clear that the correct item to substitute for Type is QualifiedIdent:-

QualifiedIdent = [ PackageName "." ] identifier .

’T’ omits package name, so finally T is declared as an identifier here.

So the grammar permits it and interprets this declaration as a declaration of a new type P which is a pointer type of base type T.

Now for the rest of the code:-

func (t *T) Meth()  {}
func (t T)  Meth2() {}

func main() {
	t := &T{}
	p := P(t)
	p.Meth()
	p.Meth2()
}

Here we declare two methods for receiver T, one parameterised by T and the other by a pointer to T.

In the main function we instantiate T via a composite literal, take its address then cast it to p.

Finally, we are at the contentious bit of the code - calling the methods declared for T on a P type.

Why is this contentious? Well, according to the spec, declaring a pointer type is special in that, though declared as a separate type from *T, you cannot declare methods for it:-

The receiver type must be of the form T or *T where T is a type name. T is called the receiver base type or just base type. The base type must not be a pointer or interface type and must be declared in the same package as the method. The method is said to be bound to the base type and is visible only within selectors for that type (§Type declarations, §Selectors).

However, according to the types chapter of the spec, the method set of pointer type *T is that of all methods with receiver *T or T:-

A type may have a method set associated with it (§Interface types, §Method declarations). The method set of an interface type is its interface. The method set of any other named type T consists of all methods with receiver type T. The method set of the corresponding pointer type *T is the set of all methods with receiver *T or T (that is, it also contains the method set of T). Any other type has an empty method set. In a method set, each method must have a unique name.

We have to ask the question - as we’ve established P is a pointer type, do we treat it as if it were exactly equivalent to *T, and thus allow method calls on a value of type P, or given that it’s a type literal, do we treat it as if it were its own type, but then remain unable to call or define any methods on it?

As it stands, the compiler currently takes the latter approach, generating errors when compiling the above code:-

repro.go:18: p.Meth undefined (type T has no field or method Meth)
repro.go:19: p.Meth2 undefined (type T has no field or method Meth2)

So, after all that preamble, we get to the initial complaint made in the bug report - the error message refers to type T rather than type P when reporting an error, thus reporting something which is a. simply not true (type T does have methods Meth and Meth2), and b. irrelevant to the cause of the error.

My initial approach was to fix the code to produce a corrected error message, as initially requested. Firstly I adjusted my copy of the code to disable optimisations which would get in the way of debugging, by adjusting the HOST_EXTRA_CFLAGS variable in the Make.inc file, changing the -O2 option to -O0 to force no optimisation, rebuilt the project and broke out trusty ol’ gdb :-)

After a bit of exploration it became clear the error was occurring in typecheck.c, in the typecheck() method. The error occurs in this block of code:-

if(isptr[t->etype] && t->type->etype != TINTER) {
    t = t->type;
    if(t == T)
        goto error;
    n->op = ODOTPTR;
    checkwidth(t);
}
if(!lookdot(n, t, 0)) {
    if(lookdot(n, t, 1))
        yyerror("%#N undefined (cannot refer to unexported field or method %S)", n, n->right->sym);
    else
        yyerror("%#N undefined (type %T has no field or method %S)", n, t, n->right->sym);
    goto error;
}

In particular, this line is where the code is failing:-

yyerror("%#N undefined (cannot refer to unexported field or method %S)", n, n->right->sym);

After some investigation it became clear that the first block in the previous code, e.g.:-

if(isptr[t->etype] && t->type->etype != TINTER) {
    t = t->type;
    if(t == T)
        goto error;
    n->op = ODOTPTR;
    checkwidth(t);
}

Was responsible for translating t (the type under analysis) when a pointer type to its base, which was then referenced in the error message:-

yyerror("%#N undefined (type %T has no field or method %S)", n, t, n->right->sym);

The solution, then, was to store the type before translating it and to display that in the error message. I submitted the patch (after a false start - hey still getting used to it all :-), and waited for the code review discussion. The resultant code ended up looking like this:-

Firstly I added a new local to the declarations at the top of the function:-

Type *t, *tp, *missing, *have;

Then used it to reference the original type:-

tp = t;
if(isptr[t->etype] && t->type->etype != TINTER) {
    t = t->type;
    if(t == T)
        goto error;
    n->op = ODOTPTR;
    checkwidth(t);
}
if(!lookdot(n, t, 0)) {
    if(lookdot(n, t, 1))
        yyerror("%#N undefined (cannot refer to unexported field or method %S)", n, n->right->sym);
    else
        yyerror("%#N undefined (type %T has no field or method %S)", n, tp, n->right->sym);
    goto error;
}

Which fixed the issue and produced error messages as you would expect, e.g.:-

repro.go:18: p.Meth undefined (type P has no field or method Meth)
repro.go:19: p.Meth2 undefined (type P has no field or method Meth2)

This is a very trivial fix of course, but I’m just starting out so I think it’s forgivable :-), and most importantly it fixes the bug as requested.

However, the reviewers threw a curveball - it turned out the bug might not actually be a bug at all. Damn. Back to the drawing board.

After a bit more dabbling with gdb, and a lot of effort pulling out a list of functions called by go in the process of compiling my repro code in order to understand go a little better I eventually stumbled on a solution. After a lot of messing around posting several iterations of the patch due to not understanding the code submission procedure all too well, I managed to get the patch submitted.

Here is the block of code I narrowed down as the one concerned with determining how to resolve a pointer type literal, basically the entire lookdot() function:-

Note: Capitalised T and S represent the null pointer values for types and symbols respectively.

static int
lookdot(Node *n, Type *t, int dostrcmp)
{
    Type *f1, *f2, *tt, *rcvr;
    Sym *s;

    s = n->right->sym;

    dowidth(t);
    f1 = T;
    if(t->etype == TSTRUCT || t->etype == TINTER)
        f1 = lookdot1(s, t, t->type, dostrcmp);

    f2 = T;
    if(n->left->type == t || n->left->type->sym == S) {
        f2 = methtype(t);
        if(f2 != T) {
            // Use f2->method, not f2->xmethod: adddot has
            // already inserted all the necessary embedded dots.
            f2 = lookdot1(s, f2, f2->method, dostrcmp);
        }
    }

    if(f1 != T) {
        if(f2 != T)
            yyerror("ambiguous DOT reference %S as both field and method",
                n->right->sym);
        if(f1->width == BADWIDTH)
            fatal("lookdot badwidth %T %p", f1, f1);
        n->xoffset = f1->width;
        n->type = f1->type;
        if(t->etype == TINTER) {
            if(isptr[n->left->type->etype]) {
                n->left = nod(OIND, n->left, N);    // implicitstar
                typecheck(&n->left, Erv);
            }
            n->op = ODOTINTER;
        }
        return 1;
    }

    if(f2 != T) {
        tt = n->left->type;
        dowidth(tt);
        rcvr = getthisx(f2->type)->type->type;
        if(!eqtype(rcvr, tt)) {
            if(rcvr->etype == tptr && eqtype(rcvr->type, tt)) {
                checklvalue(n->left, "call pointer method on");
                addrescapes(n->left);
                n->left = nod(OADDR, n->left, N);
                n->left->implicit = 1;
                typecheck(&n->left, Etype|Erv);
            } else if(tt->etype == tptr && eqtype(tt->type, rcvr)) {
                n->left = nod(OIND, n->left, N);
                n->left->implicit = 1;
                typecheck(&n->left, Etype|Erv);
            } else {
                // method is attached to wrong type?
                fatal("method mismatch: %T for %T", rcvr, tt);
            }
        }
        n->right = methodname(n->right, n->left->type);
        n->xoffset = f2->width;
        n->type = f2->type;
        n->op = ODOTMETH;
        return 1;
    }

    return 0;
}

lookdot() is concerned, unsurprisingly, with dealing with what to do when a dot is encountered in the token stream, and thus what to do when our p.Meth() and p.Meth2() calls occur.

Let’s tackle this one block of code at a time. After local declarations we have:-

dowidth(t);
f1 = T;
if(t->etype == TSTRUCT || t->etype == TINTER)
    f1 = lookdot1(s, t, t->type, dostrcmp);

f2 = T;
if(n->left->type == t || n->left->type->sym == S) {
    f2 = methtype(t);
    if(f2 != T) {
        // Use f2->method, not f2->xmethod: adddot has
        // already inserted all the necessary embedded dots.
        f2 = lookdot1(s, f2, f2->method, dostrcmp);
    }
}

What is t? Well, by examining the call stack during compilation we can determine which code is declaring this t before passing it to the function; It turns out the following code is responsible, wherein it is clear that t is the type of the node to the left of the dot:-

l = n->left;
if((t = l->type) == T)
    goto error;

Though this must be caveated against the fact that t is altered before calling lookdot() in the case that t is a pointer type (as described in the previous patch attempt), i.e.:-

if(isptr[t->etype] && t->type->etype != TINTER) {
    t = t->type;
    if(t == T)
        goto error;
    n->op = ODOTPTR;
    checkwidth(t);
}

After determining appropriate data structure alignment for our type via dowidth(), we examine two separate possibilities - the type of the identifier on the left is a structure or an interface (TSTRUCT or TINTER), or whether it is a pointer. In each case we determine whether it is a field or a method on the righthand side.

Since we are dealing with a method on a pointer type, we can restrict the code we look at to the following:-

static int
lookdot(Node *n, Type *t, int dostrcmp)
{
    ...

    f2 = T;
    if(n->left->type == t || n->left->type->sym == S) {
        f2 = methtype(t);
        if(f2 != T) {
            // Use f2->method, not f2->xmethod: adddot has
            // already inserted all the necessary embedded dots.
            f2 = lookdot1(s, f2, f2->method, dostrcmp);
        }
    }

    ...

    if(f2 != T) {
        tt = n->left->type;
        dowidth(tt);
        rcvr = getthisx(f2->type)->type->type;
        if(!eqtype(rcvr, tt)) {
            if(rcvr->etype == tptr && eqtype(rcvr->type, tt)) {
                checklvalue(n->left, "call pointer method on");
                addrescapes(n->left);
                n->left = nod(OADDR, n->left, N);
                n->left->implicit = 1;
                typecheck(&n->left, Etype|Erv);
            } else if(tt->etype == tptr && eqtype(tt->type, rcvr)) {
                n->left = nod(OIND, n->left, N);
                n->left->implicit = 1;
                typecheck(&n->left, Etype|Erv);
            } else {
                // method is attached to wrong type?
                fatal("method mismatch: %T for %T", rcvr, tt);
            }
        }
        n->right = methodname(n->right, n->left->type);
        n->xoffset = f2->width;
        n->type = f2->type;
        n->op = ODOTMETH;
        return 1;
    }

    return 0;
}

Our first issue is that the code which determines the type to hang the method off cannot handle the case where the left-hand node’s type is not the same as the submitted type (which has now been set to its base type, which is P rather than T int his case). The solution is to specifically check for the instance of a pointer type literal.

After a small refactoring, bringing the local tt out of the second block and using it in the first, we are left with:-

tt = n->left->Type;

...

f2 = T;
if(tt == t || tt->sym == S || (n->op == ODOTPTR && tt->type == t)) {
    f2 = methtype(t);
    if(f2 != T) {
        // Use f2->method, not f2->xmethod: adddot has
        // already inserted all the necessary embedded dots.
        f2 = lookdot1(s, f2, f2->method, dostrcmp);
    }
}

This, however, does not fully resolve the problem. Though the problem is fixed when the type of receiver type is not a pointer (but the node type is), handled by the else if below, the scenario where both the receiver and node type are pointers is not handled:-

if(!eqtype(rcvr, tt)) {
    if(rcvr->etype == tptr && eqtype(rcvr->type, tt)) {
        checklvalue(n->left, "call pointer method on");
        addrescapes(n->left);
        n->left = nod(OADDR, n->left, N);
        n->left->implicit = 1;
        typecheck(&n->left, Etype|Erv);
    } else if(tt->etype == tptr && eqtype(tt->type, rcvr)) {
        n->left = nod(OIND, n->left, N);
        n->left->implicit = 1;
        typecheck(&n->left, Etype|Erv);
    } else {
        // method is attached to wrong type?
        fatal("method mismatch: %T for %T", rcvr, tt);
    }
}

The first case handles the receiver being a pointer and the node type not being one and inserts an implicit address-of node before the left node. The second case handles the reverse scenario, in which case an implicit indirect node is inserted. Combining both actions and adding a case for both receiver and node pointers gives us the following:-

if(!eqtype(rcvr, tt)) {
    if(rcvr->etype == tptr && eqtype(rcvr->type, tt)) {
        checklvalue(n->left, "call pointer method on");
        addrescapes(n->left);
        n->left = nod(OADDR, n->left, N);
        n->left->implicit = 1;
        typecheck(&n->left, Etype|Erv);
    } else if(tt->etype == tptr && eqtype(tt->type, rcvr)) {
        n->left = nod(OIND, n->left, N);
        n->left->implicit = 1;
        typecheck(&n->left, Etype|Erv);
    } else if (tt->etype == tptr && rcvr->etype == tptr &&
        eqtype(tt->type, rcvr->type)) {
        checklvalue(n->left, "call pointer method on");
        addrescapes(n->left);
        n->left = nod(OIND, n->left, N);
        n->left->implicit = 1;
        n->left = nod(OADDR, n->left, N);
        n->left->implicit = 1;
        typecheck(&n->left, Etype|Erv);
    } else {
        // method is attached to wrong type?
        fatal("method mismatch: %T for %T", rcvr, tt);
    }
}

We ‘indirect’ the node to the left of the dot (in our repro code this is equivalent to *P returning T) then take the address of the result (giving us *T) and therefore ensuring the candidate type matches the receiver. This solution fixes the bug and passed all tests. Yay!

Unfortunately there was yet another twist in the road - this now breaks an already fixed bug, so the patch now relies on an important decision being made with regard to what the language ought to do. Due to the fact that there are many other issues of higher priority right now, the decision on which route to take has not yet been taken, and the patch has been put on hold.

Regardless, it’s cool from my perspective - I gained some familiarity with the code and can now go fix something else of higher priority while waiting on a decision on this one :-)

F# Rocks

2010-11-28T00:00:00+00:00

Update: I didn’t realise - there is a video of me giving the talk up on the web over at skillsmatter - now you get to see me be embarrassingly nervous for yourself :)

So, in an effort to work on presentation skills and a genuine desire to contribute to the community, I decided to contribute a presentation at the Hacker News London Meetup group, last thursday.

Unfortunately, being incredibly nervous, and given the fact that the number of attendees had expanded to around 150 (!), I didn’t deliver a wonderfully confident performance - all the usual signs of being a presentation virgin - babbling, eyes down, obvious discomfort… I got good feedback on the content however, and since presentation skill is about practice, practice and more practice I am overall very happy that I did it.

The evening was wonderful - thanks very much Dmitri and Ben for that!

So, without further ado - here are the slides:-

The overall intention of the talk was to give an idea of the ‘feel’ of working with F# - most presentations I’ve seen on languages focus either on a low-level tutorial or a high-level discussion; in both cases you are often left wondering what it’s actually like to work with the language by examining some real, yet probably incomprehensible for those who haven’t used it, code.

Rather than go over the slides in detail in this blog post, I am working on a series of posts entitled ‘F# for the Working C# Programmer’ where I will provide a proper tutorial on the language.

Conclusions - meeting other hackers is wonderful, having the balls to stand up and put yourself out there is good for you.

C# Norvig Spelling Corrector

2010-11-02T00:00:00+00:00

Quite some time ago Peter Norvig implemented a spelling corrector that performs the ‘did you mean x’ suggestions google (and now many other sites) provide, written in an impressive 21 lines of Python.

I thought it would be interesting to port to C#, so I’ve written three versions of the code - a version that remains faithful to the Python implementation, an idiomatic version where I attempt to solve the problem following my approximation of idiomatic C# code :), and a ‘minified’ version for code golf purposes.

Note that no version is production engineered (for example, there is little to no error checking, no tests, etc.), nor have they been subject to the kind of algorithmic explorations that would be wise in real use. You have been warned!

Note also that the idiomatic version relies on better type inference in .net 4 in the OrderByDescending() method in Correct() - if you’re using 3.5, you will need to provide type arguments.

You will need the file Peter provides as a source text, big.txt.

The code is MIT licenced, feel free to use it for whatever.

All the code is available on github.

Faithful

Note this version reads from stdin until it receives an empty string.

using System;
using System.Linq;
using System.Text.RegularExpressions;
using System.Collections.Generic;
using System.IO;

static class Faithful
{
    private static IEnumerable<string> words(string text)
    {
        return Regex.Matches(text.ToLower(), "[a-z]+")
                    .Cast<Match>()
                    .Select(m => m.Value);
    }

    private static Func<string, int?> train(IEnumerable<string> features)
    {
        var dict = features.GroupBy(f => f)
                           .ToDictionary(g => g.Key, g => g.Count());

        return f => dict.ContainsKey(f) ? dict[f] : (int?)null;
    }

    private static Func<string, int?> NWORDS;

    private const string alphabet = "abcdefghijklmnopqrstuvwxyz";

    private static IEnumerable<string> edits1(string word)
    {
        var splits =     from i in Enumerable.Range(0, word.Length)
                         select new {a = word.Substring(0, i), b = word.Substring(i)};
        var deletes    = from s in splits
                         where s.b != "" // we know it can't be null
                         select s.a + s.b.Substring(1);
        var transposes = from s in splits
                         where s.b.Length > 1
                         select s.a + s.b[1] + s.b[0] + s.b.Substring(2);
        var replaces   = from s in splits
                         from c in alphabet
                         where s.b != ""
                         select s.a + c + s.b.Substring(1);
        var inserts    = from s in splits
                         from c in alphabet
                         select s.a + c + s.b;

        return deletes
        .Union(transposes) // union translates into a set
        .Union(replaces)
        .Union(inserts);
    }

    private static IEnumerable<string> known_edits2(string word)
    {
        return (from e1 in edits1(word)
                from e2 in edits1(e1)
                where NWORDS(e2) != null
                select e2)
               .Distinct();
    }

    private static IEnumerable<string> known(IEnumerable<string> words)
    {
        return words.Where(w => NWORDS(w) != null);
    }

    private static string correct(string word)
    {
        var candidates =
            new[] { known(new[] {word}),
                    known(edits1(word)),
                    known_edits2(word),
                    new[] {word} }
                  .First(s => s.Any());

        return candidates.OrderByDescending(c => NWORDS(c) ?? 1).First();
    }

    private static void ReadFromStdIn()
    {
        string word;
        while (!string.IsNullOrEmpty(word = (Console.ReadLine() ?? "").Trim()))
        {
            Console.WriteLine(correct(word));
        }        
    }

    public static void Main(string[] args)
    {
        if (!File.Exists("big.txt"))
        {
            Console.Error.WriteLine("Cannot find big.txt.");
            return;
        }

        NWORDS = train(words(File.ReadAllText("big.txt")));

        ReadFromStdIn();
    }
}

Idiomatic

Note this version also reads from stdin until it receives an empty string.

using System;
using System.Linq;
using System.Text.RegularExpressions;
using System.Collections.Generic;
using System.IO;

interface ICorpus
{
    int Rank(string word);
    bool Contains(string word);
    IEnumerable<string> Known(IEnumerable<string> words);
}

static class StringExtensions
{
    public static string From(this string str, int n)
    {
        if(str == null) return null;

        var len = str.Length;

        if (n >= len) return "";
        if (n == 0 || -n >= len) return str;

        return str.Substring((len + n) % len, (len - n) % len);
    }

    public static string To(this string str, int n)
    {
        if (str == null) return null;

        var len = str.Length;

        if(n == 0 || -n >= len) return "";
        if (n >= len) return str;

        return str.Substring(0, (len + n) % len);
    }
}

class Corpus : ICorpus
{
    private readonly Dictionary<string, int> rankings;

    private static IEnumerable<string> ExtractWords(string str)
    {
        return Regex.Matches(str, "[a-z]+", RegexOptions.IgnoreCase)
                    .Cast<Match>()
                    .Select(m => m.Value);
    }

    public Corpus(string sample) : this(ExtractWords(sample)) {}

    public Corpus(IEnumerable<string> sample)
    {
        rankings = sample.Select(w => w.ToLower())
                         .GroupBy(w => w)
                         .ToDictionary(w => w.Key, w => w.Count());
    }

    public int Rank(string word)
    {
        int ret;
        return rankings.TryGetValue(word, out ret) ? ret : 1;
    }

    public bool Contains(string word)
    {
        return rankings.ContainsKey(word);
    }

    public IEnumerable<string> Known(IEnumerable<string> words)
    {
        return words.Where(Contains);
    }
}

class SpellCorrect
{
    private readonly ICorpus corpus;

    public SpellCorrect(ICorpus corpus)
    {
        this.corpus = corpus;
    }

    private const string Alphabet = "abcdefghijklmnopqrstuvwxyz";

    private IEnumerable<string> Edits(string word)
    {
        var splits     = from i in Enumerable.Range(0, word.Length)
                         select new {a = word.To(i), b = word.From(i)};
        var deletes    = from s in splits
                         where s.b != "" // Guaranteed not null
                         select s.a + s.b.From(1);
        var transposes = from s in splits
                         where s.b.Length > 1
                         select s.a + s.b[1] + s.b[0] + s.b.From(2);
        var replaces   = from s in splits
                         from c in Alphabet
                         select s.a + c + s.b.From(1);
        var inserts    = from s in splits
                         from c in Alphabet
                         select s.a + c + s.b;

        return deletes
        .Union(transposes)
        .Union(replaces)
        .Union(inserts);
    }

    private IEnumerable<string> Corrections(string word)
    {
        if (corpus.Contains(word)) return new[] {word};

        var edits = Edits(word);
        
        var knownEdits = corpus.Known(edits);
        if (knownEdits.Any()) return knownEdits;

        var secondPass = from e1 in edits
                         from e2 in Edits(e1)
                         where corpus.Contains(e2)
                         select e2;

        return secondPass.Any() ? secondPass : new[] { word };
    }

    public string Correct(string word)
    {
        var corrections = Corrections(word).OrderByDescending(corpus.Rank);
        
        return corrections.First();
    }
}

static class Idiom
{
    private static SpellCorrect corrector;

    private static void ReadFromStdIn()
    {
        string word;
        while (!string.IsNullOrEmpty(word = (Console.ReadLine() ?? "").Trim()))
        {
            Console.WriteLine(corrector.Correct(word));
        }
    }

    public static void Main(string[] args)
    {
        if (!File.Exists("big.txt"))
        {
            Console.Error.WriteLine("Cannot find big.txt.");
            return;
        }

        var sample = File.ReadAllText("big.txt");
        var corpus = new Corpus(sample);
        corrector = new SpellCorrect(corpus);

        ReadFromStdIn();
    }
}

Minified

This version is stripped down and as a result, not wonderfully readable.

using System;
using System.IO;
using System.Linq;
using System.Text.RegularExpressions;
using StrEnum = System.Collections.Generic.IEnumerable<string>;

static class Minified
{
    const string Alphabet = "abcdefghijklmnopqrstuvwxyz";

    static StrEnum Edits1(string w)
    {
        // Deletion
        return (from i in Enumerable.Range(0, w.Length)
                select w.Substring(0, i) + w.Substring(i + 1))
        // Transposition
         .Union(from i in Enumerable.Range(0, w.Length - 1)
                select w.Substring(0, i) + w.Substring(i + 1, 1) +
                       w.Substring(i, 1) + w.Substring(i + 2))
        // Alteration
         .Union(from i in Enumerable.Range(0, w.Length)
                from c in Alphabet
                select w.Substring(0, i) + c + w.Substring(i + 1))
        // Insertion
         .Union(from i in Enumerable.Range(0, w.Length + 1)
                from c in Alphabet
                select w.Substring(0, i) + c + w.Substring(i));
    }

    static void Main(string[] args)
    {
        var nWords = (from Match m in Regex.Matches(File.ReadAllText("big.txt").ToLower(), "[a-z]+")
                      group m.Value by m.Value)
                     .ToDictionary(gr => gr.Key, gr => gr.Count());

        Func<StrEnum, StrEnum> nullIfEmpty = c => c.Any() ? c : null;

        var candidates =
            nullIfEmpty(new[] {args[0]}.Where(nWords.ContainsKey))
            ?? nullIfEmpty(Edits1(args[0]).Where(nWords.ContainsKey))
            ?? nullIfEmpty((from e1 in Edits1(args[0])
                            from e2 in Edits1(e1)
                            where nWords.ContainsKey(e2)
                            select e2).Distinct());

        Console.WriteLine(
            candidates == null
                ? args[0]
                : (from cand in candidates
                   orderby (nWords.ContainsKey(cand) ? nWords[cand] : 1) descending
                   select cand).First());
    }
}