FogBugz v8.13.104 was released at the end of May 2015, with little fuss and no fanfare. And why would it? The
release notes list just a couple of bug fixes and minor changes. But under the hood, it was a huge release. For the first time since FogBugz' creation some 15 years ago, it contained 0 lines of VBScript or Wasabi code. The '
End' of an era.
For those of you that dont know, Wasabi was our own in-house .NET compiler. In this post, Im going give you the background of Wasabi, including why we created it and what it enabled us to do, before giving you the inside scoop on why weve now killed it off. In a follow-up post, well talk about some parts of Wasabi that well be open-sourcing.
A Time-Saving Beginning
FogBugz was created as an internal tool to help track bugs in our first product, CityDesk, and support our consultancy work. It was written in VBScript and was Windows only. But from its first public release, customers asked for a Linux version. We didnt want to throw away what was already a large codebase and start from scratch on a new cross-platform version. So we hired our first summer intern to create Thistle. Thistle was a compiler, written in Java, that translated a limited subset of VBScript into PHP. It consisted of a tokenizer, a simple parser, and a finite-state machine based PHP generator. For anything that could not be expressed in Thistle, there was a special syntax to embed PHP code directly in the ASP. It wasnt pretty, but without requiring a total rewrite of FogBugz, it meant we could support both Windows and Unix.
Thistle received a couple of small updates over the next few years, such as adding the ability to generate ASP, so that we could insert code coverage logging information into the generated code.
FogBasic: Further Down the Rabbit Hole
Then around 2005, we leaped wholeheartedly down the rabbit hole. Thistle became FogBasic and we had visions of creating a new language that overcame many of the issues with writing ASP. This decision wasn't
without its critics, but we really wanted to make it so that as many programmer errors as possible were found at compile time instead of runtime. So we added numerous features and extensions to the language, like updating the lexer to more rigorously match the language, parsing a full abstract syntax tree, and adding type inference.
In the summer of 2006, FogBasic became known as Wasabi, and we continued to make improvements. By this point, almost all of the new code in FogBugz 6.0 used one or more Wasabi features. Wasabi could generate ASP, PHP, and client-side JavaScript. It supported
Picture Functions, reduced the amount of duplicate code we needed to write by
generating client-side code and had support for things like
Fragment Caching and Memoization enabled through
compile-time code generators. Wasabi 3, released to the public alongside FogBugz 7 in 2009, had modern .NET-style features like Interfaces, Enums, Try/Catch, and base classes. FogBugz 7 ran on Mono or Microsoft .NET instead of PHP or Classic ASP, generated by Wasabi 3's new CodeDOM-based code generator.
Maintenance Problems
As time wore on, our technical debt finally began to come due. Compilers like Wasabi and their associated runtime libraries are highly complex pieces of software. We hadnt open-sourced it, so this meant any investment had to be done by us at the expense of our main revenue-generating products. While we were busy working on exciting new things, Wasabi stagnated. It was a huge dependency that required a full-time developer -- not cheap for a company of our size. It occasionally barfed on a piece of code that was completely reasonable to humans. It was slow to compile. Visual Studio wasn't able to easily edit or attach a debugger to FogBugz. Just documenting it was a chore: FogBugz' Lead Developer at the time, Aaron Maenpaa, wrote '
Wasabi - The ??? parts', a book released internally in 2011 that attempted to document Wasabi's features and many quirks.
Life also got in the way. The people who wrote the original Wasabi compiler moved on for one reason or another. Some married partners who lived elsewhere; others went over to work on other products from Fog Creek. Of course, we couldnt just hire someone with prior knowledge of it. Well, except for me, as I had worked on it back in 2008, then left, but was re-hired four years later! But otherwise, all new hires had an extensive period of learning Wasabi, regardless of their previous experience.
Whats more, we werent living in a vacuum. Programming languages were of course improving outside of Fog Creek. Indeed, a lot of the features Wasabi added were things that C# 2.0 delivered out of the box, and there have been three more major versions of C# since then. Developers began to feel like their brilliant ideas were being constrained by the limitations of our little Wasabi universe.
So, alongside the work on FogBugz Ocelot, the latest version of FogBugz, we decided to begin removing Wasabi from FogBugz. There had been other proposals to do so in the past. When Wasabi 3 was started, one of the proposals was to do a one-shot compile to Python, but this was rejected in favor of the .NET backend. In 2011, Ted Unangst proposed "Project Soy Sauce", a combination of .NET Reflector and Perl scripts to turn the Wasabi binary output into nearly-readable C# source code. This time, we finally succeeded, because of
Roslyn, the newly open-sourced .NET Compiler Platform.
Pulling the Plug
So with the decision made, work began, which included:
- Step 1: change the output of Wasabi from a .NET assembly to C#. The old .NET code generator was based on System.CodeDom, an earlier Microsoft technology that generated valid C# and VB.NET code from a lowest-common-denominator abstract syntax tree. We added its C# output to a new project in the FogBugz solution with a precompilation step that invoked Wasabi with the requisite command-line arguments.
- Step 2: Switch from CodeDOM to Roslyn. It took 2 weeks to reach functional parity between CodeDomGenerator and RoslynGenerator, at which point we excised the former from Wasabi.
- Step 3: Improve the output of RoslynGenerator. So many little edge cases were improved here, including '!=', do/while, hex literals, logical and arithmetic negation, constants, switch statements, for loops with simple bounds, else if, lambdas, initializers, extraneous casts and parentheses, string literals, Wasabi Union types, and optional parameters.
- Step 4: Kill Wasabi. Commit the C# files that Wasabi output, and remove Wasabi and the .was files from the tip of the FogBugz repository. We also had to rewrite a little language file generator that had been generating Wasabi to instead generate C#.
Lessons Learned
Having our own programming language helped us grow FogBugz, a multi-decade project, from VBScript to PHP and JavaScript, and finally switch over to the brave new world of .NET. Wasabi made a lot of sense a decade ago, when we had a blooming code base we wanted to transplant into new ecosystems. It made sense in 2008 to switch the backend to .NET, which was easier to maintain and a faster platform on which to run FogBugz. Since we were consolidating into a single backend, it might have made sense at the time to do a one-time compile using a hand-written C# generator. Once Roslyn was released, it made even more sense to finish switching completely over to C#.
Building an in-house compiler rarely makes sense. It is much more economical to lean heavily on other peoples work and use standard, open technology. You must be prepared to not only create your own universe but to then nurture it long-term. We
weren't ignorant of this when the project began, but it's much easier to switch languages at the beginning of a project than halfway through. A homegrown compiler might make sense when trying to port to a new platform, which Wasabi helped us do several times.
In
the next post, we dive into the new Roslyn-based code generator in Wasabi that were open-sourcing. We hope that by doing so, anyone who is considering generating C# code programmatically will have guideposts to using Roslyn as their code generator.