Suggested Improvements to JSON

Today I'm publishing an essay about my suggested improvements to JSON. I really like JSON a lot, but I think that it could use a few tweaks to make it easier to use for the common man.

One question some have asked is: even if these changes to JSON were accepted, how would you transition existing systems? I would lean towards what I call the Nike approach, which is: "just do it." That is, start updating the JSON.parse() method in the browser to accept these extensions to JSON. What I am proposing is a strict superset of what's in RFC 4627 today (which you may recall claims that "[a] JSON parser MAY accept non-JSON forms or extensions"), so it would not break anyone who continued to use ES3 style JSON.

At first, you may think that sounds ridiculous -- how could we update the spec without versioning? Well, if you haven't been paying attention to the HTML5 movement, we are already doing this sort of thing all the time. Browser behavior continues to change/improve, and you just have to roll with the punches. It's not ideal, but yet the Web moves forward, and it's faster than waiting for standards bodies to agree on something.

Though if the Nike approach is too heretical, then I think that versioning is also a reasonable option. In the browser, a read-only JSON.version property could be added, though I don't imagine most developers would check it an runtime, anyway. Like most things on the web, a least-common-denominator approach would be used by those who want to be safe, which would ignore the version number. Only those who do user-agent sniffing on the server would be able to serve a slimmer JavaScript library, though that is already true for many other browser features today. I trust Browserscope and caniuse.com much more than any sort of formal specification, anyway.

Special thanks to Kushal, Dolapo, and Mihai for providing feedback on my essay.

What I learned from üjs

For April Fool's, I released a mock JavaScript library, üjs (pronounced "umlaut JS"). The library was "mock" in the "mockumentary" Spin̈al Tap sense, not the unit testing sense. It took me about a day to create the site, and I learned some interesting lessons along the way, which I thought I would share:

• In purchasing the domain name üjs.com, I learned about Punycode. Basically, when you type üjs.com into the browser, the browser will translate it to the Punycode equivalent, which is http://xn--js-wka.com/. On one hand, this is what prevents you from buying göögle.com and creating a giant phishing scheme; on the other hand, because most users are unfamiliar with Punycode, going to a legitimate domain like üjs.com looks like a giant phishing scheme due to the rewrite.
  
• I only promoted üjs through three channels: Twitter, Hacker News, and Reddit. On Twitter, I discovered that using a Punycode domain as your punchline really limits its reach because instead of tweeting about üjs.com, many tweeted about xn--js-wka.com (because that's what you can copy from the location bar after you visit the site), or (somewhat ironically) a URL-shortened version of the domain. I suspect more people would have followed the shared link if they could see the original domain name.
  
• According to Google Analytics, just over half of my traffic (50.04%) on April 1 was from Hacker News (where I made it to the front page!). Another 9.89% was from Reddit. Analytics also claims that only 1.57% of my traffic came from Twitter, though 29.76% of my traffic was "direct," so I assume that was from Twitter, as well. On April 1, I had 3690 visits, and then another 443 on April 2 (presumably from the eastern hemisphere who woke up on Saturday to an aftermath of Internet April Fools' pranks).
  
• GoDaddy was not able to do a domain search for üjs.com, presumably due to the non-ASCII characters. It turned out that name.com was able to, so I ended up going with them. (Presumably if I understood Punycode well enough at the outset of this project, I could have registered xn--js-wka.com through their site.)
  
• I found a discount code for name.com, so my total cost out of pocket for this project was $8.75 (my hosting fees are a sunk cost).
  
• Not that it was a substantial investment, but I was hopeful/curious whether I could break even in some way. I felt that traditional ads would have been a little over the top, so instead I decided to include two Amazon Associates links. Although I produced 350 clicks to Amazon (!), I failed to generate any conversions, so I did not make any money off of my ads.
  
• Chartbeat is really, really cool. It made it much more fun to watch all of the web traffic to üjs.com during the day. (I wish that I generally had enough traffic to make it worth using Chartbeat all the time!) I believe that I had 144 simultaneous visitors during the peak of üjs, and I was amazed at how dispersed the traffic was from across the globe.
  
• One thing that I did not realize is that Chartbeat does not do aggregate statistics. Fortunately, I set up Google Analytics in addition to Chartbeat, so I had access to both types of data.
  
• Response times were about 1s on average during peak traffic times. At first, I thought that was horrendously slow, but then I realized that there were a large number of requests coming from outside the US, which increased the average. Most of the requests from the US loaded in the low hundreds of milliseconds, which made me feel good about my hosting choice (who really is excellent, btw).
  
• The n̈ in Spin̈al Tap is not a valid Unicode character. Instead, it is displayed by printing an n followed by an umlaut, and what I can only assume is the kerning makes the umlaut display over the previous character. (The umlaut does not display correctly on my Chrome 10 on Windows, but it's fine on Linux.) Other characters, such as ü are valid Unicode and can be displayed with a single code point. Wikipedia has a list of letters that "come with umlauts," so I used those characters whenever possible on üjs.com, but for others, I had to use the "letter followed by an umlaut" trick.
  
• var n\u0308; is a valid JavaScript variable declaration, but var \u00fc; is not.
  
• Initially, the GitHub badge on my page did not link to anything, as I just wanted to satirize the trend I've been seeing in open source lately. Though after a request from a coworker, I imported all of the files I used to create üjs into GitHub. (Incidentally, when I tried to name the GitHub project üjs, they replaced the ü with a hyphen, so I renamed it umlautjs).
  

In the end, even though I did not make my $8.75 back, I had a really great time with this project. Although it's clear that some people on the Web don't enjoy April Fools, I think it is a nice opportunity to see some good satire. (My personal favorite was Angry Nerds by Atlassian.)

Further, satire is not completely frivolous: it is (an arguably passive-aggressive) way of making a point. In üjs, mine was this: these tiny JavaScript libraries do not help move the needle when it comes to JavaScript development from the community standpoint. Instead of contributing a tiny library, why not focus on contributing a tiny fix to a big library? Think about how many more people you will affect with a bug fix to jQuery than you will by publishing a single JavaScript file with your favorite four helper functions? Or if you are going to create a new library, make sure that you are doing something substantially different than what else is out there. For example, Closure and jQuery are based on different design principles. Both have their use cases, and they serve very different classes of development, so it makes sense for those separate projects to exist and grow.

If you have been following my blog, you probably know that I'm really big on "JavaScript programming in the large," which will be the subject of my talk at Google I/O this year. I hope to see you there!

Want to learn more about Closure? Pick up a copy of my new book, Closure: The Definitive Guide (O'Reilly), and learn how to build sophisticated web applications like Gmail and Google Maps!

Awesome New JavaScript Library!

Despite months of advocating Closure, I've finally given up and found a superior JavaScript library (and it's not jQuery!): http://üjs.com/

What are the most important classes for high school students to succeed in software engineering?

What are the most important classes for high school students to succeed in software engineering? That is the question that I try to answer in an essay of same name.

Also, this is the first essay I have written using NJEdit, which is the editing software that I built (and have since open sourced) in order to write Closure: The Definitive Guide. It helps me focus more on content while worrying less about formatting, though it still has a ways to go before becoming my "one click" publishing solution.

A unique feature of NJEdit is that when I produce the HTML to publish my essay, I also produce the DocBook XML version as a by-product! It's not a big selling point today, but if I ever want to publish anything to print again, I'll be ready! For open-source projects that are slowly creating HTML documentation that they hope to publish as a print book one day, NJEdit might be the solution.

And if it is, maybe someone will help me fix its bugs...

Want to learn more about Closure? Pick up a copy of my new book, Closure: The Definitive Guide (O'Reilly), and learn how to build sophisticated web applications like Gmail and Google Maps!

My latest Closure talks

Last Thursday, January 6, I had a doubleheader, as I gave one talk in the afternoon at an undergraduate MIT Web programming competition (6.470), and another talk in the evening at the Boston JavaScript Meetup. It was a long day (I took the train up from NYC that morning), but I got to talk to so many interesting people that I was far more energized at the end of the day than exhausted!

I cleaned up the slide decks from both talks so they are suitable for sharing. Because I was informed that the MIT programming competition had a lot of freshmen in it, I didn't go into as much detail about Closure as I normally would have because I didn't want to overwhelm them. (I ended up taking so much out of my "standard" talk that I ended up finishing ~30 minutes early, though that was likely a welcome relief to the students as I was the last speaker in a full day of lecture.) As you can see, the MIT talk is only 17 slides:



The talk I prepared for the Boston JS Meetup was the most technical one I have given to date. It was my first time presenting for a group that actually has "JavaScript" in the name, so it was refreshing not to have to spend the first 15 minutes explaining to the audience that JavaScript is a real language and that you can do serious things with it. By comparison, my second talk went much longer (about an hour and a half?), as there was a lot more material to cover as well as tons of great questions from the (very astute!) audience during my presentation:



The one thing that these presentations do not capture is the plovr demo that I have been giving during my talks. (This was also the first time that I demoed using plovr with jQuery, as I had just attempted using jQuery with plovr for the first time myself the night before. I have an open topic on the plovr discussion group about how to make plovr easier to use for jQuery developers, so please contribute if you have thoughts!) At some point, I'm planning to do a webcast with O'Reilly on Closure, so that might be a good opportunity to record a plovr demo that can be shared with everyone.

Want to learn more about Closure? Pick up a copy of my new book, Closure: The Definitive Guide (O'Reilly), and learn how to build sophisticated web applications like Gmail and Google Maps!

Web Content Wizard is Fixed AGAIN!

Back in 2006, I released this little tool for Google Calendar called the Web Content Wizard. With it, you can add your own "web content events" in Google Calendar. Unfortunately, there is no way to add such events so via the Google Calendar UI, so normally your only alternative is to do some gnarly web programming to talk to GData.

Things were going well until one day, the Web Content wizard just broke. The GData team changed something on their end which caused my poor little app to stop working. If I remember correctly, it was because they created this new thing about registering your web application. I learned of the problem because some users wrote in, and eventually sometime I finally found the time to debug the problem and fix it.

Then things were going well again for awhile, but then in late 2009 (when I had some newfound free time), I decided that it was finally time to migrate bolinfest.com from the Red Hat 9 server it had been running on since 2003! I had a VPS from RimuHosting (who rocks btw!) that I acquired in 2006 for my mitcalendars.com project with Mike Lambert, but I had never had the time to move my content from bolinfest.com over there. (Getting PHP 5 with its native JSON parsing methods was what finally put my over the edge, as I repeatedly failed at compiling them myself for the version of PHP 4 that was running on Shrike.)

As you can imagine, not everything worked when I did the migration in 2009. Honestly, I probably should have just gotten an even newer VPS then and migrated both servers, as now bolinfest.com is running on a dusty version of Ubuntu 8.04! (I'm running version 0.9.5 of Mercurial, which means that I don't even have the rebase extension!) Anyway, when I did the migration, I was far more preoccupied with migrating my conf files for Apache from 1.3 to 2.0 than I was with getting the Web Content Wizard to work again. Every endeavor of mine involving GData always involves a minimum of 3 hours work and a lot of cursing, and I just wasn't up for it.

Then today I got another polite email asking me why the Web Content Wizard wasn't working. I have amassed quite a few of these over the years, and I generally star them in my inbox, yet never make the time to investigate what was wrong. But for some reason, tonight I felt motivated, so I dug in and decided to debug it. I [wrongly] assumed that GData had changed on me again, so I focused my efforts there first. Surprisingly, the bulk of my time was spent getting the right error message out of PHP, as all it told me was that my request to the GData server was returning a 400. Now, I know GData is pretty lousy at a lot of things, but when it gives you a 400, in my experience, at least it gives you a reason!

I use cURL under the hood to communicate with GData from PHP, and that has never been pretty. I thought I remembered editing something in /etc/php.ini (which was now at /etc/php5/apache2/php.ini) to get cURL working correctly on the old bolinfest.com, but I could not remember what it was, nor did I appear to have documented it.

The key insight occurred when I changed this call to curl_setopt() to set the CURLOPT_FAILONERROR option to false:

curl_setopt($ch, CURLOPT_FAILONERROR, false);

Suddenly, my requests to GData contained a real error message:

The value following "version" in the XML declaration must be a quoted string

This did not make any sense because I was sure that I was sending well-formed XML to GData and it contained the <?xml version="1.0"?> preamble as it should. I decided to print it back out to make sure that things were being sent as I expected, but when I did so, I noticed that all of my double-quotes were inexplicably escaped with backslashes. That explained why the error message was complaining about well-formed XML, but not where the backslashes came from.

Then it dawned on me: php.ini has a bunch of stupid settings with the word magic in them. Sure enough, I did a search for [php HTTP_POST_VARS escape quotes], which eventually led me to the documentation for the PHP function stripslashes(), which finally led me to the documentation for magic_quotes_gpc, which basically says: "Because the average PHP developer is so likely to Little Bobby Tables himself, we put the coding behaviors for adults on a high shelf where little PHP developers can't reach them and automatically backslash escape quotes because said developers are so likely to cram that string directly into a SQL statement without escaping it."

I immediately changed the line in php.ini to magic_quotes_gpc = Off, restarted Apache, and all of a sudden, the Web Content Wizard was fixed AGAIN!

At least until my next server upgrade...

Want to learn more about Closure? Pick up a copy of my new book, Closure: The Definitive Guide (O'Reilly), and learn how to build sophisticated web applications like Gmail and Google Maps!

Released a new version of plovr so I can show it off at my two Closure lectures this Thursday

I just released a new version of plovr in time for the two talks on Closure I am giving this Thursday. The first is a guest lecture for an MIT web programming competition: 6.470. My understanding is that the course attracts a lot of freshmen, so my hope is that plovr can help those who are new to JavaScript leverage the Closure Compiler to help them identify their JavaScript errors (of which there are likely to be many!). This motivated me to fix some annoying issues in plovr that have been lingering for awhile (which the early adopters have been nice enough to put up with for far too long :).

The second talk I am going to give is part of the Boston JS Meetup, which I am very excited to attend for the first time. I'm looking forward to talking to the group and showing off what both Closure and plovr can offer to those in industry.

Also, up until now, I have been making decisions autonomously about new plovr features and config options because I did not have a good way to gather feedback from users other than the bug reports I received. To that end, I finally decided to create a Google Group for plovr where users can help influence plovr development and ask questions about how to use plovr. As I noted in the Closure Library group, I would like to extend plovr to help with CSS management and developing against jQuery (among other things), and I believe these features will only be useful to others if prospective users participate in the discussion. We'll see how all of that works out.

Finally, one interesting engineering challenge that has plagued plovr since its inception (but has finally been slayed!) is the question of how to manage its dependency on the Closure Library, Closure Compiler, and Closure Templates. With a considerable amount of help from Ilia Mirkin (and a lot of SVN and Hg gymnastics), I finally have a system that I am happy with. Basically, it requires creating a branch for each dependent project to which updates are periodically added and then merged into the main repository. As this is a general problem for open-source projects, I documented it thoroughly on the plovr wiki.

Although it was tricky to set up, it is now possible to build plovr without checking out the other Closure projects. (Previously, a new version of plovr would include my own local checkouts of the closure-library, closure-compiler, and closure-templates repositories, which were at somewhat arbitrary revisions with possibly even more arbitrary/undocumented patches of my own in them.) Further, because plovr has its own clones of the Closure projects in its repository, it is far easier to modify Closure code if it needs to be edited for plovr. For example, it was now trivial to make the constructor of com.google.javascript.jscomp.Result public, whereas previously I would have had to create a completely separate binary for the Closure Compiler with this one modification and then replace it with the one in plovr's lib directory. Further, I would forever have to keep my checkout of the Closure Compiler with the modification around so that I could make further modifications in the future while no outside developer would have any insight into that process -- no more!

This also allowed me to clean up the build process for plovr and remove some unnecessary and/or duplicate dependencies. The release of plovr I did back in December was 11.7 MB, but the new release is down to 8.4 MB, which is the smallest plovr jar I have ever published! Things are definitely moving in the right direction for the project.

Want to learn more about Closure? Pick up a copy of my new book, Closure: The Definitive Guide (O'Reilly), and learn how to build sophisticated web applications like Gmail and Google Maps!