What's the best way to run integration tests on an AJAX-heavy website?
I have been toughening up the pencilcode.net deployment environment. (git repos here.) While the pencilcode server is an nginx webserver backed by a small python-uwsgi app server, the pencilcode development environment is 100% javascript, with a node.js and npm and grunt-based build, and a small express.js test server that proxies backend calls to the production server.
If you think that's just a lot of hipster coder buzwords, wait... there's more.
This morning I broke pencilcode.net due to a code change: while refactoring the editor source code to use proper require.js organization, I messed up by bringing in an old version of seedrandom that wasn't compatible with AMD loading. Seedrandom is my own library, and so it was doubly my own fault! The result was that all password checking was broken on the site for a few hours. The bug took about 10 minutes to fix: I just had to use bowercopy to upgrade seedrandom to its latest bower package, then run the requirejs optimizer, then redeploy.
But the real bug? I don't have any integration tests on pencilcode.net. I should have never pushed broken passwords in the first place. While I have some little unit tests, I really need end-to-end integration tests that bring up the whole website, browse it with a headless browser, and click around authenticating and editing and running and saving, to make sure it all works as expected.
The 2010 way of doing this used to be a big system like Selenium. But it feels like in 2013 I should be able to do this in lighter-weight way. I have seen that grunt-phantomjs works pretty well with QUnit unit tests. But almost all the javascript-based tools I've seen are designed for tiny unit tests.
How do I run tests that exercise the whole system end-to-end? Maybe something like travist's jquery.go.js? Has anybody found the right way to do integration tests in the node.js world?
I'm setting up a guide for contributing to PencilCode.
There are now integration tests. In about 20 seconds, the "grunt test" command starts a local development server, starts a headless webkit, and tests most of the core functionality on the website, including browsing users and directories, loading, editing, running, saving, and deleting programs, and using passwords to log in. You get this all for free by just installing node.js and the grunt build tool, and building the source.
The integration tests look like this: edit_code.js - they are low-level but surprisingly clean and quick. The tests are designed to be run by mocha (a command-line javascript test runner), and they depend on node-phantom-simple, which is the lightest-weight headless webkit I could drive from a node.js build environment. The Gruntfile sets up everything. One of the curious things is that the development server actually runs as a proxy server. This allows it to intercept test-browser requests to full DNS names even though it doesn't own the DNS name.
Why Tests are Awesome
The test setup is designed to maximize developer productivity: the default test target runs tests very similar to production, with all the code compiled and minified, but the "devtest" target runs the same tests directly against unminified unoptimized source code, for easier debugging.
The beauty of a good integration test is that the code can now be aggressively changed without fear that the change will break something important. If some change breaks something, we can know in 20 seconds.
Right now my integration tests run quickly, because my test matrix is small. As the test matrix grows larger, many open source projects use jquery-turtle as well; then it will be time to refactor the code so some interesting features can be added.
The latest version of Chrome has a bad bug that seems specific to the GPU in my (Samsung series 9) laptop. It actually took me a couple days to notice the problem and realize it was Chrome's fault and not the fault of the underlying website.
Here is the bug: when you visit a fast website that has no images (and no javascript-rendered changing content), Chrome does not render anything. It just leaves the page blank. It will render as soon as you scroll or cause something to redraw.
My laptop does not have an exotic configuration - it is a 2012-era windows laptop with a very vanilla Windows 7 setup, so I suspect many people may be affected by the bug.
The funny thing, the bug is something you may not have noticed because almost every webpage we might visit day-to-day contains an image. And pages that do not have images - well, as soon as you scroll, they render, so you might have thought that the webpage was slow and then popped in when you scrolled.
If you use Chrome, try it out on your computer. Does it happen to you? To test, here are a couple links to webpages that have no images. If they render for you, try pressing "refresh" to see if they render once they're cached and fast:
http://www.htmlandcssbook.com/code-samples/chapter-01/example.html
http://davidbau.com/bugreport/simplest-document-broken.html
If you see the problem, put in information about your hardware, and vote for the bug to be fixed here:
(Update: found an old issue that appears to have reported the same problem in beta - https://code.google.com/p/chromium/issues/detail?id=325309)
I spent a day last week at Worcester Technical High School with all their CS students, using Pencil Code as a teaching tool.
Vocational students are motivated by real-world applications, and the teachers at WTHS are amazing. They bring their entire group of students through four years of rigorous CS classes, ending with AP computer science in their Senior year. "There is a difference," the teachers explain, "between offering CS exposure and teaching mastery. We teach mastery."
For this group of CS-focused students who have been learning Java, ASP.NET, and HTML, Pencil Code is a terrific tool. The instant feedback lets them apply and experiment with difficult concepts quickly. And because it is so open, Pencil Code is lets them assemble concepts from varied areas.
Here are the worksheets we used in Worcester:
Combining Three Languages using HTML
Exploring Recursion with Fractals
Using Subclasses to make Moving Sprites
In all three of these lessons, the turtle is a starting point, but it is just a stepping stone into real-world applications and deeper concepts.
The day was terrific. We found that we taught some unexpected lessons. The teachers pointed out that the indent-based syntax of CoffeeScript (and the instant-error checking in the IDE) connected with several kids for the first time who now could really "get" how scoping works. Some kids never indent their blocks, but with CoffeeScript, they have to! And surprise - once it is indented, they can see clearly how it works.
After the day, the AP students in the group suggested "We should start with Pencil Code!"
I am holding hackathons to build more lessons and materials that take advantage of the Pencil Code environment, and to improve the tool itself based on what we are seeing in classrooms. The first hackathon is on February 14 - sign up at hack.pencilcode.net.