Testing, 1, 2, 3

If you undertake a Webkit project, at some point, you are probably going to want to test it. Call me silly, but it seems like the right thing to do.  Webkit has some regression testing frameworks that are available to you, and so I figured I would share my thoughts on these frameworks.

The first set of regression tests are the JavaScript core tests.  To launch these tests, you run a Perl script called run-javascriptcore-tests.  The Webkit source tree has an application in it called "jsc" which is needed to run the tests.  This little app initializes the JavaScript engine and runs the JavaScript code that is passed to it by the perl shell.  The jsc app requires the JavaScriptCore shared library, and it also requires that certain symbols are accessible inside that shared library.  Not really a big deal if you are on a Unix type system, but if you are doing a port on Windows, it is something to keep in mind, as the functions need to be exported properly.  Also, this app is compiled as part of a standard build if you are using  GTK, or QT, but for Windows and Mac, it is compiled by a script called build-jsc.  It will run the tests that are located in the source tree under the JavaScriptCore directory, and store the results in a file called actual.html.  It will tell you if you have any regressions on the command line after a run.  If you are porting to a new system, it is probably a good idea to try and run these tests, as it makes sure that your engine is as compatible with the JavaScript standard as the JavaScript engine that is in the Webkit source tree.

The second set of regression tests are the LayoutTests.  These tests require a little bit more of an investment in time.  These tests are launched from a script called run-webkit-tests.  It grabs tests from the directory LayoutTests, and passes them to and applciation called DumpRenderTree.  You will need to port the DumpRenderTree application.  The name pretty much says it all.  It links against the WebKit shared object, and it takes a page given to it by the launching script, creates an offscreen buffer which it passes to the WebView class, and then tells that WebView class to render the page.  It also has a set of controllers and delegates to manipulate the pages that are loaded so events can be tested inside of the page.  Once the test is complete, it will dump the contents of the page,  either as a render layout tree, or as the textual contents of the page, to a file and then the launching script compares that with a known result. You can also add an option to dump the contents to an image to see what the rendering actually looks like.  As I said earlier, this frameworks requires an investment of time.  You need to port DumpRenderTree and get that to work with your rendering system, on your target.  This port will also require you to implement the delegates and controllers that you interested in, such as the LayoutTestController, and the Editing delegate.  Also, there are approximately 14000 tests in the framework.  Roughly 40%  to 50% will need to have the results verified for your platform as they are considered to be platform specific¹.  You can skip tests using a skip file, and only test the areas that you are interested in, but you still may be looking at upwards of 6000 tests.  The other thing to keep in mind is that these tests are to be used for finding regressions moving forward.  The platform independent ones give you a sense that your implementation of the WebKit shared objects behaves like the other platforms, but  the rendering ones are for your platform only, and therefore you need that initial baseline to compare with.  For this reason, if you don't have a lot of time,  you may want to forgo these tests and set up a simpler rendering test framework.

1. Matching rendering between different GUI's is difficult.  For example, something rendered on Windows will look very different compared to something rendered on a Mac.

Rodney