Tuesday, November 1, 2011

Qt autotests - SFE vs KDE4

Qt comes with an extensive number of self tests and it's interesting to run them from time to time. In the FOSSqt.spec file of the kde4-solaris project, they can be turned on by defining run_autotests to 1, similarly in the SFEqt-gpp.spec of our SFE brethren.
A great number of them is failing on Solaris for various reasons - be it compiler issues, code bugs, or anything else, there's simply too many failures to demotivate anyone from trying to fix it.
So I thought it might be more interesting to compare the results from our build and SFE's - that way the difference would be more or less based on compiler differences (SFE uses gcc) and not anything else.

So here comes the comparison of the results for Qt 4.7.4:

Our Qt ran 491 tests, SFE's 489. There were 77 differences.

17 of the differences were irrelevant (difference in pointer address, test duration, slightly different messages, etc.). This includes tst_bic, tst_gestures, tst_maketestselftest, tst_modeltest, tst_q_func_info, tst_qcontiguouscache, tst_qdbusperformance, tst_qelapsedtimer, tst_qgraphicsitem, tst_qmainwindow, tst_qmenu, tst_qmutex, tst_qobjectperformance, tst_qscriptjstestsuite, tst_qscriptv8testsuite

36 cases where we fail and SFE passes: tst_collections,
tst_compiler (mere prejudice, we should try to override the stupid #if defined(Q_CC_SUN) there),
tst_mediaobject (something's wrong with our phonon),  
tst_q3sqlcursor, tst_q3sqlselectcursor, tst_qitemmodel, tst_qsql, tst_qsqldatabase, tst_qsqldriver, tst_qsqlquery, tst_qsqlquerymodel, tst_qsqlrelationaltablemodel, tst_qsqltablemodel, tst_qsqlthread, (hm, sqlite not set up properly),
tst_qaccessibility (a big fail here),
tst_qapplication (may be caused just by some difference in how the tests were run),
tst_qdatetime, tst_qdatetimeedit (caused by different environment when running the test - need to run it in the CET timezone),
tst_qdom, tst_qtextcodec, tst_qxmlsimplereader (mmm, asian encodings!),
tst_qgraphicsscene (this one is weird),
tst_qicoimageformat, tst_qicon, tst_qimage, tst_qimagereader, tst_qimagewriter, tst_qpixmap (deserves investigation, image format support),
tst_qmake (weird, need to have a look - perhaps bad mkspec used?)
tst_qnetworkconfiguration, tst_qnetworkconfigurationmanager, tst_qnetworksession (!)
tst_uic, tst_uic3 (likely QTBUG-21523)

2 tests that SFE builds but we don't: tst_qdbusmarshall (QTBUG-22106), tst_qtconcurrentmap (QTBUG-12987)

4 tests that we build but SFE doesn't: tst_guiapplauncher, tst_qglthreads, tst_qwidget_window, tst_qwidget

20 tests where SFE fails while we pass:
tst_qgraphicsview (not really a pass, but a few more tests are run in our build)
tst_qmessagebox (X Error: BadWindow)
tst_qstring (no wide character support in the g++ build? Strange ... )

It looks like the tests that included some performance measurements (tst_qcontiguouscache, tst_qdbusperformance, tst_qmutex, tst_qprocess) show better numbers for our bits, but I'd be far from interpreting it in that way. Both tests were run in a VirtualBox environment and the load on the host was quite different on the two occasions.

Webrev of the results (to save space only the udiff diffs were preserved)
The results bzipped
The tests source for reference