Qt comes with an extensive number of self tests and it's interesting to run them from time to time. In the FOSSqt.spec file of the kde4-solaris project, they can be turned on by defining run_autotests to 1, similarly in the SFEqt-gpp.spec of our SFE brethren.
A great number of them is failing on Solaris for various reasons - be it compiler issues, code bugs, or anything else, there's simply too many failures to demotivate anyone from trying to fix it.
So I thought it might be more interesting to compare the results from our build and SFE's - that way the difference would be more or less based on compiler differences (SFE uses gcc) and not anything else.
So here comes the comparison of the results for Qt 4.7.4:
Our Qt ran 491 tests, SFE's 489. There were 77 differences.
17 of the differences were irrelevant (difference in pointer address, test duration, slightly different messages, etc.). This includes tst_bic, tst_gestures, tst_maketestselftest, tst_modeltest, tst_q_func_info, tst_qcontiguouscache, tst_qdbusperformance, tst_qelapsedtimer, tst_qgraphicsitem, tst_qmainwindow, tst_qmenu, tst_qmutex, tst_qobjectperformance, tst_qscriptjstestsuite, tst_qscriptv8testsuite,
36 cases where we fail and SFE passes: tst_collections,
tst_compiler (mere prejudice, we should try to override the stupid #if defined(Q_CC_SUN) there),
tst_mediaobject (something's wrong with our phonon),
tst_q3sqlcursor, tst_q3sqlselectcursor, tst_qitemmodel, tst_qsql, tst_qsqldatabase, tst_qsqldriver, tst_qsqlquery, tst_qsqlquerymodel, tst_qsqlrelationaltablemodel, tst_qsqltablemodel, tst_qsqlthread, (hm, sqlite not set up properly),
tst_qaccessibility (a big fail here),
tst_qapplication (may be caused just by some difference in how the tests were run),
tst_qdatetime, tst_qdatetimeedit (caused by different environment when running the test - need to run it in the CET timezone),
tst_qdom, tst_qtextcodec, tst_qxmlsimplereader (mmm, asian encodings!),
tst_qgraphicsscene (this one is weird),
tst_qicoimageformat, tst_qicon, tst_qimage, tst_qimagereader, tst_qimagewriter, tst_qpixmap (deserves investigation, image format support),
tst_qmake (weird, need to have a look - perhaps bad mkspec used?)
tst_qmovie
tst_qnetworkconfiguration, tst_qnetworkconfigurationmanager, tst_qnetworksession (!)
tst_qwebpage
tst_uic, tst_uic3 (likely QTBUG-21523)
2 tests that SFE builds but we don't: tst_qdbusmarshall (QTBUG-22106), tst_qtconcurrentmap (QTBUG-12987)
4 tests that we build but SFE doesn't: tst_guiapplauncher, tst_qglthreads, tst_qwidget_window, tst_qwidget
20 tests where SFE fails while we pass:
tst_qabstractslider
tst_qcolumnview
tst_qcompleter
tst_qdialog
tst_qeventloop
tst_qfilesystemmodel
tst_qgl
tst_qglbuffer
tst_qgraphicsproxywidget
tst_qgraphicstransform
tst_qgraphicsview (not really a pass, but a few more tests are run in our build)
tst_qmdiarea
tst_qmessagebox (X Error: BadWindow)
tst_qprocess
tst_qreadwritelock
tst_qscriptengine
tst_qstring (no wide character support in the g++ build? Strange ... )
tst_qthread
tst_qtimeline
tst_xmlpatternsdiagnosticsts
It looks like the tests that included some performance measurements (tst_qcontiguouscache, tst_qdbusperformance, tst_qmutex, tst_qprocess) show better numbers for our bits, but I'd be far from interpreting it in that way. Both tests were run in a VirtualBox environment and the load on the host was quite different on the two occasions.
Webrev of the results (to save space only the udiff diffs were preserved)
The results bzipped
The tests source for reference
A great number of them is failing on Solaris for various reasons - be it compiler issues, code bugs, or anything else, there's simply too many failures to demotivate anyone from trying to fix it.
So I thought it might be more interesting to compare the results from our build and SFE's - that way the difference would be more or less based on compiler differences (SFE uses gcc) and not anything else.
So here comes the comparison of the results for Qt 4.7.4:
Our Qt ran 491 tests, SFE's 489. There were 77 differences.
17 of the differences were irrelevant (difference in pointer address, test duration, slightly different messages, etc.). This includes tst_bic, tst_gestures, tst_maketestselftest, tst_modeltest, tst_q_func_info, tst_qcontiguouscache, tst_qdbusperformance, tst_qelapsedtimer, tst_qgraphicsitem, tst_qmainwindow, tst_qmenu, tst_qmutex, tst_qobjectperformance, tst_qscriptjstestsuite, tst_qscriptv8testsuite,
36 cases where we fail and SFE passes: tst_collections,
tst_compiler (mere prejudice, we should try to override the stupid #if defined(Q_CC_SUN) there),
tst_mediaobject (something's wrong with our phonon),
tst_q3sqlcursor, tst_q3sqlselectcursor, tst_qitemmodel, tst_qsql, tst_qsqldatabase, tst_qsqldriver, tst_qsqlquery, tst_qsqlquerymodel, tst_qsqlrelationaltablemodel, tst_qsqltablemodel, tst_qsqlthread, (hm, sqlite not set up properly),
tst_qaccessibility (a big fail here),
tst_qapplication (may be caused just by some difference in how the tests were run),
tst_qdatetime, tst_qdatetimeedit (caused by different environment when running the test - need to run it in the CET timezone),
tst_qdom, tst_qtextcodec, tst_qxmlsimplereader (mmm, asian encodings!),
tst_qgraphicsscene (this one is weird),
tst_qicoimageformat, tst_qicon, tst_qimage, tst_qimagereader, tst_qimagewriter, tst_qpixmap (deserves investigation, image format support),
tst_qmake (weird, need to have a look - perhaps bad mkspec used?)
tst_qmovie
tst_qnetworkconfiguration, tst_qnetworkconfigurationmanager, tst_qnetworksession (!)
tst_qwebpage
tst_uic, tst_uic3 (likely QTBUG-21523)
2 tests that SFE builds but we don't: tst_qdbusmarshall (QTBUG-22106), tst_qtconcurrentmap (QTBUG-12987)
4 tests that we build but SFE doesn't: tst_guiapplauncher, tst_qglthreads, tst_qwidget_window, tst_qwidget
20 tests where SFE fails while we pass:
tst_qabstractslider
tst_qcolumnview
tst_qcompleter
tst_qdialog
tst_qeventloop
tst_qfilesystemmodel
tst_qgl
tst_qglbuffer
tst_qgraphicsproxywidget
tst_qgraphicstransform
tst_qgraphicsview (not really a pass, but a few more tests are run in our build)
tst_qmdiarea
tst_qmessagebox (X Error: BadWindow)
tst_qprocess
tst_qreadwritelock
tst_qscriptengine
tst_qstring (no wide character support in the g++ build? Strange ... )
tst_qthread
tst_qtimeline
tst_xmlpatternsdiagnosticsts
It looks like the tests that included some performance measurements (tst_qcontiguouscache, tst_qdbusperformance, tst_qmutex, tst_qprocess) show better numbers for our bits, but I'd be far from interpreting it in that way. Both tests were run in a VirtualBox environment and the load on the host was quite different on the two occasions.
Webrev of the results (to save space only the udiff diffs were preserved)
The results bzipped
The tests source for reference
No comments:
Post a Comment