Today I was notified that our custom-developed lab software suite had crashed.
Having a look, it turns out that it hadn't crashed, but using top showed in fact that it was using 100% CPU time.
First I did an strace on the process, which showed hundreds of lines looking like:
clock_gettime(CLOCK_MONOTONIC, {836478, 309454188}) = 0
clock_gettime(CLOCK_MONOTONIC, {836478, 311665927}) = 0
clock_gettime(CLOCK_MONOTONIC, {836478, 313925541}) = 0Then I decided to get a stacktrace from the process using gdb:
labbox1:~$ gdb --pid=17732
(gdb) bt
#0 0xb75b3e5d in memset () from /lib/tls/libc.so.6
#1 0x084dad4a in QTextEngine::LayoutData::reallocate ()
#2 0x084de989 in QTextEngine::attributes ()
#3 0x084e8c33 in QTextLine::layout_helper ()
#4 0x084ea124 in QTextLine::setLineWidth ()
#5 0x085211e5 in QTextDocumentLayoutPrivate::layoutBlock ()
#6 0x08527825 in QTextDocumentLayoutPrivate::layoutFlow ()
#7 0x0852544f in QTextDocumentLayoutPrivate::layoutFrame ()
#8 0x08525910 in QTextDocumentLayoutPrivate::layoutFrame ()
#9 0x08525b6c in QTextDocumentLayout::doLayout ()
#10 0x08525c10 in QTextDocumentLayoutPrivate::ensureLayoutedByPosition ()
#11 0x08525c98 in QTextDocumentLayout::blockBoundingRect ()
#12 0x0852ccb6 in QTextCursorPrivate::blockLayout ()
#13 0x0852ea62 in QTextCursorPrivate::setX ()
#14 0x0853289d in QTextCursor::deleteChar ()
#15 0x080b7fe8 in ScriptWindow::showOutput () at src/Debug.cpp:50
#16 0x080b846f in ScriptWindow::runOutput () at src/Debug.cpp:50
#17 0x0810afd9 in ScriptWindow::qt_metacall (this=0x8fcc560,
_c=QMetaObject::InvokeMetaMethod, _id=48, _a=0xbfd419c8)
at src/moc_Windows.cpp:248
#18 0x08a14892 in QMetaObject::activate ()
#19 0x08a14f54 in QMetaObject::activate ()
#20 0x089a4786 in QProcess::readyReadStandardOutput ()
#21 0x089a94db in QProcessPrivate::_q_canReadStandardOutput ()
#22 0x089a997e in QProcess::qt_metacall ()
#23 0x08a14892 in QMetaObject::activate ()
#24 0x08a14f54 in QMetaObject::activate ()
#25 0x08a31b61 in QSocketNotifier::activated ()
#26 0x08a1a90f in QSocketNotifier::event ()
#27 0x0832426f in QApplicationPrivate::notify_helper ()
#28 0x083296b9 in QApplication::notify ()
#29 0x08a00097 in QCoreApplication::notifyInternal ()
#30 0x08a258dd in socketNotifierSourceDispatch ()
#31 0xb78c1731 in g_main_context_dispatch () from /usr/lib/libglib-2.0.so.0
#32 0xb78c47a6 in g_main_context_check () from /usr/lib/libglib-2.0.so.0
#33 0xb78c4d27 in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0
#34 0x08a25a38 in QEventDispatcherGlib::processEvents ()
#35 0x083a9665 in QGuiEventDispatcherGlib::processEvents ()
#36 0x089ff0fd in QEventLoop::processEvents ()
#37 0x089ff37d in QEventLoop::exec ()
#38 0x08a014e2 in QCoreApplication::exec ()
#39 0x083249d7 in QApplication::exec ()
#40 0x0805d809 in main ()After the weekend, we'll analyze this at our leisure but if anyone has helpful comments, let me know.
Update: today I saw that Debian Package of the Day highlighted memstat which is a tool displaying virtual memory usage. Pity I didn't know this earlier, since it looks really useful.