Using LTTng

From OMAPpedia

Revision as of 21:56, 7 June 2011 by Greenscientist (Talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

The following wiki how to use LTTng (Linux Trace Toolkit - Next Generation). It describes its different functionnality and components, and to make it work on OMAP.

Note: The procedure was tested under linux-omap-2.6 kernel at v2.6.33 .


[edit] What is LTTng?

LTTng, the Linux Trace Toolkit Next Generation, is a project that aims at producing a highly efficient full system tracing solution.

It is composed of several components to allow tracing of the kernel, of userspace, trace viewing and analysis and trace streaming.

LTTng is open source software. It is being actively developed with the community.

LTTng itself is a kernel patch. It is used with

LTTV is designed to handle huge traces (10 GB and more). It has a plugin-based architecture and provides both graphical and text-mode user interfaces. LTTng includes a set of kernel instrumentation points useful for debugging a wide range of bugs, that are otherwise extremely challenging. These include, for example, performance problems on parallel systems and on real-time systems. Custom instrumentation is easy to add. LTTng is designed for minimal performance impact and has a near zero impact when not tracing. LTTng has at least basic support for all Linux-supported architectures.

[edit] Installation from sources

[edit] Prerequisites

You have to install the following libraries and programs to compile LTT-ng and his components on the host PC.

$ sudo apt-get install libstdc++6 libgtk2.0-dev libglib2.0-dev libpopt-dev libpango1.0-dev libc6-dev

Check the compatibilities between LTTng, ltt-control, LTTV and the Linux Kernel at Compatibility List

[edit] Getting and apply LTTng patches

Note: if you use the patches above, you can jump the wget instruction

In the kernel sources directory, execute the following instructions.

$ wget
(see for package listing)
$ tar jxvf patch-2.6.X-lttng-0.x.xx.tar.gz
$ mv patch-2.6.X-lttng-0.x.xx patches
$ quilt push -a

[edit] Building a LTTng kernel

$ make CROSS_COMPILE=arm-none-linux-gnueabi- ARCH=arm distclean
$ make CROSS_COMPILE=arm-none-linux-gnueabi- ARCH=arm omap_zoom2_defconfig 
$ make CROSS_COMPILE=arm-none-linux-gnueabi- ARCH=arm xconfig
General setup -> CONFIG_LTT
General setup -> Linux Trace Toolkit Next Generation -> CONFIG_LTT_TRACEPROBES
General setup -> Linux Trace Toolkit Next Generation -> CONFIG_LTT_DEBUG_EVENT_SIZE
General setup -> Linux Trace Toolkit Next Generation -> CONFIG_LTT_USERSPACE_EVENT
General setup -> Linux Trace Toolkit Next Generation -> CONFIG_LTT_STATEDUMP
  • Note: Some of the previous options will be already activated with a dot in xconfig. It means that they will be compile as module, but for more simplicity check them a second time to compile them directly in the kernel.
$ make CROSS_COMPILE=arm-none-linux-gnueabi- ARCH=arm uImage

(For further information about the compilation of the kernel see Linux OMAP Kernel Project)

[edit] Editing the system wide configuration

You must activate debugfs and specify a mount point. This is typically done in fstab such that it happens at boot time. If you have never used DebugFS before, these operation would do this for you :

# mkdir /mnt/debugfs
# cp /etc/fstab /etc/fstab.lttng.bkp
# echo "debugfs         /mnt/debugfs    debugfs rw              0       0"  >> /etc/fstab
# mount /mnt/debugfs 

[edit] Getting and installing the ltt-control package

$ wget
$ tar zxvf ltt-control-0.x-xxxx20xx.tar.gz
$ cd ltt-control-0.x-xxxx20xx
$ ./configure --host=arm-none-linux-gnueabi --prefix=/tmp/mmc2
$ make
$ sudo -s
# make install

These commands will install ltt-control into the mmc. To finish the installation, type the following command on the board.

# export LTT_DAEMON=/bin/lttd

[edit] Getting and installing the LTTV package (on the visualisation machine)

$ wget
$ tar zxvf lttv-0.x.xx-xxxx20xx.tar.gz
$ cd lttv-0.x.xx-xxxx20xx
$ ./configure
$ make
$ sudo -s
# make install
(run ldconfig to ensure new shared objects are taken into account)
# ldconfig

[edit] Using LTTng

# ltt-armall

[edit] Use text mode LTTng to control tracing

The tracing can be controlled from a terminal by using the lttctl command.

# lttctl -C -w /tmp/trace1 trace1
# lttctl -D trace1

[edit] Tracing in "Hybrid" mode

Another mode can be use to trace data : hybrid. It can be especially useful when studying big workloads on a long period of time.

When using this mode, the most important, low rate control information will be recorded during all the trace by lttd (i.e. process creation/exit). The high rate information (i.e. interrupt/traps/syscall entry/exit) will be kept in a flight recorder buffer (now named flight-channelname_X).

The following lttctl commands take an hybrid trace :

Create trace channel, start lttd on normal channels, start tracing:
# lttctl -C -w /tmp/trace2 -o channel.kernel.overwrite=1 trace2

Stop tracing, start lttd on flight recorder channels, destroy trace channels :
#lttctl -D -w /tmp/trace2 trace2

Each "overwrite" channel is flight recorder channel.

[edit] Tracing in flight recorder mode

The flight recorder mode writes data into overwritten buffers for all channels, including control channels, except for the facilities tracefiles. It consists of setting all channels to "overwrite". it will dump only the last buffers to disk.

The following lttctl commands take a flight recorder trace :

lttctl -C -w /tmp/trace3 -o channel.all.overwrite=1 trace3
lttctl -D -w /tmp/trace3 trace3

[edit] Analyse Trace Data with LTTV

The trace are composed of several file which contain a lot of metadata not understanding by a human beeging. That's why we have to use LTTV to analyse all this among of data. LTTV can be use by two different ways:

[edit] LTTV Text mode

The text mode will formatted the data to understandable text file which contains every informations. These files can be very heavy, but still usefull if we need to looking for something into the trace with a grep command.

A simple trace dump in text format is available with :

$ lttv -m textDump -t /tmp/trace

If you use the --help option on the textDump module, you will see all the detail about the switches that can be used to show per cpu statistics and per process statistics.

[edit] Use LTTV GUI mode


This is the main part of LTTV, it have a module architecture which allow to enable or disable some library options Type the following command to launch LTTV

$ lttv-gui

A user guide of LTTV is available here

[edit] Adding new instrumentations with the markers

[edit] Adding kernel instrumentation

See Documentation/markers.txt and Documentation/trace/tracepoints.txt in your kernel tree.

Also see ltt/probes/ for LTTng probe examples.

[edit] Adding userspace instrumentation

The easy quick-and-dirty way to perform userspace tracing is currently to write an string to /mnt/debugfs/ltt/write_event.

echo "some text to record" > /mnt/debugfs/ltt/write_event

It will appear in the trace under event :

  • channel : userspace
  • event name : event

LTTng has a specific tools for tracing userspace activity (LTTng-UST) , but this one works only on x86-32 and x86-64 architecture.

[edit] LTTng on Android

Note: For more informations about Android see here.

A series of patch was developed to enable LTTng on Android. After applying the patches, LTTng works the same way as the versions above. Unfortunately, some options of LTTng (PM support for example) are not enabled, but the main option of tracing works with good performance.

LTTng for Android can work only with old versions of lttcontrol and LTTV (According to the Compatibility List). The following tools was tested, and works with this version of LTTng.

[edit] Benchmark

As every measuring tool, LTTng brings an overhead for every function call by the kernel. To get a good idea of the overhead, three benchmark was tested

[edit] Hackbench

Hackbench of LTTng

According to this graph, we can see that LTTng brings an overhead when the trace is active, its brings an average of 90% of overhead for this task which implies a lot of scheduler switch. The most important point is that LTTng practically doesnt influence the system when the trace is unactive (only 3% of overhead for this example). Comparing to FTrace event tracer, which trace similar things, the overhead of LTTng is much smaller because Ftrace event tracer can bring until 230% percent of overhead, and also Ftrace has always a non negligible overhead even when the tracer is off.

[edit] Uncompressing a large .tar.gz file

Decompression of a large tar.gz file

This test consists to decompressing a large tar.gz file (an ubuntu minimal filesystem ~90MB) and measuring the time elapse to do it. During this test, we can see that the overhead of LTTng is low in any case. The Hybrid Mode and mostly the Flight record mode have a better overhead. The explication come from the fact that these mode don't record every trace into file but some of them are written only in a stamp

[edit] For more Information

Personal tools