Debugging RPMsg

From OMAPpedia

Revision as of 22:45, 27 April 2012 by Jgutierrez (Talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search


[edit] Kernel Debugging

The RPMsg Kernel debugging relies on the fact that DebugFS is setup properly. The following sub-sections give out necessary information. All the below commands may not work if you do not have root permissions on the target device.

[edit] Remote Processor Traces

These traces enable the host processor (MPU) to see the remote processor traces through shared memory. Each of the remote processors write traces into a corresponding circular shared memory buffer, and user can display them through the debugfs. Please look up the section below for adding traces on the remote processor side.

[edit] Current Traces

# cat /debug/remoteproc/omap-rproc.1/trace0
# cat /debug/remoteproc/omap-rproc.1/trace1
bash$ adb root
bash$ adb shell "cat /debug/remoteproc/omap-rproc.1/trace0"
bash$ adb shell "cat /debug/remoteproc/omap-rproc.1/trace0" > my-current-trace0.log

On Ducati Core 0/SysM3,

 [      0.000] DEH: Watchdog started
 [      0.000] CORE0 starting..
 [      0.000] 18 resources at 0x80062000
 [      0.000] copyTask 50: Entered...:
 [      0.000] VirtQueue_startup: buf_addr address of 0xa0000000 received
 [      0.000] registering rpmsg-client-sample service on 50 with HOST
 [      0.000] copyTask 51: Entered...:
 [      0.001] registering rpmsg-client-sample service on 51 with HOST
 [      0.001] serviceMgr: started on port: 60
 [      0.001] registering rpmsg-omx service on 60 with HOST

On Ducati Core 0/AppM3,

 [      0.000] DEH: Watchdog started
 [      0.000] [ERR=5] src/mmsw_platform.c:[152]:Ducati Image  xxxxxxxxxxxx (Compile Time: Nov 10 2011 17:50:08)xxxxxxxxxxxx 
 [      0.004] serviceMgr: started on port: 60
 [      0.004] registering rpmsg-omx service on 60 with HOST
 [      0.004] [t=0x001acd03] ti.ipc.rpmsg.MessageQCopy: MessageQCopy_send: no object for endpoint: 53

[edit] Previous Traces

In case of an MMU Fault or Exception on the remote processors, RPMsg performs a recovery of the sub-system and reloads & restarts the processors resetting all the trace buffers. However, before the recovery is kicked in, the current traces are preserved in another debugfs file for inspection.

 # cat /debug/remoteproc/omap-rproc.1/trace0_last
 # cat /debug/remoteproc/omap-rproc.1/trace1_last

[edit] Kernel Runtime Traces

Most of the rpmsg kernel modules use dev_dbg functions for giving out useful information. This will allow dynamic trace control, and can be used to print only the desired traces from interested modules, functions etc.

[edit] Ducati Hibernation Control

The RPMsg driver is runtime-pm enabled, and as such useful information about the power management status is exported through the sysfs.

# ls -l /sys/devices/platform/omap/omap-rproc.1
lrwxrwxrwx root     root              2000-01-02 00:40 driver -> ../../../../bus/platform/drivers/omap-rproc
-r--r--r-- root     root         4096 2000-01-02 00:40 modalias
drwxr-xr-x root     root              2000-01-02 00:30 power
lrwxrwxrwx root     root              2000-01-02 00:40 subsystem -> ../../../../bus/platform
-rw-r--r-- root     root         4096 2000-01-02 00:30 uevent

The power directory contains the following sysfs entries, that gives detailed info about the power management status of the remote processor device.

The following shows an example of the values from these files:

# cd /sys/devices/platform/omap/omap-rproc.1/power
# cat autosuspend_delay_ms
# control
# runtime_active_time
# runtime_status
# runtime_suspended_time

Ducati Hibernation needs to be turned off if one were to perform stop-mode debugging using a JTAG. This can be done by writing into the control file. This will wake up the Ducati, if it already suspended, and will not allow Ducati to hibernate.

# pwd
# echo on > control
# cat runtime_status

You should be able to verify this by reading the runtime_suspended_time and runtime_active_time. The former will stop incrementing, while the latter will continue incrementing.

To re-enable hibernation, simply write auto into the control file.

# echo auto > control

Ducati will again be suspended after the autosuspend_delay_ms time has elapsed, if there is no activity.

[edit] Resource Usage

The Host Processor running Linux is in charge of the device power management, and is responsible for turning on/off various clocks, putting different constraints, performing OPP switching, Voltage and Frequency Scaling etc. The Remote Processors have to request the host processor to enable/disable clocks before accessing/using any of the peripherals. They could also put constraints (bw, latency, frequency) to achieve desired metrics. These resources are tracked on the MPU side, and are released in the event of a remote processor crash.

The requests from remote processors are handled through a rpmsg resmgr kernel module, and at any point of time, the information about the current active resource requests, is published through a debugfs file, resources. Typically, there is one folder for each remote processor. rpmsg2 in below example is typically for Ducati AppM3 core.

# ls /debug/rpmsg_resmgr

This list of resources can then be displayed with the following command

# cat /debug/rpmsg_resmgr/rpmsg2/resources

The following is an example of resources in use when playing a 1080p video clip:

Resource Name:IPU
Source address:102

Resource Name:SL2IF
Source address:102

Resource Name:IVA SEQ1
Source address:102

Resource Name:IVA SEQ0
Source address:102

Resource Name:IVA HD
Source address:102

The display gives out the following information

This information can be useful to:

[edit] Ducati Crash Information

When Ducati crashes due to an MMU fault or an exception, the BIOS-side Exception Handler prints the context information to the trace buffer, and also dumps the register state to an exception buffer. This exception data provides information such as PC, LR, Stack Pointer to trace it back to function that has caused the MMU fault or an exception.

Please look the remote processor traces for this crash information. The raw exception data buffer is also published through a debugfs file, cdump. This data can be retrieved by executing

For Core0/SysM3, 
 # cat /debug/remoteproc/omap-rproc.1/cdump0
For Core1/SysM3, 
 # cat /debug/remoteproc/omap-rproc.1/cdump1

NOTE: This context information is only very consistent for an internal Exception. The context information is prepared by the remote processor's exception handler, and is processed as an exception interrupt is raised back to the corresponding processor core. A Ducati WatchDog and MMU Fault are initially triggered directly on the host-processor, and as such sometimes the remote processor may not have processed the context dump before the host processor has printed out the information and triggered the recovery.

[edit] Ducati Core Dump Info

When a remote processor has crashed, there is an ability to be able to temporarily halt the recovery and retrieve the core dump file used by Ducati. This is useful for performing any post-mortem analysis using a customized gdb tool. This option is currently meaningful only for Android.

To enable halt-on-crash,

bash$  adb shell "echo enable > /debug/remoteproc/omap-rproc.1/core"

NOTE: This needs to be done before the ducati can crash, otherwise recovery will continue. This option is disabled by default.

Once the crash happens, use adb pull

bash$ adb pull /debug/remoteproc/omap-rproc.1/core my-ducati-core

Once the core dump file has been retrieved, execute the following command to proceed with the recovery. turn off remoteproc halt-on-crash and proceed with the recovery, once you have retrieved the core dump, do

bash$ adb shell "echo disable > /debug/remoteproc/omap-rproc.1/core"

The "echo disable" command will turn off the remoteproc halt-on-crash for subsequent runs as well. It would behave as if the option has not been enabled. To keep the halt-on-crash enabled for subsequent runs, but to continue to proceed with the recovery for the current crash, execute

bash$ adb shell "echo continue > /debug/remoteproc/omap-rproc.1/core"

[edit] Ducati Binary Version Info

The Ducati firmware binary has always to be named ducati-m3.bin for the firmware loading to work. Since the binary can be built from any two base images, version information about the baseimages and sysbios-rpmsg has been published through a debugfs entry, version. The firmware binary has the git tag and commit id information embedded in it during the build, and it gives version details about about SYS/BIOS tools, like XDC, BIOS, Code Generation Tools, Framework Components, Codec Engine, OSAL etc., used for building the respective baseimage.

Follow command displays the version/commit-id information for rpmsg and ducati MM.

# cat /debug/remoteproc/omap-rproc.1/version

Following is a sample output from the command:


This information can be used to give out the following details:

NOTE: The binary version information relies on the git describe command, so gives out information pertinent to each individual customer’s private tree management.

[edit] SYS/BIOS Debugging

[edit] Adding Traces

Traces can be added to any code other than HWI handlers. The simplest method is to use System_printf() and its variants. It works like the Standard C printf().

#include <xdc/runtime/System.h>
System_printf(" %02d [op %08x] %08x (ret from call to %08x)\n",
              (*cnt)++, entry->op, entry->ret, entry->target);

The output destination for System_printf() is controlled by the System.SupportProxy setting found in <sysbios-rpmsg root>/src/ti/config/IpcCommon.cfg.xs.

 *  The SysMin used here vs StdMin, as trace buffer address is required for
 *  Linux trace debug driver, plus provides better performance.
var System      = xdc.useModule('xdc.runtime.System');
var SysMin      = xdc.useModule('ti.trace.SysMin');
System.SupportProxy = SysMin;

Note that this SysMin is from ti.trace.SysMin which is a module included with RPMsg, and is a customized version of the SysMin module from xdc.runtime. Output from ti.trace.SysMin goes to the trace buffer and helps the kernel print the traces properly by keeping track of an current write pointer. See also the documentation for the xdc.runtime.System module.

[edit] Readrprc Utility

The Ducati firmware binary, ducati-m3.bin, is always named the same irrespective of what ELF baseimages it is created from. readrprc is an useful utility that developers could use intending to find out what a ducati firmware binary is comprised of. readrprc utility is distributed in the src/utils folder of the sysbios-rpmsg tree.

Following is an example output:

bash$ readrprc ducati-m3.bin
magic number RPRC
header version 2
header size 1012
header data
d3e5ad970ccc451e076e3445c56feec4  ../ti/examples/srvmgr/ti_platform_omap4430_core0/debug/test_omx_sysm3.xem3
4395316 2011-11-22 19:06:10.872609680 -0600
b13022c3379b38240c7c675941dacab3  ../ti/examples/srvmgr/ti_platform_omap4430_core1/debug/test_omx_appm3.xem3
4389796 2011-11-22 19:06:10.872609680 -0600

section: 0, address: 0x80062000, size: 0x     558
resource table: 76
resource: 4, da: 0x9f000000, pa: 0x       0, len: 0x    8000, name: 0
resource: 4, da: 0x9f070000, pa: 0x       0, len: 0x    8000, name: 1
resource: 5, da: 0x   14211, pa: 0x       0, len: 0x       0, name: 0
resource: 5, da: 0x  1101ed, pa: 0x       0, len: 0x       0, name: 1
resource: 6, da: 0x9f060000, pa: 0x       0, len: 0x     200, name: 0
resource: 6, da: 0x9f0d0000, pa: 0x       0, len: 0x     200, name: 1
resource: 1, da: 0x60000000, pa: 0x60000000, len: 0x10000000, name: IPU_TILER_MODE_0_1
resource: 1, da: 0x70000000, pa: 0x70000000, len: 0x 8000000, name: IPU_TILER_MODE_2
resource: 1, da: 0x78000000, pa: 0x78000000, len: 0x 8000000, name: IPU_TILER_MODE_3
resource: 1, da: 0xaa000000, pa: 0x4a000000, len: 0x 1000000, name: IPU_PERIPHERAL_L4CFG
resource: 1, da: 0xa8000000, pa: 0x48000000, len: 0x 1000000, name: IPU_PERIPHERAL_L4PER
resource: 1, da: 0xba000000, pa: 0x5a000000, len: 0x 1000000, name: IPU_IVAHD_CONFIG
resource: 1, da: 0xbb000000, pa: 0x5b000000, len: 0x 1000000, name: IPU_IVAHD_SL2
resource: 0, da: 0xa0000000, pa: 0xb3a00000, len: 0x  100000, name: IPU_MEM_IPC_VRING
resource: 0, da: 0x9f000000, pa: 0xb3b00000, len: 0x  100000, name: IPU_MEM_IPC_DATA
resource: 0, da: 0x       0, pa: 0xb3c00000, len: 0x  600000, name: IPU_MEM_TEXT
resource: 0, da: 0x80000000, pa: 0xb4300000, len: 0x 6000000, name: IPU_MEM_DATA
resource: 0, da: 0x88000000, pa: 0xba300000, len: 0x 5a00000, name: IPU_MEM_IOBUFS

section: 2, address: 0x       0, size: 0x      3c
section: 2, address: 0x     400, size: 0x     140
section: 2, address: 0x    4000, size: 0x   17bc4
section: 2, address: 0x80063cd0, size: 0x     200
section: 2, address: 0x9f0e0000, size: 0x   102f8
section: 2, address: 0x       0, size: 0x      3c
section: 2, address: 0x     800, size: 0x     140
section: 2, address: 0x  100000, size: 0x   17b94
section: 2, address: 0x80163778, size: 0x     200
section: 2, address: 0x9f0e0000, size: 0x   102f8

The output includes a firmware header for verifying firmware structure compatibility between A9 and remote processors; and raw firmware data that gives out useful information like

The firmware header is followed by a section table used by the loader on the kernel. Each section number is an identifier instructing the kernel's firmware loader to treat it differently. A section identified by section :0 is a special section, and this needs to be first section in the ducati-m3.bin. This section carries the resource table information, such as the Device and DDR memories used by remote processors, trace sections in DDR, and enables the rest of the sections to be loaded into memory correctly.

[edit] Reset/Boot section corruption

Upon reset Cortex-M3 CPU fetchs and copies the contents of its (virtual) address 0x0 into the stack pointer (SP) and the contents of (virtual) address 0x4 into the PC register. So after bringing M3 of reset, it will jump to the address pointed by (virtual) address 0x4 (the one PC is pointing too) and will start executing code from there. So we can say that address 0x4 (in the reset vector) is pointing to the Boot section.

If part of these sections is corrupted, the behavior of the M3 core will be unknown the next time that will be taken out of reset. Since these two sections are located in the range from 0x0 to 0x3c for omap4, it is an easy target for corruption, when deferencing/writing to a NULL pointer.

[edit] Symptoms of a reset/boot section corruption

To verify that the issue is due to a reset/boot section corruption

1) Deselect Auto-suspend option from Device Drivers in menuconfing

bash$ cd <kernel directory> 
bash$ make menuconfig


- Optionally IPU suspend could be disabled on run time before the use case that causes the issue

# echo on > /sys/devices/platform/omap/omap-rproc.1/power/control

3) Find the physical address that corresponds to virtual address 0x0 to 0x3c. This can be achieved by running the readrprc tool over the ducati-m3.bin.

Note: You can pull the ducati-m3.bin if it is not in your local build.

bash$ cd <sysbios-rpmsg root>/src/utils
bash$ ./readrprc <pull directory>/ducati-m3.bin | grep IPU_MEM_TEXT

For omap4:

resource: 0, da: 0x0, pa: 0xb3c00000, len: 0x600000, name: IPU_MEM_TEXT

For omap5:

resource: 0, da: 0x0, pa: 0xb0000000, len: 0x600000, name: IPU_MEM_TEXT

4) Dump the reset/boot section memory from virtual address 0x0 to 0x3c, just after IPU is loaded the first time (before executing test case that is causing the issue).

For omap4:

| Address (hex) | Data (hex) |
| 0xB3C00000    | 0x00000000 |
| 0xB3C00004    | 0x00000009 |
| 0xB3C00008    | 0x68004808 |
| 0xB3C0000C    | 0xD1042800 |
| 0xB3C00010    | 0xE020F8DF |
| 0xB3C00014    | 0xE000F8DE |
| 0xB3C00018    | 0x48054770 |
| 0xB3C0001C    | 0xD1FD1E40 |
| 0xB3C00020    | 0xE014F8DF |
| 0xB3C00024    | 0xE000F8DE |
| 0xB3C00028    | 0x00004770 |
| 0xB3C0002C    | 0xE00FFFE0 |
| 0xB3C00030    | 0x00030D40 |
| 0xB3C00034    | 0x00000404 |
| 0xB3C00038    | 0x00000804 |
| 0xB3C0003C    | 0xFFFFFFFF |

5) Run/execute the use case that is causing the issue

6) Dump again the reset/boot section memory.

7) Compare memory dump in step (4) against the one in step (5). If they are different the reset/boot section has been corrupted most likely due to a NULL pointer dereference.

[edit] Debugging reset/boot section corruption with CCS

1) Deselect both "Autosuspend support for remoteproc" and "OMAP remoteproc watchdog timer" from Device Driver on menuconfig

bash$ cd <kernel directory> 
bash$ make menuconfig


2) After loading the symbols for the Cortex-M3. Connect to the target, so it will pause the code execution.


3) Go to the View menu and select Breakpoints.


4) In the Breakpoints window add a new Watchpoint


5) Enter the corrupted address in the Watchpoint window.

For example if your memory dump in step (5) of #Symptoms of a reset/boot section corruption section is like:

| Address (hex) | Data (hex) |
| 0xB3C00000    | 0x00000000 |
| 0xB3C00004    | 0x00000009 |
| 0xB3C00008    | 0x68004808 |
| 0xB3C0000C    | 0xD1042800 |
| 0xB3C00010    | 0x00000000 |
| 0xB3C00014    | 0xE000F8DE |
| 0xB3C00018    | 0x48054770 |
| 0xB3C0001C    | 0xD1FD1E40 |
| 0xB3C00020    | 0xE014F8DF |
| 0xB3C00024    | 0xE000F8DE |
| 0xB3C00028    | 0x00004770 |
| 0xB3C0002C    | 0xE00FFFE0 |
| 0xB3C00030    | 0x00030D40 |
| 0xB3C00034    | 0x00000404 |
| 0xB3C00038    | 0x00000804 |
| 0xB3C0003C    | 0xFFFFFFFF |

You know that PA: 0xB3C00010 which corresponds to VA: 0x00000010 was corrupted.


So, enter the corrupted virtual address Location in the Watchpoint entry window. Make sure you select the Access Type field as Memory Write

6) Resume the code execution in the target by pressing the Play button


7) Execute the use case that causes the issues.

When the corruption happens, the Watchpoint should stop the execution and you can observe exactly the line of code causing the corruption through the Code or Disassembly Window.


[edit] Rebuilding a BIOS module for debugging

Sometimes it will be necessary to add some traces within a BIOS module, for debugging purposes. In that case the BIOS module needs to be rebuilt.

1) Copy the file config.bld.default from the etc directory into the packages directory. Rename the file as biosConfig.bld.

sudo cp -v <bios_directory>/etc/config.bld.default <bios_directory>/packages/biosConfig.bld

2) Open the biosConfig.bld with your favorite text editor

cd <bios_directory>/packages/biosConfig.bld
sudo vim biosConfig.bld

Note: Change write permissions if needed.

3) Modify the rootDirPre variable to point to your tools directory. For example, if your tools are instaled at /data/omapts/linux/dsp-tc/

var rootDirPre = "/data/omapts/linux/dsp-tc/";

4) Add the next lines if rebuilding your module for M3/M4 target (IPU).

/* Setup for M3 and M4 targets. M4 targets are still built in M3 mode. */
var M3 = xdc.useModule('ti.targets.arm.elf.M3');
M3.rootDir =  rootDirPre + "cgtarm-4.9.0" + rootDirPost;
M3.ccOpts.suffix += " -ms ";
M3.platform = "ti.platforms.evmLM3S9B92";

Note: cgtarm-4.9.0 is the name of the directory of your Code Generation Tools for ARM

5) Add the next lines if rebuilding your module for DSP target.

// Setup for C64P ELF target
var C64P = xdc.useModule('ti.targets.elf.C64T');
C64P.rootDir = rootDirPre + "cgt6x-7.2.0" + rootDirPost;
C64P.ccOpts.suffix += c6xSuffix + "--embed_inline_assembly ";
C64P.platform = "ti.platforms.evmDM6446";

Note: cgt6x-7.2.0 is the name of the directory of your Code Generation Tools for C64x

6) Near the end of the file, look for the array called "Build.targets". Add the M3 or/and C64P target according to your needs.

Build.targets = [
        // tiTargets.C28_large,
        // tiTargets.C28_float,
  . . .
        // armElfTargets.M3,

7) Export your xdc tools. (Use the same you are using to buils your sysbios-rpmsg)

export PATH=$PATH:<tools_directory>xdctools_3_20_07_86

8) Add the traces in the module you want to debug.

9) Build just the module you modified

For example if you added some traces or modified <bios_directory>/packages/ti/sysbios/family/arm/ducati/omap4430/Power.c file, you only need to pass this directory to the xdc build command.

cd <bios_directory>/packages/
xdc XDCBUILDCFG=./biosConfig.bld -PR ti/sysbios/family/arm/ducati/omap4430/

Note: Change permissions if needed.

10) Rebuild again your sysbios-rpmsg and ducatimm projects so they will be linked against the modified BIOS modules.

Personal tools