A tour of Frysk’s utilities (part 2)

This round-up (part 2 of 3) will look at ferror, fexe and fmap. The next (and final in this series) blog entry will covers: fstack, ftrace and fhpd.

ferror- This is a utility that will help you find the source of your programming errors. In does a lot of grepping for you, and allows you to work back from the error message it catches. It does this by watching for the write() system call to be executed in the process or executable specified. When ferror sees a write system call, it matches the write arguments to the search string you provided. When the match occurs, a back-trace is printed. I find my main use is to extract just where an error is occurring/originates, and working back from that origin. For example:

ferror -e "No such file or directory" /bin/ls foo/null

Tracing 22661.22661

/bin/ls: cannot access foo/null: No such file or directory

Process is trying to output No such file or directory

Stack trace:

Task #22661
#0 0x00000038cbac6e80 in __write_nocancel () from .../libc-2.7.so
#1 0x00000038cba6c343 in _IO_file_write@@GLIBC_2.2.5 () from .../libc-2.7.so
#2 0x00000038cba6d803 in _IO_file_xsputn@@GLIBC_2.2.5 () from .../libc-2.7.so
#3 0x00000038cba475e8 in buffered_vfprintf () from .../libc-2.7.so
#4 0x00000038cba430bf in _IO_vfprintf () from .../libc-2.7.so
#5 0x00000038cba6133b in __fxprintf () from .../libc-2.7.so
#6 0x00000038cbad3d07 in error_tail () from .../libc-2.7.so
#7 0x00000038cbad4083 in __error () from .../libc-2.7.so
#8 0x0000000000402f9b in [unknown] from .../ls
#9 0x0000000000403e0f in [unknown] from .../ls
#10 0x0000000000407542 in [unknown] from .../ls
#11 0x00000038cba1e074 in __libc_start_main () from .../libc-2.7.so
#12 0x0000000000402369 in [unknown] from .../ls

This above trace gives you a good indication of where to start (and finish)

fexe – This utlity will print the backing executable behind a pid or corefle. For example:

 fexe 17305

/usr/lib64/firefox-2.0.0.12/firefox-bin

fmaps – This prints out the process maps of a corefile (simulated), a process (actual from /proc/self/maps) or an on-file excutable (simulated to executable maps). Lets take a corefile example:

sleep 5000 &
[2] 25681

fcore 25681

fmaps core.25681 /bin/sleep 

0x400000-0x405000 r-xp 0x0 -1:-1 -1 /bin/sleep
0x604000-0x605000 rw-p 0x0 -1:-1 -1 /bin/sleep
0x605000-0x626000 rw-p 0x0 -1:-1 -1 /bin/sleep
0x3235400000-0x3235408000 r-xp 0x0 -1:-1 -1 /lib64/librt.so.1
0x3235607000-0x3235608000 r--p 0x0 -1:-1 -1 /lib64/librt.so.1
0x3235608000-0x3235609000 rw-p 0x0 -1:-1 -1 /lib64/librt.so.1
0x38ca800000-0x38ca81b000 r-xp 0x0 -1:-1 -1 /lib64/ld-linux-x86-64.so.2
0x38caa1a000-0x38caa1b000 r--p 0x0 -1:-1 -1 /lib64/ld-linux-x86-64.so.2
0x38caa1b000-0x38caa1c000 rw-p 0x0 -1:-1 -1 /lib64/ld-linux-x86-64.so.2
0x38cba00000-0x38cbb4d000 r-xp 0x0 -1:-1 -1 /lib64/libc.so.6
0x38cbd4d000-0x38cbd51000 r--p 0x0 -1:-1 -1 /lib64/libc.so.6
0x38cbd51000-0x38cbd52000 rw-p 0x0 -1:-1 -1 /lib64/libc.so.6
0x38cbd52000-0x38cbd57000 rw-p 0x0 -1:-1 -1 /lib64/libc.so.6
0x38cc600000-0x38cc616000 r-xp 0x0 -1:-1 -1 /lib64/libpthread.so.0
0x38cc815000-0x38cc816000 r--p 0x0 -1:-1 -1 /lib64/libpthread.so.0
0x38cc816000-0x38cc817000 rw-p 0x0 -1:-1 -1 /lib64/libpthread.so.0
0x38cc817000-0x38cc81b000 rw-p 0x0 -1:-1 -1 /lib64/libpthread.so.0
0x2aaaaaaab000-0x2aaaaaaad000 rw-p 0x0 -1:-1 -1
0x2aaaaaad2000-0x2aaaaaad4000 rw-p 0x0 -1:-1 -1
0x2aaaaaad4000-0x2aaaaf52e000 r--p 0x0 -1:-1 -1
0x7ffff381c000-0x7ffff3831000 rw-p 0x0 -1:-1 -1
0x7ffff39fe000-0x7ffff3a00000 r-xp 0x0 -1:-1 -1 [vdso]
0xffffffffff600000-0xffffffffff601000 r-xp 0x0 -1:-1 -1

An important point to note in this example; the actual process maps are not stored in a corefile – they have to be reconstructed and “simulated”. This is done by disassembling the linkmap table in the corefile, and querying each elf file object noted in the linkmap. It collates the elf object file-maps and builds thee map-table.

With the live process use case: fmaps 1234 would print out the actual maps as described in the /proc/self/maps table.

7 comments

  1. Dave Malcolm says:

    Nice: ferror looks useful; doesn’t seem to use PATH though, so have to give abs path for executable. Filed downstream as:
    https://bugzilla.redhat.com/show_bug.cgi?id=436241

    These small tools that leverage the frysk core are neat. How about one that monitors all processes that talk to a socket (e.g. give me a backtrace every time something talks to the X server, to dbus, etc)?

    ferror and fmaps seems to be new since F8 was released (I had to “yum upgrade frysk” on this F8 box to get them): is it worth adding these Frysk enhancements to the Fedora 9 features page? Might be another way of getting more people interested in Frysk:
    http://fedoraproject.org/wiki/Releases/9/FeatureList

    Hope this helps
    Dave

  2. Phil says:

    Thanks for the input Dave I looked at the two bugs you posted downstream, and I’ll nudge Sami about ferror today. Both great features.

    The socket idea is neat Currently thinking on how to do it Maybe something similar to ferror that looks for the system call?

    Hmmmm fsocket …

    ;)

  3. Tom Tromey says:

    Matching on the strerror() result seems weird to me. Maybe the errno or symbolic constant would be nicer. And why just write()?

    I’d like to be able to run: ferror make check, but have it only print stack traces when some program somewhere SEGVs. Is that possible?

  4. Phil says:

    Hi Tom,

    It’s an interesting point for sure. One of your features is currently possible:

    “I’d like to be able to run: ferror make check, but have it only print stack traces when some program somewhere SEGVs. Is that possible?”

    you can use fcatch for that exact purpose:

    fcatch /usr/bin/make check

    will do that. A bug Dave filed is that the f* utilities do not traverse $PATH so the paths have to absolute. That’s irritating. Not sure how it could be fixed in in Java?

    Possibly there is case here for merging these two utilities into a more generic, catch system call, check arguments, print stack trace.

  5. Petr says:

    RE “Possibly there is case here for merging these two utilities into a more generic, catch system call, check arguments, print stack trace.”

    That’s where ftrace is heading :)
    E.g. stack tracing on sigsegvs can be done via ftrace -sig=#segv
    Similar with system calls, ftrace -sys=write will trace write system calls for you, with hash mark (ftrace -sys=#write) it will stack trace on them.
    Similar with library symbols.

  6. [...] some examples of utilities created on top of the Frysk framework that you might find handy (part1, part2, part3). We now also have the frysk manual pages [...]

  7. herbaltras says:

    ? ??????? ??? ????????? ? herbalcompany.ru, ?????? ?????? ???????????, ?????? ??? ???????, ? ?????? ????! ???????!

Post a comment

Copyright © Phil Muldoon

Built on Notes Blog Core
Powered by WordPress