A tour of Frysk’s utilities (part 3)

This final whistle stop tour of Frysk’s utilites will cover: fstack, ftrace and fhpd.

fstack – Similar in functionality to pstack. This utility will display the process stack in a similar manner to pstack. This utility will work with a live process (ie specify fstack <pid>) or with a corefile (ie fstack <corefile <exe>). In the corefile case, the stack will represent how it was when the coredump was taken. For example:

sleep 5000 &
[1] 695

fcore 695
fstack core.695 /bin/sleep 

Task #695
#0 0x00000038cba9ac30 in __nanosleep_nocancel () from .../libc.so.6
#1 0x0000000000402f2b in rpl_nanosleep() .../nanosleep.c#71
#2 0x0000000000402a54 in xnanosleep() .../xnanosleep.c#100
#3 0x000000000040158c in main () from .../sleep
#4 0x00000038cba1e074 in __libc_start_main () from .../libc.so.6
#5 0x00000000004011d9 in _start () from .../sleep

ftrace – this utility will either attach to a process, or run an executable specified on the command-line, and trace its system-calls. For example, trace the write system-call in /binls:

ftrace  -sys="write" /bin/ls foo
2803.2803 attached /bin/ls

/bin/ls: 2803.2803 syscall write(2, "/bin/ls: ", 9) = -1 ERRNO=38
cannot access foo

2803.2803 syscall write(2, "cannot access foo", 17) = -1 ERRNO=38
: No such file or directory

2803.2803 syscall write(2, ": No such file or di...", 27) = -1 ERRNO=38

2803.2803 syscall write(2, " No such file or di...", 1) = -1 ERRNO=38
2803.2803 exited with status 2

In the above example if we wanted to look at all system calls, we would specify -sys”*”.

There are many useful actions and filters that ftrace can perform on the process. For example, -stack: this will print a stack back-trace whenever the matched system-call is detected in the process. A few more examples: specify “-p pid” to attach to an existing process. “-c” to trace a process’ children, “-m” to detect and print when a library is mapped/unmapped and so on. The scope of use is beyond this little blog post, so I encourage you to use and experiment.

fhpd – is a command-line debugger based on the Frysk core (engine if you will). It is largely based off the HPD specification. The scope and use of this debugger in a fair and consistent manner is well beyond this blog post, but we’ll look at a few brief examples. In this example, we’ll load up a core-file with its backing executable. You can of course, attach to a pid, or load an executable from the command-line.

fhpd core.695 /bin/sleep

Attached to core file: core.695

To look at the stack back-trace you would type:

(fhpd) where
#0 0x00000038cba9ac30 in __nanosleep_nocancel () from /lib64/libc.so.6
#1 0x0000000000402f2b in rpl_nanosleep () from /bin/sleep
#2 0x0000000000402a54 in xnanosleep () from /bin/sleep
#3 0x000000000040158c in main () from /bin/sleep
#4 0x00000038cba1e074 in __libc_start_main () from /lib64/libc.so.6
#5 0x00000000004011d9 in _start () from /bin/sleep

And to look at frame specific information:

(fhpd) down
#1 0x0000000000402f2b in rpl_nanosleep(const struct timespec {
__time_t tv_sec;
long int tv_nsec;
} * requested_delay,struct timespec {
__time_t tv_sec;
long int tv_nsec;
} * remaining_delay) /usr/src/debug/coreutils-6.9/lib/nanosleep.c#71

Followed by a source listing:

(fhpd) list
61     /* nanosleep mishandles large sleeps due to internal overflow
62        problems, so check that the proper amount of time has actually
63        elapsed.  */
65     struct timespec delay = *requested_delay;
66     struct timespec t0;
67     getnow (&t0);
69     for (;;)
70       {
->  71         int r = nanosleep (&delay, remaining_delay);
72         if (r == 0)
73          {
74            time_t secs_sofar;
75            struct timespec now;
76            getnow (&now);
78            secs_sofar = now.tv_sec - t0.tv_sec;
79            if (requested_delay->tv_sec < secs_sofar)
80              return 0;

I’ll stop here. There are many, many commands. We did not even look at breakpoints, or stepping or loading executables or hundreds of different things. But this blog is not a tutorial, rather a taste, and I encourage you to experiment and find out for yourself, and play around with fhpd. And where things are broken (Frysk is in constant development) submit patches, bug reports, or come and let us know on irc (irc.gimp.org, channel: #frysk).


  1. Dave Malcolm says:

    Thanks for blogging about these tools. An idea that occurred to me reading this: could fstack (or the underlying infrastructure) be used to build a simple profiler? ( keep sampling the stack at regular intervals from a running process, and accumulate results). It would allow you to take (rough) profiles without needing to recompile with special flags. Maybe an “fprofile” tool? (it it doesn’t already exist)

  2. Phil says:

    Hi Dave,

    While periodic sampling is certainly possible, one must always keep in mind that Frysk achieves what it achieves via ptrace, which is disruptive and slow. Another thing is a lot of this stuff is in the Frysk core callable via API, so the f* utilities are really just he tips of the icebergs. They are normally just calling an existing API. So sure fprofiler is very possible.

    However, might I suggest you have a look at SystemTap to look at the periodic sampling of back-traces? I believe this is ongoing work at the moment, not finished. But I do think that it is on their roadmap – if not already underway. (http://sourceware.org/systemtap)

Post a comment

Copyright © Phil Muldoon

Built on Notes Blog Core
Powered by WordPress