Re: Feature Parity: Remote fork following

[Pedro Alves <palves at redhat dot com>]

Hi!

On 07/15/2013 08:47 PM, Luis Machado wrote:
> Hi,
> 
> One of the things that exists in the feature parity wish list
> (http://sourceware.org/gdb/wiki/LocalRemoteFeatureParity) is remote fork
> following. I'm volunteering for that work. I came up with the following 
> draft and i thought i'd make it available for discussion so we can all 
> agree upon a good enough design.

Thanks!

> Remote fork following is currently available only for native targets. In 
> order to expand this feature to remote targets, we need remote protocol 
> changes.
> 
> First, we have two remote modes (remote and extended-remote). remote is 
> only capable of debugging a single process at a time. extended-remote is 
> capable of debugging multiple processes, so it looks like the best fit 
> for this feature.
> 
> Though extended-remote is the right protocol to deal with remote 
> fork/vfork events, we may still want to consider a fallback mechanism 
> for the single-process remote protocol.

Yeah.  The follow-fork modes worked in GDB long before GDB had awareness
of multiple inferiors.

A more meaningful case to consider would be multi-process extension
availability.

The trouble with single-process protocol for this I think, is that GDB needs to be
able to detach breakpoints (etc.) from the sibling it'll detach from (say, the child),
without the multi-process extensions, you're likely to end up with a very different
mechanism compared to multi-process extensions (w/ PIDs available).  That is, given
that lots of packets work on the current selected thread/context, how would GDB
select the child, w/ follow-parent/detach-on-fork, and redirect memory/register/etc.
requests to it?  It'd need a new packet to select a different process, but that's
sort of reinventing the multi-process extensions, anyway.

Originally, the multi-process extensions were always off if you connected
with "target remote"; they'd only be available to extended-remote.  But
nowadays that's no longer true -- multi-process is always available even
in "target remote", if the target supports them.  The push in that direction
had follow-fork in mind.  So, if multi-process extensions make this easier,
we can just require them.

As for remote vs extended-remote, the protocol should be agnostic of it,
given that w/ extended-remote we can still set detach-on-fork off.

Keep in mind that it's not just about following forks and debugging
multiple process in the same session.  Even with detach-on-fork off/follow-parent,
we should be removing breakpoints etc. from the child before detaching it,
but we don't do it presently.  It just happens that most programs exec soon
after fork, so they mask out the issue.

> 
> Consider a process P1, with Px being its childs. The possible variations 
> we have are:
> 
> 
> follow-fork-mode parent
> detach-on-fork on.
> ---
> - P1 forks P2
> - P1 continues under gdb's control and is moved outside of the fork call.
> - P2 is detached from gdb and runs freely.
> 
> 
> follow-fork-mode parent
> detach-on-fork off.
> ---
> - P1 forks P2
> - P1 continues under gdb's control and is moved outside of the fork call.
> - P2 is added to the list of debugged processes and is left sitting at 
> the fork call.
> - GDB controls both P1 and P2
> 
> 
> follow-fork-mode child
> detach-on-fork on.
> ---
> - P1 forks P2
> - P1 is detached from gdb and runs freely.
> - P2 is added to the list of debugged processes and is moved outside of 
> the fork call.
> 
> 
> follow-fork-mode child
> detach-on-fork off.
> ---
> - P1 forks P2
> - P1 is left sitting at the fork call.
> - P2 is added to the list of debugged processes and runs freely under 
> GDB's control.
> 
> 
> Remote protocol changes
> -----------------------
> 
> In order to inform gdbserver of the fork-handling settings, we need a 
> couple new packets and one additional feature query string.
> 
> qFollowForkMode:n - Sets the fork-following behavior for the remote side.
> n is 0 - Follow parent
> n is 1 - Follow child
> 
> qDetachOnFork:n - Sets whether the remote stub should detach the 
> child/parent on a fork (depends on whether it is following the parent or 
> the child).
> n is 0 - Do not detach child on fork
> n is 1 - Detach child on fork
> 
> FollowFork+ - If the remote side adsertises support for this feature it 
> means both qFollowForkMode and qDetachOnFork packets are supported and 
> can be issued by GDB to configure the behavior on the remote stub. The 
> default settings are Follow Parent and Detach on fork.
> 
> With the changes above we have a way of configuring the remote stub to 
> handle fork events properly, but we still need a way to inform GDB of a 
> fork event since this information comes from PTRACE.

GDB needs to undo things from the child before detaching it
(gdb-side breakpoints, displaced stepping, etc.).  Only if it's really
sure nothing will need to be removed, could pushing follow-fork-mode/detach-on-fork
to the remote side be useful.  I do believe such an optimization (where
the target handles fork following) might be useful for plain "continue until
SEGV in some child" scenario, but otherwise, I believe it to be better to start
with the basics first.  Make catch fork/vfork/exec work, and build up from
that.  Proper vfork handling, where you have shared memory region between parent/child
until TARGET_WAITKIND_VFORK_DONE (which needs RSP modelling as well) also needs to
be factored in.

(Note that the current vfork modelling in GDB has a design bug -- at the
kernel level, vfork is really per-thread, while GDB assumes it's
per-inferior.)

> 
> The simplest way to inform GDB of such event is through the usual stop 
> reply packet T, via the stop reason field. For our purposes we add a 
> couple new stop reasons: fork and vfork. Those should accompany the 
> reply and let GDB know that either a fork or vfork took place. Maybe we 
> should also add a execd reason for future use?

Adding support for fork without execd is practically useless...  As
mentioned, there's also TARGET_WAITKIND_VFORK_DONE to consider.

> 
> Stop Reply Packet T will be augmented to reply the following stop reasons:
> 
> T AA fork... - The packet indicates that a fork has happened in the 
> program being debugged.
> T AA vfork... - The packet indicates that a vfork has happened in the 
> program being debugged.
> T AA execd... - The packet indicates that the inferior execd a new image 
> (TBD)
> 
> With the protocol extended and considering gdbserver's event loop is 
> ready to handle and report fork events, we need to define the control 
> flow. Two scenarios come to mind:
> 
> A - GDB has less control over the remote stub and the stub is clever 
> enough to decide what to do and report back to GDB (suitable for 
> all-stop mode?).

The need to properly handle removing breakpoints, etc. from the child/parent
before detaching calls for doing B first, and A as a future optimization that
will only be applicable in certain scenarios, I believe.

> B - GDB has more control over the remote stub and the stub mainly 
> reports the fork event back to GDB for it to decide what to do. GDB will 
> then proceed to guide the stub/program through the
> correct forking process and behavior (suitable for non-stop due to 
> displaced stepping and software single-stepping?).

Right.

> 
> Control flow A
> 
> - GDB connects to the stub and checks the availability of FollowFork+.
> - GDB sends the settings for FollowForkMode and DetachOnFork to the stub 
> on connection startup or whenever these settings change.
> - The stub runs the program (continue or step) and eventually notices it 
> forked (fork or vfork).
> - The stub knows both the settings of FollowForkMode and DetachOnFork 
> and follows those to reach the end result.
> - The stub informs GDB of new processes if the child was followed.

This is very incomplete, as mentioned above.

> 
> Control flow B
> 
> - GDB connects to the stub and checks the availability of ForkFollowing.
> - GDB sends the settings for FollowForkMode and DetachOnFork to the stub 
> on connection startup or whenever these settings change.
> - The stub runs the program (continue or step) and eventually notices it 
> forked (fork or vfork).
> - The stub sends the notification back to GDB stating it has detected a 
> fork or vfork.
> - GDB instructs the stub what to do with said process. Either follow the 
> parent or child process.
> - [TBD] The stub informs GDB the inferior has execd a new image and GDB 
> tells the stub what to do.
> - The stub informs GDB of new processes if the child was followed.
> 
> A delicate scenario that may require the second control flow is non-stop 
> mode with displaced stepping. GDB needs to do some fixups for the 
> newly-created process before letting it run freely.

Right.  The devil is in the details.  :-)

Lost of fork handling details are presently in
linux-nat.c:linux_child_follow_fork (and other targets).
Modelling this properly I think should result in common code
handling most of the details instead.

There's an old hack/quick/attempt at fork support in the
multiprocess-20081120-branch branch (in gdb's cvs/git; see

gdb/
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/remote.c.diff?r1=1.329.2.1&r2=1.329.2.2&cvsroot=src&only_with_tag=multiprocess-20081120-branch&f=h
git 8b11acc6150c279279bc286f4298e79586f171e6

gdbserver/
http://sourceware.org/cgi-bin/cvsweb.cgi/src/gdb/gdbserver/linux-low.c.diff?r1=1.81&r2=1.81.2.1&cvsroot=src&only_with_tag=multiprocess-20081120-branch&f=h
git 95035af3c98af4ea5f905ef1ce576454862699e7
),

but that's against a very old tree, and it doesn't address
some of the points above.  Probably not much useful anymore.

-- 
Pedro Alves