Re: Feature Parity: Remote fork following

[Luis Machado <lgustavo at codesourcery dot com>]

On 07/24/2013 12:45 PM, Pedro Alves wrote:
> > Remote fork following is currently available only for native targets. In
> > order to expand this feature to remote targets, we need remote protocol
> > changes.
> >
> > First, we have two remote modes (remote and extended-remote). remote is
> > only capable of debugging a single process at a time. extended-remote is
> > capable of debugging multiple processes, so it looks like the best fit
> > for this feature.
> >
> > Though extended-remote is the right protocol to deal with remote
> > fork/vfork events, we may still want to consider a fallback mechanism
> > for the single-process remote protocol.
>
> Yeah.  The follow-fork modes worked in GDB long before GDB had awareness
> of multiple inferiors.
>
> A more meaningful case to consider would be multi-process extension
> availability.
>
> The trouble with single-process protocol for this I think, is that GDB needs to be
> able to detach breakpoints (etc.) from the sibling it'll detach from (say, the child),
> without the multi-process extensions, you're likely to end up with a very different
> mechanism compared to multi-process extensions (w/ PIDs available).  That is, given
> that lots of packets work on the current selected thread/context, how would GDB
> select the child, w/ follow-parent/detach-on-fork, and redirect memory/register/etc.
> requests to it?  It'd need a new packet to select a different process, but that's
> sort of reinventing the multi-process extensions, anyway.
>
> Originally, the multi-process extensions were always off if you connected
> with "target remote"; they'd only be available to extended-remote.  But
> nowadays that's no longer true -- multi-process is always available even
> in "target remote", if the target supports them.  The push in that direction
> had follow-fork in mind.  So, if multi-process extensions make this easier,
> we can just require them.

Right. Since we are moving towards more complex cases - multiprocess, 
non-stop, multi-cores etc - it sounds more practical to just require the 
multi-process extensions to be there.

Doing new additions to keep the single-process protocol working as 
before does not seem too productive. Backwards compatibility is always 
good, but we have to be careful about it not becoming a dead weight.

Risking creating more work, how does it sound to require even the 
single-process (target remote) mode to use PID's for context, just like 
the extended-remote mode does? I wonder how hard that is and how much 
work is involved there. Implementing single-process mode on top of 
multi-process... similar to all-stop on top of non-stop.

A big problem is backwards compatibility in this case. There are a bunch 
of gdbserver/gdb versions out there that would try to speak to each 
other without mentioning PID's no matter what.

> As for remote vs extended-remote, the protocol should be agnostic of it,
> given that w/ extended-remote we can still set detach-on-fork off.

Agreed.

> Keep in mind that it's not just about following forks and debugging
> multiple process in the same session.  Even with detach-on-fork off/follow-parent,
> we should be removing breakpoints etc. from the child before detaching it,
> but we don't do it presently.  It just happens that most programs exec soon
> after fork, so they mask out the issue.

Right. Yes, this needs to be cared for.

> > Consider a process P1, with Px being its childs. The possible variations
> > we have are:
> >
> >
> > follow-fork-mode parent
> > detach-on-fork on.
> > ---
> > - P1 forks P2
> > - P1 continues under gdb's control and is moved outside of the fork call.
> > - P2 is detached from gdb and runs freely.
> >
> >
> > follow-fork-mode parent
> > detach-on-fork off.
> > ---
> > - P1 forks P2
> > - P1 continues under gdb's control and is moved outside of the fork call.
> > - P2 is added to the list of debugged processes and is left sitting at
> > the fork call.
> > - GDB controls both P1 and P2
> >
> >
> > follow-fork-mode child
> > detach-on-fork on.
> > ---
> > - P1 forks P2
> > - P1 is detached from gdb and runs freely.
> > - P2 is added to the list of debugged processes and is moved outside of
> > the fork call.
> >
> >
> > follow-fork-mode child
> > detach-on-fork off.
> > ---
> > - P1 forks P2
> > - P1 is left sitting at the fork call.
> > - P2 is added to the list of debugged processes and runs freely under
> > GDB's control.
> >
> >
> > Remote protocol changes
> > -----------------------
> >
> > In order to inform gdbserver of the fork-handling settings, we need a
> > couple new packets and one additional feature query string.
> >
> > qFollowForkMode:n - Sets the fork-following behavior for the remote side.
> > n is 0 - Follow parent
> > n is 1 - Follow child
> >
> > qDetachOnFork:n - Sets whether the remote stub should detach the
> > child/parent on a fork (depends on whether it is following the parent or
> > the child).
> > n is 0 - Do not detach child on fork
> > n is 1 - Detach child on fork
> >
> > FollowFork+ - If the remote side adsertises support for this feature it
> > means both qFollowForkMode and qDetachOnFork packets are supported and
> > can be issued by GDB to configure the behavior on the remote stub. The
> > default settings are Follow Parent and Detach on fork.
> >
> > With the changes above we have a way of configuring the remote stub to
> > handle fork events properly, but we still need a way to inform GDB of a
> > fork event since this information comes from PTRACE.
>
> GDB needs to undo things from the child before detaching it
> (gdb-side breakpoints, displaced stepping, etc.).  Only if it's really
> sure nothing will need to be removed, could pushing follow-fork-mode/detach-on-fork
> to the remote side be useful.  I do believe such an optimization (where
> the target handles fork following) might be useful for plain "continue until
> SEGV in some child" scenario, but otherwise, I believe it to be better to start
> with the basics first.  Make catch fork/vfork/exec work, and build up from
> that.  Proper vfork handling, where you have shared memory region between parent/child
> until TARGET_WAITKIND_VFORK_DONE (which needs RSP modelling as well) also needs to
> be factored in.

Handling these events inside gdbserver seems to be more straightforward 
than doing back-and-forth to inform GDB of things that are happening.

The problem i see is the lack of displaced stepping support in 
gdbserver. Everything is controlled by GDB in this case. In the future, 
i imagine displaced stepping will just disappear and the debugger will 
just ask for "stepi/step" and things will happen.

Right now i'm not sure how complicated it would be to move such logic 
torwards the lower targets. Probably not that much, but quite some code 
would need to be shared.

With that said, we may have to re-work this remote fork/exec code when 
common code is available to gdbserver, but it does look simpler today to 
do the back-and-forth with GDB, for simplicity.

> (Note that the current vfork modelling in GDB has a design bug -- at the
> kernel level, vfork is really per-thread, while GDB assumes it's
> per-inferior.)
>
> > The simplest way to inform GDB of such event is through the usual stop
> > reply packet T, via the stop reason field. For our purposes we add a
> > couple new stop reasons: fork and vfork. Those should accompany the
> > reply and let GDB know that either a fork or vfork took place. Maybe we
> > should also add a execd reason for future use?
>
> Adding support for fork without execd is practically useless...  As
> mentioned, there's also TARGET_WAITKIND_VFORK_DONE to consider.

Right. Let's add it to the list as well. We would then have stop reasons 
for the following events: fork, vfork, vfork_done, execd.

> > With the protocol extended and considering gdbserver's event loop is
> > ready to handle and report fork events, we need to define the control
> > flow. Two scenarios come to mind:
> >
> > A - GDB has less control over the remote stub and the stub is clever
> > enough to decide what to do and report back to GDB (suitable for
> > all-stop mode?).
>
> The need to properly handle removing breakpoints, etc. from the child/parent
> before detaching calls for doing B first, and A as a future optimization that
> will only be applicable in certain scenarios, I believe.

As discussed above, this looks very useful, but at this stage gdbserver 
still does not have what it takes to do this autonomously due to pending 
features, so let us drop it for now.

> > Control flow B
> >
> > - GDB connects to the stub and checks the availability of ForkFollowing.
> > - GDB sends the settings for FollowForkMode and DetachOnFork to the stub
> > on connection startup or whenever these settings change.
> > - The stub runs the program (continue or step) and eventually notices it
> > forked (fork or vfork).
> > - The stub sends the notification back to GDB stating it has detected a
> > fork or vfork.
> > - GDB instructs the stub what to do with said process. Either follow the
> > parent or child process.
> > - [TBD] The stub informs GDB the inferior has execd a new image and GDB
> > tells the stub what to do.
> > - The stub informs GDB of new processes if the child was followed.
> >
> > A delicate scenario that may require the second control flow is non-stop
> > mode with displaced stepping. GDB needs to do some fixups for the
> > newly-created process before letting it run freely.
>
> Right.  The devil is in the details.  :-)
>
> Lost of fork handling details are presently in
> linux-nat.c:linux_child_follow_fork (and other targets).
> Modelling this properly I think should result in common code
> handling most of the details instead.

Hopefully we will come out of this with some nice code that can be used 
by both GDB and gdbserver. :-)