Re: streaming output options (was Re: [external-perftools-list] notes from 20070503 Systemtap Meeting)

[Tom Zanussi <zanussi at us dot ibm dot com>]

On Tue, 2007-05-08 at 14:11 -0400, Martin Hunt wrote:
> On Tue, 2007-05-08 at 13:26 -0400, Frank Ch. Eigler wrote:
> 
> > I believe the primary concern should not be the overall speed of
> > merging, but rather, the non-intrusiveness of the trace writing
> > process.  Specifically, two processors writing trace data at the same
> > time should not casually block each other.
> 
> I thought your concern was with having a single easy-to-read output
> channel?
> 
> Let me explain in more detail.  Perhaps I am missing some options, but
> these seem to be the options we have.
> 
> 1. A single file that contains the data.  Can use with "cat",
> third-party apps. To do this, merging must be in the kernel. That means
> some blocking must occur.  On my test system, I'm showing that with two
> processors in a tight loop writing to a single output channel, a probe
> that does a 100 byte printf takes 350ns in bulk mode compared to 400ns
> in streaming mode where it must get spinlock then memcpy the data to the
> channel. This is worst case. Note that 100bytes every 350ns = 272
> Mbytes/sec, which is a very fast logging rate.
> 
> 2. Per-cpu files that are merged into a single stream in userspace. This
> stream can be made available to other apps, but that merging will be
> tricky and require that the apps either use staprun or some library we
> provide (UTT?). This option is not flight-recorder friendly because
> unsent output will be spread over the percpu files and cannot be simple
> ASCII (it must have headers sequence numbers and lengths). Perhaps
> someone more experience with kdump can suggest some way to make the data
> more accessible?
> 
> Anything else?

I think it would be good to support a 'combined' cpuMerged file that
could be cat'ed, and is something people seem to want, usually in the
context of flight-recorder tracing e.g. awhile back I sent a relay patch
to the logdev maintainer that would make it easy for logdev to grab,
sort and present the current relay contents for his read(2)
implementation.  He was interested in being able to per-cpu logging, but
also wanted to get at his data easily and without the need for an
external app.  This need has also come up more recently for something
else as well.

But for normal tracing, I think UTT should work ok - it's basically just
refactored blktrace code, so shouldn't differ much from that in
performance.

I also don't think it would be a big deal to make UTT flight-recorder
friendly - set overwrite mode to 1 and teach the userspace code to drain
and merge the current contents of the per-cpu buffers is all you'd need
to do, I think.

Tom