This is the mail archive of the systemtap@sourceware.org mailing list for the systemtap project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: [pcp] suitability of PCP for event tracing


On 09/01/2010 05:49 AM, Frank Ch. Eigler wrote:
Hi -

Thanks, Nathan, Ken, Greg, Mark, for clarifying the status quo and
some of the history.

We understand that the two problem domains are traditionally handled
with the event-tracing -vs- stats-monitoring distinction.  We're trying
to see where best to focus efforts to make some small steps to bridge
the two, where plenty of compromises are possible.  We'd prefer to
help build on an existing project with a nice community than to do new
stuff.

yes certainly :)


For the poll-based data gathering issue, a couple of approaches came up:

(1) bypassing pmcd and generating an pmarchive file directly from
     trace data This appears to imply continuing the archive-vs-live
     dichotomy that makes it difficult for clients to process both
     recent and current data seamlessly together.

one of the issues with the live vrs archive dichotomy is that live data is always available (since you're requesting it explicitly from a PMDA that is otherwise passive), whereas the archive data is not available unless configured to be collected before-hand (see pmlogger). There is too much data to collect everything all the time - it's too impracticable and intrusive, so some form of filtering and/or aggregation needs to be done (see pmlogsummary, and Greg's old project too).

Since using such
     files would probably also need a custom client, then we'd not be
     using much of the pcp infrastructure, only as a passive data
     encoding layer.  This may not be worthwhile.

(2) protocol extensions for live-push on pmda and pmcd-client interfaces
     This clearly larger effort is only worth undertaking with the
     community's sympathy and assistance.  It might have some
     interesting integration possibilities with the other tools,
     espectially pmie (the inference engine).

yep - I suspect Ken and maybe Nathan would have further comments on this



For the static-pmns issue, the possibility of dynamic instance domains, metric subspaces is probably sufficient, if the event parameters are limited to only 1-2 degrees of freedom. (In contrast, imagine browsing a trace of NFS or kernel VFS operations; these have ~5 parameters.)

PCP instance domains are traditionally single dimensional, though there are a few exceptions such as kernel.percpu.interrupts. It's easy enough to split multi-dimensional data structures out into multiple metrics with a common instance domain.

For the scalar-payloads issue, the BLOB/STRING metric types are indeed
available but are opaque to other tools, so don't compose well.  Would
you consider one additional data type, something like a JSON[1]
string?  It would be self-describing, with pmie and general processing
opportunities, though those numbers would lack the PMDA_PMUNITS
dimensioning.

this could work using string or binary blob data types in the existing protocols - though there is a size limit. And one of the blessed features of PCP is the client monitoring tools can more or less monitor any metrics - so any solution here would also need specially crafted client tools. Extensions to the perl binding would probably work best, e.g. interface with perl-JASON-*

For the filtering issue, pmStore() is an interesting possibility,
allowing the PMDAs to bear the brunt.  OTOH, if pmcd evolved into a
data-push-capable widget, it could serve as a filtering proxy,
requiring separate API or interpretation of the pmStore data.

well pmcd is already data-push capable using the pmstore interface, allowing clients to store values for certain metrics in some of the PMDAs. Filtering and parsing is done by the PMDA itself and pmcd just acts as a proxy passthru (kind of a back-channel to the pull interface).

pmstore hasn't really been used in anger like this though - more just
for setting config & control options and the like. The same (or similar)
protocol has also been used for a data source to open a socket directly
to a PMDA and tie into the PMDA's select loop, rather than going via pmcd.


For the web-based frontend issue, yeah, javascript+svg+etc. sounds most promising, especially if it can be made to speak the native wire protocol to pmdc. This would seem to argue for a stateful archive-serving pmdc, or perhaps a archive-serving proxy, as in Greg's old project.

Time averaging, aggregation and filtering were all ambitious aims of the project Greg's talking about - I wonder if that code could ever be resurrected and open sourced? One abomination here was that a PMDA could also be a client - and potentially query itself for metrics(!)

Is this sounding reasonable?


it's going to take a lot more discussion, but enthusiasm seems to be on our side :)

Cheers
-- Mark Goodwin


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]