Re: [pcp] suitability of PCP for event tracing

[Ken McDonell <kenj at internode dot on dot net>]

Apologies for my tardiness in responding, but I'm travelling at the 
moment (typing this on a train and then on a ferry somewhere in Norway).

On 1/09/2010 5:49 AM, Frank Ch. Eigler wrote:
> Hi -
>
> Thanks, Nathan, Ken, Greg, Mark, for clarifying the status quo and
> some of the history.
>
>  ...
>
> (2) protocol extensions for live-push on pmda and pmcd-client interfaces
>      This clearly larger effort is only worth undertaking with the
>      community's sympathy and assistance.  It might have some
>      interesting integration possibilities with the other tools,
>      espectially pmie (the inference engine).

I'd like to go back to a comment Nathan made at the start of this 
thread, namely to try and get a clear idea of the problem we're trying 
to solve here and the typical use cases.  I think it is important to get 
all of this on the table before we start too much of a discussion about 
possible evolutionary change for PCP (something I am very supportive of, 
in general terms).

Some of the suggestions to date include ...

+ being able to push data from pmcd asynchronously to clients, as 
opposed to the time-based pulling from the clients that we support today

+ data filtering predicates pushed from a client to pmcd and then on to 
a pmda to enable or restrict the types of events or conditions on event 
parameters that would be evaluated before asynchronously sending 
matching events to the client

+ handling event records with associated event parameters as an extended 
data type

+ additional per-client state data being held in pmcd to allow rate 
aggregation (and similar temporal averaging) to be done at pmcd, rather 
than the client [note I have a long-standing objection to this approach 
based on the original design criteria that pmcd needs to be mean and 
lean to reduce impact, and data reduction and analysis should be pushed 
out to the clients where the computation can be done without impacting 
the system being monitored ... but maybe it is time to revist this, as 
the current environments where PCP is being used may differ from those 
we were concerned with in 1994]

+ better support for web-based monitoring tools (although Javascript 
evolution may make this less pressing that it was 5 years ago)

+ better support for analysis that spans the timeline between the 
current and the recent past

This is already a long list, with the work items spanning about 2 orders 
of magnitude of effort.  It would be good to drive towards consensus on 
this list of items, and then prioritizing them.

Depending on the set of goals we agree on, there may even be a place to 
consider maintaining the poll-based mechanism, but the export data is a 
variable length buffer of all event records (each aggregated and 
self-identifying as below) seen since the last poll-based sample.

Returning to Frank's point, I'm not sure pmie would be able to consume 
asynchronous events ... it is already a very complicated predicate 
engine with the notion of rules being scheduled and evaluated with fixed 
(but not necessarily identical) evaluation intervals for each rule. Some 
of the aggregation, existential, universal and percentile predicates 
don't have sound semantics in the presence of asynchrounous data 
arrival, e.g. some_inst(), all_inst(), count_sample(), etc.

> For the static-pmns issue, the possibility of dynamic instance
> domains, metric subspaces is probably sufficient, if the event
> parameters are limited to only 1-2 degrees of freedom.  (In contrast,
> imagine browsing a trace of NFS or kernel VFS operations; these have
> ~5 parameters.)

I am not sure this is a problem.  Each event has a unique timestamp, so 
each parameter could be encoded as a PCP metric of the appropriate type 
and semantics. If that is not adequate, then the best approach would 
seem to be to extend the base data types to include some sort of 
self-encoded aggregate ... I don't have a strong view on which of the 
existing "standards" should be adopted here, but it does not appear to 
be a hard problem at the PCP protocol layer ... constructing the 
aggregate would be PMDA (or similar) responsibility and interpretation 
of the aggregate is a client responsibility, although it would be more 
consistent with the PCP approach if the aggregate included the semantics 
and metadata for the parameters, even if this is only delivered once per 
client connection.

> ...

On 3/09/2010 5:39 AM, Frank Ch. Eigler wrote:
> ...
>
> OK, then it looks like we'd have at least a few separate pieces to
> work on:
>
> * extensions to the PMCD<->PMDA API/protocol to allow PMDAs to push
>    event data, and corresponding extensions for PMclients<->PMCD

I'd really like to see some more discussion on how people think this is 
going to work.  None of the PCP libraries are thread-safe (again a 
deliberate design decision at the original point of conception), and 
asynchronous delivery of data from pmdas through pmcd to clients 
increases the likelihood that people will want to use multiple threads 
to handle PCP calls.  There are some asynchronous calls that were 
grafted onto libpcp later on, but these have very little use in existing 
code and no QA coverage.

> * teaching some of the existing clients to process such data

As I mentioned above, I think we need to preserve the metadata concepts 
in the PCP protocols so that this data does not become opaque and only 
understood by the producer and consumer (one of my long-stranding 
complaints about SMNP and the MIB concept which PCP has so far done a 
better job of addressing).

> * a systemtap PMDA that listens to pmStore filtering/control
instructions;
>    probably using plain type STRING for JSON payload

Currently there is no client identification (and hence no notion of a 
session) that is passed down from pmcd to the pmdas, so how would this 
filtering work in presence of multiplexed requests coming from a number 
of clients?  And when would the fitlering stop?  And is it possible that 
multiple clients could request filter predicates that are mutually 
exclusive?

> * a PMCD<->XMLRPC bridge

I am not sure that pmcd is the right place to put this ... if the bridge 
was client of pmcd, this would be more PCP-like, and match the way in 
which pmproxy (and to a lesser extent derived metrics) are supported.

> * the web application itself

I'm not a web guy, but this seems the simplest piece of the puzzle ... 8^)>

I suspect the proposals here are substantive enough that they require a 
white paper and discussion, rather than a convoluted email thread.  If I 
could get some feedback and answers to my questions, I'd be happy to put 
together an initial document to guide the discussion ... if someone else 
wants to drive, that's fine by me also.