RE: limiting strace?, was: failure of unzip and recent cygwin1.dll

[Igor Pechtchanski <pechtcha at cs dot nyu dot edu>]

On Wed, 18 Feb 2004, Ross Boulet wrote:

> > On Wed, 18 Feb 2004, Ross Boulet wrote:
> >
> [...snip...]
> > > Just adding my 2 cents and hopefully to learn something:
> > >
> > > I know how to use pipes but I don't know how they are handled "under
> > > the covers".  Wouldn't piping the strace to tail or grep still
> > > produce a temporary file of a large size?  I don't know if the
> > > original poster is concerned with disk space or only the output.
> > > If it's a space issue and a temporary file is created for the pipe,
> > > I can see why he would want to limit the strace output directly.
> > > If its just the result, it seems tail or grep would be good options.
> > >
> > > Ross
> >
> > Pipes are direct process-to-process communication devices in most
> > (all?) modern operating systems.  Older systems (read: MS-DOS)
> > didn't have true pipes, and had to emulate them with redirection to
> > temporary files. AFAIK, Cygwin doesn't use temporary files for pipes
> > on any of the systems that it runs on.
> >       Igor
>
> Just for my own edification and at the risk of being slightly OT....
>
> When you run:
>
> $ foo | bar
>
> I'm assuming the output from foo is stored in memory and then passed to
> bar as the output is generated.  I deduce this from experience when I run
>
> $ foo | less
>
> Less starts processing the output as it comes in.  Less would obviously
> require memory to store the entire output to facilitate paging back.
> However, grep would be looking at each line and only keeping what its been
> told.  So in
>
> $ foo | grep sometext
>
> Is the memory consumed by output from foo freed as soon as grep reads it and
> sees that sometext is not in the output line?

Ok, time for "Pipes 101".  A pipe is essentially a bounded (character)
buffer.  The producer application fills the buffer (by writing to the
pipe), and the consumer application empties it (by reading from the pipe).
If the buffer is full, the producer application blocks (inside the write)
until room is available.  If there's no data to read, the consumer
application blocks.  The amount of pipe data stored in memory is
determined only by the size of the buffer.  For unbuffered pipes, the
amount is 1 character.

Once the data is read from the buffer, its further fate is up to the
consumer application.  Some applications (like grep) act as filters, and
require only enough memory for the current line.  Some (like sort) will
read the whole input and store it in memory before processing it (or use
temporary files internally, but these have nothing to do with the pipe
itself).  If you're interested in the behavior of any particular
application, I suggest you look at the sources of the corresponding
package (available for *all* Cygwin packages).
	Igor
P.S. FYI, less doesn't necessarily read all of the input, and has options
limiting the amount of allocated memory.
-- 
				http://cs.nyu.edu/~pechtcha/
      |\      _,,,---,,_		pechtcha@cs.nyu.edu
ZZZzz /,`.-'`'    -.  ;-;;,_		igor@watson.ibm.com
     |,4-  ) )-,_. ,\ (  `'-'		Igor Pechtchanski, Ph.D.
    '---''(_/--'  `-'\_) fL	a.k.a JaguaR-R-R-r-r-r-.-.-.  Meow!

"I have since come to realize that being between your mentor and his route
to the bathroom is a major career booster."  -- Patrick Naughton

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/