The checking for python problem (was Re: Can RPM packages be installed into Cygwin?)

Linda Walsh cygwin@tlinx.org
Tue Apr 17 07:02:00 GMT 2012


Buchbinder, Barry (NIH/NIAID) [E] wrote:

> De-Jian Zhao sent the following at Monday, April 09, 2012 11:12 AM
>>> On 4/8/2012 5:10 AM, De-Jian Zhao wrote:
>>>> On 2012-4-7 16:12, marco atzeri wrote:
>>>> Thanks, Marco. I was not trying to run a native linux apps, and instead
>>>> tried to rebuild the app from source. The BLAST software provides the
>>>> Build.sh and Check.sh scripts under the directory of
>>>> ncbi-blast-2.2.26+-src.tar\ncbi-blast-2.2.26+-src\c++\compilers\cygwin\.
>>>> So I think it is possible to run the software under cygwin. However, the
>>>> compilation stopped at "checking for python", even if python was
>>>> installed. That's to say, I installed the latest python and the
>>>> compilation could find it, but the compilation tried to search for
>>>> python2.3 and 2.4 further, and stopped when searching for 2.4.

----------------

I ran into a sim bug in the rpm build script.

It's python component doing a retard search on /usr/src/packages/BUILD...
(at least that was my prob)...

The culprit on my system was:
"brp-check-python" in /usr/lib/rpm.

It does a 'find' in /usr/src/packages/BUILD of all 'c' files using
a horrible N^2 algorithm (time is proportional to square of # files):

     find /usr/src/packages/BUILD/ -iname '**.c' -print0 | \
      xargs -r0 python /usr/lib/rpm/brp-check-python.data/ssizecheck.py > $TMPDIR/$$

Why they would use find, then use '**', which checks each inode to see
if it matches any pattern ending in *.c  -- recursive through all directories
for each inode.  So if you have 20 subdirs total, it would be 400 deep,

The time to run just a regular find
on my system with about 3815 subdirs (not files, but subdirs (57239 inodes!) -- 
was ~30 seconds the first time -- the 2nd time <1second (but I have a large
system cache).

Windows does not have a large system cache (my system is configured for
'large cache - file serving' and only has a 1GB file cache (as measured
by process explorer) If you have mem configured for programs like most users, 
drop that in HALF or LESS depending on your memory.

I.e. if you have alot of files in BUILD and non-RAID disk,  or
worse -- a network disk, it will be slow -- but worse .. is the added
cygwin overhead for 'find' to do a full stat.  It has to open/close each
file to give full POSIX semantics!..   Then it is doing the recursive
match on each inode... OUCH!!!!!!

Sorry for the long winded explanation, but I wanted you to understand
how horribly long that process might take .. with alot of files, several minutes
would not be long.  If I had those linux files on my windows sys and did the 
same, it would likely take closer to 30 minutes or longer.


Next time, you see a process stuck like that -- use process explorer, or process 
hacker to get clues... and for detail, use process Monitor (search for all 3 on 
google)... hacker has the most sophisticated and dangerous features, proc-exp 
has been dumbed down as it's owned by MS -- but is still more than enough for 
most people, and proc-mon is like trying to read a book with a microscope...
ALOT of output... but you'll likely find the active culprit being that script

(at least from your description -- and given that message was resulting in a 
doubling or more of build times on a non-python project....)...

FYI -- my 'repair' on my system to the file was to add an 'exit 0' as the
first statement... the file doesn't seem to do anything useful anyway...
something about checking for files in BETA...


/usr/lib/rpm> diff -u brp-check-python.orig brp-check-python
--- brp-check-python.orig       2011-01-13 05:25:30.000000000 -0800
+++ brp-check-python    2012-01-16 01:49:12.616412621 -0800
@@ -1,4 +1,5 @@
  #!/bin/sh
+exit 0
  # If using normal root, avoid changing anything.
  if [ -z "$RPM_BUILD_ROOT" -o "$RPM_BUILD_ROOT" = "/" ]; then
          exit 0

---
That file does a


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple



More information about the Cygwin mailing list