Running 4096 parallel cat processes on remote share results in only 1018 succeeding

[Nathan Fairchild]

When I run a script like so: 

#!/bin/bash 
ulimit -n 3200 
for i in $(seq $1) 
do exec cat /u/pe/env_files/transpath.map > out$i.log & 
done 

Where /u maps to a Windows share in /etc/fstab I get the following when running with 4096 instances (this happens whether I use a mapped /etc/fstab path or UNC path): 

$ ./run_many.sh 4096 
cat: /u/pe/env_files/transpath.map: No such file or directory 
cat: /u/pe/env_files/transpath.map: No such file or directory 
cat: /u/pe/env_files/transpath.map: No such file or directory 
cat: /u/pe/env_files/transpath.map: No such file or directory 
cat: /u/pe/env_files/transpath.map: No such file or directory 
./run_many.sh: fork: retry: Resource temporarily unavailable 
./run_many.sh: fork: retry: Resource temporarily unavailable 
./run_many.sh: fork: retry: Resource temporarily unavailable 
./run_many.sh: fork: retry: Resource temporarily unavailable 
./run_many.sh: fork: Resource temporarily unavailable 

$ grep -l PATH out* | wc -l 
1018 


I think I'm probably hitting the 256 process limit because of the I/O slowdown the network presents? I don't get this issue running on (much faster) local disk. 

$ ulimit -a 
core file size          (blocks, -c) unlimited 
data seg size           (kbytes, -d) unlimited 
file size               (blocks, -f) unlimited 
open files                      (-n) 3200 
pipe size            (512 bytes, -p) 8 
stack size              (kbytes, -s) 2025 
cpu time               (seconds, -t) unlimited 
max user processes              (-u) 256 
virtual memory          (kbytes, -v) unlimited 


Is there a way to increase max user processes so I don't run into this limit? For my workload I need about 256 per cpu core (call it 8192), not 256 total. 

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple