This is the mail archive of the cygwin mailing list for the Cygwin project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: download cygwin packages in multiple session


Igor Pechtchanski <pechtcha <at> cs.nyu.edu> writes:

> 
> On Thu, 15 Dec 2005, Cj2k5 wrote:
> 
> > Hi all
> >
> > I have created a shell script because I dont have an internet connection
> > and this script allow me to do a dowload session limited in n Mbytes
> > using wget. I then fill a usb flash memory key and go home to update my
> > local cygwin miror. Then repeat the session.
> >
> > The script compare a latest setup.ini with a local miror of cygwin
> > packages and generate multiple text file that will be use by wget.
> >
> > I did not have my script when sending this message, but tomorrow I will
> > look for reply to this message and if there are people interesting to
> > it, I will post the script to a new message.
> 
> This may not be apparent from that FAQ entry, but Michael Chase's
> clean_setup.pl, referenced from [1], does that and more.  It would be good
> to know how your script differs from it...
> 	Igor
> [1] <http://cygwin.com/faq/faq.setup.html#faq.setup.disk-space>

Hi Igor

I submit a second version of the shell script. It correct some bugs
and have new features.

I did not yet look at the clean_setup.pl script, but I will today.

If it's not the appropriate place to post this shell script, let me know.
I will try to find an other place.

Regards, Cj2k5



file: dl-cygwin.doc
--------------------------------------------------------------------

Why?

    This script is for people that do not have
    internet access and have to download cygwin
    in multiple session. The reason because setup.exe
    cannot help in this situation, is that it must 
    have access to a local miror of the packages if 
    a user want to make an update to new packages.

    To get these files, a user will have to go at 
    a Public Internet Access (internet cafe,
    library etc) and use a USB flash memory key
    to transport packages files. 

    Some PIA manager would not allow a user to 
    monopolise a station if there are more then 
    one transport of package files to do, 
    or they wont let them leave temporary 
    file on hardisk (not secure).

    Using this shell script, 
    1. user can download a fresh setup.ini,
    2. generate a list of missing packages,
    3. download the files at a PIA using wget,
    4. go to home and update local miror,
    5. repeat 3 and 4 if still missing files.

What is it?

    This is a shell script that generate listing of
    package files that are missing in a cygwin local
    release directory, base on a new setup.ini file.

    It can also do a md5 checksum on downloaded files.

    The shell script depend on:

    - sh, gawk, sort, uniq, cat and sed commands
    - a new setup.ini file
    - and optionaly an relatively old local 
      package repository.

    The script do not manage cygwin instalation nor 
    manage local packages miror, it only generate
    text files.

    These text files can be use to manage a local 
    packages miror in a shell session, 
    or use with wget as a list of files to download.

    The generated list are splited base on a total
    of packages bytes size limit given as argument.
    It default to 60000000 bytes. To generate only 
    one file, use '-s 0' in the argument list.

    There are 7 type of listing that can be genrated:

    1. listing for all the packages bin (_DL_new_all_bin-nnn.txt)
    2. listing for all the packages sources (_DL_new_all_src-nnn.txt)
    3. partial listing from existing bin packages (_DL_new_bin-nnn.txt)
    4. partial listing from existing sources packages (_DL_new_src-nnn.txt)
    5. listing for all the packages to [re]move (_mov.txt)
    6. report on total files and total size per file list
       (_00report_all.txt, _00report.txt)
    7. 3 listing of checksum on packages for: 
       - files that dont move and pass the checksum (_md5_ok.txt),
       - files that should be move and cannot be check (_md5_mov.txt),
       - and files that fail the checksum (_md5_no.txt).

Availables options:

      --help -h        print this help and exit
      --version -v     print version information and exit
      -ab              listing for all the packages bin
      -as              listing for all the packages sources
      -pb              partial listing from existing bin packages
      -ps              partial listing from existing sources packages
      -d               debug (will not erase temp files)
      -m               create a [re]move list (release dir against setup.ini)
      -c               compare with md5 checksum (-r dir against setup.ini)
      -i file          new setup.ini file
      -s bytes         file group size limit in byte (default 60000000),
                       ('-s 0' mean generate only one file)
      -r dir           location of the local 'release' miror directory
                       (default ./release)
      -t dir           target directory you want the files generated
                       (default ./results)"

Todo

    - better doc and examples
    - generate new setup.ini files for use with setup.exe
    - more test
    - check if it's useseful for me (done)
    - check if it's realy useseful for the others

Cj2k5

--------------------------------------------------------------------
file: dl-cygwin.sh
--------------------------------------------------------------------
#!/bin/sh
#
# Copyright (C) 2005 Free Software Foundation, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#

# partial download from setup.ini
# depend on sort, uniq and gawk

progname=$0
package=cygwin-etc
version=0.0.3

# for creating groups listing for all the packages bin
grpdlallbin=no
# for creating groups listing for all the packages sources
grpdlallsrc=no
# create a [re]move list
movelist=no
# create a md5 check list
checklist=no
# for creating partial listing for binaris from already existing packages
grpdlpartbin=no
# for creating partial listing for sources from already existing packages
grpdlpartsrc=no
# for debug do not erase temp files
debug=no
# get setup.ini file from the script argument
inifile=setup.ini
# file group size listing in byte
grpsise="60000000"
# location of the local 'release' miror directory
releasedir=./release
# target directory you want the files generated
targetdir=./results

# func_usage
# outputs to stdout the --help usage message.
func_usage ()
{
  echo "\
Usage: $progname [OPTION]...

Create miscelanous list of files to help manage or update cygwin packages.
Generated files are splited base on group of packages files size limit.

Options:
      --help -h        print this help and exit
      --version -v     print version information and exit
      -ab              listing for all the packages bin
      -as              listing for all the packages sources
      -pb              partial listing from existing bin packages
      -ps              partial listing from existing sources packages
      -d               debug will not erase temp files
      -m               create a [re]move list (release dir against setup.ini)
      -c               compare a md5check list (release dir against setup.ini)
      -i file          new setup.ini file
      -s bytes         file group size limit in byte (default 60000000),
                       ('-s 0' mean generate only one file)
      -r dir           location of the local 'release' miror directory
                       (default $releasedir)
      -t dir           target directory you want the files generated
                       (default $targetdir)"
  echo "
Written by Jacques Champagne
Report bugs to <whereyoufindit.bugs>."
}

# func_version
# outputs to stdout the --version message.
func_version ()
{
  echo "$progname (GNU $package) $version"
  echo "Copyright (C) 2005 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE."
  echo "Written by" "Jacques Champagne"
}

# Command-line option processing.
# Removes the OPTIONS from the arguments. Sets the variables:
# I do not fully understand this arg opts mecanism, 
# I use it anyway. Cj2k5
{
  while test $# -gt 0; do
    case "$1" in
      -ab  )
        shift
        grpdlallbin=yes ;;
      -as  )
        shift
        grpdlallsrc=yes ;;
      -pb  )
        shift
        grpdlpartbin=yes ;;
      -ps  )
        shift
        grpdlpartsrc=yes ;;
      -d  )
        shift
        debug=yes ;;
      -c  )
        shift
        checklist=yes ;;
      -m  )
        shift
        movelist=yes ;;
      -i  )
        shift
        inifile=$1
        shift ;;
      -r  )
        shift
        releasedir=$1
        shift ;;
      -s  )
        shift
        grpsise=$1
        shift ;;
      -t  )
        shift
        targetdir=$1
        shift ;;
      --help | --hel | --he | --h | -h )
        func_usage ; exit 0 ;;
      --version | --versio | --versi | --vers | --ver | --ve | --v | -v )
        func_version ; exit 0 ;;
      -- ) # Stop option prcessing
        shift; break ;;
      -* )
        echo "$progname: unknown option $1" 1>&2
        echo "Try '$progname --help' for more information." 1>&2
        exit 1 ;;
      * )
        break ;;
    esac
  done
}

# Command-line argument processing.
# Analyzes the remaining arguments.
{
  if test $# -gt 0; then
    func_usage 1>&2
    exit 1
  fi
}

releasedir=`echo $releasedir|sed -e 's,/$,,'`
# test if release dir exist echo "$releasedir"
if ! test -d $releasedir ; then 
echo "$releasedir does not exist"
exit 1
fi

targetdir=`echo $targetdir|sed -e 's,/$,,'`
# test if target dir exist echo "$targetdir"
if ! test -d $targetdir ; then 
echo "$targetdir does not exist"
exit 1
fi

# test if setup.ini file exist
if ! test -f $inifile ; then 
echo "$inifile not found"
exit 1
fi

# needed by gawk
export grpsise targetdir releasedir

# clean up old list files
rm $targetdir/_*txt
 
# generate a global download list from setup.ini file for binary download
if test $grpdlallbin != no ; then
gawk '/^install:/ {fn=$2;s=$3;m=$4;\
        line= s " " fn " " m;\
        t=sub("[^/]+$","",fn);\
        if (fn!=lfn) {line= line " " fn "setup.hint " fn "md5.sum"; lfn=fn};\
        print line;}' $inifile | gawk '\
BEGIN{\
    ofileA= ENVIRON["targetdir"] "/_DL_new_all_bin-";\
    ofileB= ".txt";\
    ofile= ofileA "000" ofileB;\
    result= ENVIRON["targetdir"] "/_00report_all.txt"};\
/^/ {fn=$2; setup=$4; md5sum=$5; t=$1; tt+=t; u+=1;\
 if (tt>ENVIRON["grpsise"] && ENVIRON["grpsise"]+0!=0) {\
    fflush(ofile);\
    print "file " ofile >> result;\
    print "total files " u-1 >> result;\
    print "total size " tt-t >> result;\
    print "" >> result;\
    tt=t; u=1; ttt+=1; ttf="" ttt;\
    if (ttt<10) {ttf="00" ttt};\
    if (ttt>9 && ttt<100) {ttf="0" ttt};\
    ofile=ofileA ttf ofileB;}\
 print fn >> ofile;\
 if (setup) {u+=1; print setup >> ofile}\
 if (md5sum) {u+=1; print md5sum >> ofile}}\
END{\
    print "file " ofile >> result;\
    print "total files " u >> result;\
    print "total size " tt >> result;\
    print "" >> result;\
    print "number of bin listing group " ttt+1 >> result;\
    print "" >> result}'
fi

# generate a global download list from setup.ini file for sources download
if test $grpdlallsrc != no ; then
gawk '/^source:/ {print $2 " " $3;}' $inifile \
| sort | uniq | gawk '\
BEGIN{\
    ofileA= ENVIRON["targetdir"] "/_DL_new_all_src-";\
    ofileB= ".txt";\
    ofile= ofileA "000" ofileB;\
    result= ENVIRON["targetdir"] "/_00report_all.txt"};\
/^/ {fn=$1; t=$2;\
 if (fn!=lfn) {tt+=t; u+=1;\
   if (tt>ENVIRON["grpsise"] && ENVIRON["grpsise"]+0!=0) {\
    fflush(ofile);\
    print "file " ofile >> result;\
    print "total files " u-1 >> result;\
    print "total size " tt-t >> result;\
    print "" >> result;\
    tt=t; u=1; ttt+=1; ttf="" ttt;\
    if (ttt<10) {ttf="00" ttt};\
    if (ttt>9 && ttt<100) {ttf="0" ttt};\
    ofile=ofileA ttf ofileB};\
    print fn >> ofile;\
    lfn=fn}};\
END{\
    print "file " ofile >> result;\
    print "total files " u >> result;\
    print "total size " tt >> result;\
    print "" >> result;\
    print "number of src listing group " ttt+1 >> result;\
    print "" >> result};' 
fi

# generate a list of files to [re]move 
if test $movelist != no ; then
find $releasedir -iname '*' -type f -print > $targetdir/_mov_all_old.txt

gawk '/^install:/ {fn=$2;\
        line= fn;\
        t=sub("[^/]+$","",fn);\
        if (fn!=lfn) {line= line "\n" fn "setup.hint\n" fn "md5.sum"; lfn=fn};\
        print line;}' $inifile > $targetdir/_mov_all_new.tmp
gawk '/^source:/ {print $2;}' $inifile >> $targetdir/_mov_all_new.tmp

cat $targetdir/_mov_all_new.tmp |sort|uniq|\
    sed -e 's,release/,'$releasedir'/,' > $targetdir/_mov_all_new.txt

cat $targetdir/_mov_all_old.txt |gawk '\
BEGIN {i=0;\
  c = "cat " ENVIRON["targetdir"] "/_mov_all_new.txt" ;\
  while ((c | getline) > 0) {i++; f[i] = $0}; close(c)}\
/^/ {fd=0; s=$0; fn=$0; t=sub("[^/]+$","",fn);\
  for (n in f) {if (f[n] == s) {fd=-1}};\
  if (!fd) {print s;\
    if(fn!=lfn) {print fn "setup.hint";\
      print fn "md5.sum"; lfn=fn}}}' |sort|uniq  > $targetdir/_mov.txt
 if test $debug = no ; then
  rm $targetdir/_mov_all_new.tmp $targetdir/_mov_all_new.txt
$targetdir/_mov_all_old.txt
 fi
fi

# generate a list of md5 checksum of package files and compare with setup.ini md5
if test $checklist != no ; then
gawk '/^install:/ {print $4 " " $2;}' $inifile > $targetdir/_md5_all_new.tmp
gawk '/^source:/ {print $4 " " $2;}' $inifile >> $targetdir/_md5_all_new.tmp
cat $targetdir/_md5_all_new.tmp |\
    sed -e 's, release/, '$releasedir'/,' |sort > $targetdir/_md5_all_new.txt

find $releasedir -iname '*bz2' -type f -exec md5sum '{}' ';' \
     |sed -e 's, [*], ,' > $targetdir/_md5_all_old.txt

cat $targetdir/_md5_all_old.txt |gawk '\
BEGIN {i=0;j=0;k=0;\
  c = "cat " ENVIRON["targetdir"] "/_md5_all_new.txt" ;\
  while ((c | getline) > 0) {i++; f[i] = $0;}; close(c)};\
/^/ {fd=0; m=$1; s=$2; t=$0; ok=""; no="";\
  for (n in f) {if (index(f[n],s)) {fd=-1;\
     if (index(f[n],m)) {ok=t} else {no=t} }};\
  if (!fd) {print m " " s >> ENVIRON["targetdir"] "/_md5_mov.txt"};\
  if (ok) {print ok >> ENVIRON["targetdir"] "/_md5_ok.txt"};\
  if (no) {print no >> ENVIRON["targetdir"] "/_md5_no.txt"}}'\
  > $targetdir/_md5_errors.txt 2>&1
 if test $debug = no ; then
  rm $targetdir/_md5_all_new.tmp $targetdir/_md5_all_new.txt
  rm $targetdir/_md5_all_old.txt $targetdir/_md5_errors.txt
 fi
fi

# generate a partial download list from existing local packages files
# to create group list files for binary download
if test $grpdlpartbin != no ; then
    gawk '/^install:/ {fn=$2;s=$3;m=$4;\
    line= s " " fn " " m;\
    t=sub("[^/]+$","",fn);\
    if (fn!=lfn) {\
      line= line " " fn "setup.hint " fn "md5.sum";\
      lfn=fn}; print line;}' \
    $inifile |sed -e 's, release/, '$releasedir'/,' |gawk '\
    BEGIN{\
       ofileA= ENVIRON["targetdir"] "/_DL_new_bin-";\
       ofileB= ".txt";\
       ofile= ofileA "000" ofileB;\
       result= ENVIRON["targetdir"] "/_00report.txt"};\
    /^/ {md5=$3; fn=$2; setup=$4; md5sum=$5; c="md5sum " fn;\
     while ((c|getline)>0) {r=$1; continue}; close(c);\
     if (match(md5,r)){}else{\
       t=$1; tt+=t; u+=1;\
       if (tt > ENVIRON["grpsise"] && ENVIRON["grpsise"]+0!=0) {\
         fflush(ofile);\
         print "file " ofile >> result;\
         print "total files " u-1 >> result;\
         print "total size " tt-t >> result;\
         print "" >> result;\
         tt=t; u=1; ttt+=1; ttf="" ttt;\
         if (ttt<10) {ttf="00" ttt};\
         if (ttt>9 && ttt<100) {ttf="0" ttt};\
         ofile=ofileA ttf ofileB;};\
       t=sub(ENVIRON["releasedir"] "/","release/",fn);\
       print fn >> ofile;\
       if (setup) {u+=1; print setup >> ofile;}\
       if (md5sum) {u+=1; print md5sum >> ofile;};};\
    }; END{\
           print "file " ofile >> result;\
           print "total files " u >> result;\
           print "total size " tt >> result;\
           print "" >> result;\
           print "number of bin listing group " ttt+1 >> result;\
           print "" >> result};' > $targetdir/_DL_new_bin_errors.txt 2>&1
 if test $debug = no ; then
  rm $targetdir/_DL_new_bin_errors.txt
 fi
fi

# generate a partial download list from existing local packages files
# to create group list file for sources download
if test $grpdlpartsrc != no ; then
    gawk '/^source:/ {print $2 " " $3 " " $4;}' $inifile \
| sort | uniq | sed -e 's,^release/, '$releasedir'/,' | gawk '\
    BEGIN{\
       ofileA= ENVIRON["targetdir"] "/_DL_new_src-";\
       ofileB= ".txt";\
       ofile= ofileA "000" ofileB;\
       result= ENVIRON["targetdir"] "/_00report.txt"};\
    /^/ {fn=$1; t=$2; md5=$3; c="md5sum " fn;\
     while ((c|getline)>0) {r=$1; continue;}; close(c);\
     if (match(md5,r)){}else{tt+=t; u+=1;\
           if (tt>ENVIRON["grpsise"] && ENVIRON["grpsise"]+0!=0) {\
             fflush(ofile);\
             print "file " ofile >> result;\
             print "total files " u-1 >> result;\
             print "total size " tt-t >> result;\
             print "" >> result;\
             tt=t; u=1; ttt+=1; ttf="" ttt;\
             if (ttt<10) {ttf="00" ttt};\
             if (ttt>9 && ttt<100) {ttf="0" ttt};\
             ofile=ofileA ttf ofileB;};\
           t=sub(ENVIRON["releasedir"] "/","release/",fn);\
           print fn >> ofile; };\
    }; END{\
           print "file " ofile >> result;\
           print "total files " u >> result;\
           print "total size " tt >> result;\
           print "" >> result;\
           print "number of src listing group " ttt+1 >> result;\
           print "" >> result};' > $targetdir/_DL_new_src_errors.txt 2>&1
 if test $debug = no ; then
  rm $targetdir/_DL_new_src_errors.txt
 fi
fi


----------------------------------------------------------------------



--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]