mirror.sh

A web page containing 4 files: manual, notes, mirror (sh script) and Params (sh script), the separations of which should be failry obvious. This is a script program which mirrors any set of directories recursively, and any set of files from one machine to others by updating the remote files if a change has been made to them. It is assumed that the machines have uucp connection. The existence of certain mathematical C programs 'plus', in particular is also assumed. See the web pages on C programs (in the future). 'plus' is being revised to handle integers of arbitrary size. In the mean time:


   The locution
                    k = `plus -i 1 $k`

   can be replaced with the morr conventional expression

                    k=`expr 1 + $k`

Various alterations may have to be made to accommodate uxix type sytems other than the now semidefunct Xenix System V systems.

MIRROR(LOCAL) XENIX System V (Lambda) MIRROR(LOCAL) NAME mirror - uucp file and directory update/backup system SYNOPSIS mirror -[S | M | m | U | A] [-D] mirror -[S | M | m | U ] [-P] [-R ] [-D] mirror -[M | m | U ] [-P] [-N] [-R ] [-D] DESCRIPTION The mirror system automatically updates remote system files by uucp from local system files and allows local files to be updated by remote systems, by providing the remote system with an available database of checksums. INDICATIONS A network of uucp nodes each of which generates different files and for which it is desirable that every node in the net work maintain updated mirrors of some subset of the totality of files created in the network. A mirror system (topology file) is a node of an arbitrary uucp network with any number of nodes, but a mirror node communicates only with its nearest neighbors in the network. The transmissions are not directed through nodes, but only with a node directly: there is no node hopping. Each node must know, through its configuration files, its place in the network. The coordination of all the configuration files will depend on which node generates what, and what is being mirrored at each node. Every mirror node must have: the mirror script and associated utilities the directory setup in /usr/lib the correctly topologizing files /usr/lib/mirror/systems/slaves /usr/lib/mirror/systems/masters crontab entries for the running of mirror -S and mirror -M or mirror -S and mirror -m, and mirror -U that may have to be coordinated with the crontab entries on the relative slave and master systems of nearest neighbor nodes. The system centered in the shell script /usr/lib/mirror/mirror which should be scheduled to run periodically as above by cron. Page 1 (printed 11/11/97) MIRROR(LOCAL) XENIX System V (Lambda) MIRROR(LOCAL) It requires the compiled utilities: gtln, gtfield, chmog, chmogdir, plus the utility: creat, the topologizing database files: /usr/lib/mirror/systems/slaves - list of systems that we update, one system per line. /usr/lib/mirror/systems/masters - list of systems that update us one system per line. (The lists are not necessarily mutually exclusive: the direction of the slave - master relationship depends on which files are mirrored node A to node B, and which files are mirrored node B to node A.) The update priority of systems by master mode is the order of their listing in the file /usr/lib/mirror/systems/slaves. Not as significant, but the order in which the sum files constructed for remote master systems is the order in which the the master systems are listed in the file /usr/lib/mirror/systems/masters. NB Care should be taken, in the coordinated file setup, with regard to any single file F, that the following is avoided: Node A uses its F to update the F of node B and Node B uses its F to update the F of node A This can happen surreptitiously by passage through various nodes of the network. For any F, a path can be drawn through the complete graph associated with all the nodes of the network. THE LAW is then that for every F the graph that is its path must have no self-intersections. Notice that, in general, the path of F can, and most probably will bifurcate. Therefore every F dissemination path MUST be "a connected tree graph". and directories: Page 2 (printed 11/11/97) MIRROR(LOCAL) XENIX System V (Lambda) MIRROR(LOCAL) /usr/lib/mirror/systems/ containing files "slave" and "master" /usr/lib/mirror/slaves/ containing subdirectories named for each slave system /usr/lib/mirror/masters/ containing subdirectories named for each master system In each of the the last two system subdirectories there are two files files, dirs which list the full pathnames of the files and directories that are to be updated remotely if under the slave hierarchy, or that update our files if under the master hierarchy. [NOTE that the directory entries in the 'dirs' files will be treated recursively down through subdirectories; all files in the named directory AND in any subdirectories and subdirectories of those subdirectories etc., are files to be mirrored.] Also the file, sums which is a file requested from a remote slave system, if under the slave hierarchy and contains the slave's checksum information, or is the local checksum file put in place for retrieval by a remote master system. Every system produces locally in each of the slave system subdirectories a file named lsums These contain the checksums for the files that are used to update a remote slave system's files. When mirror running in master mode sees that the sums file requested from the slave system has a checksum for a given file that does not match the local version, the slave system's file updated with the local version. That is, it is queued for uucp transmission. The actual sending is left to local uucp scheduler. FILE STRUCTURE In the systems directory, the files slaves and masters list, one per line, the uucp names of the systems that are slave and master to the local system. A given remote system may be master with respect to one set of files and directories and slave with respect to another necessarily distinct and different set of files and directories. Thus a given system name may appear in either or both of these files. A line beginning with '#' followed by a space or tab is considered to be a comment line. Page 3 (printed 11/11/97) MIRROR(LOCAL) XENIX System V (Lambda) MIRROR(LOCAL) The 'files' and 'dirs' files may be commented similarly. Both of these files list the files and directories that are to be mirrored. There are two columns. The first column contains the full local pathname, while the second separated from the second by TABS, contains the full remote pathname. PERMISSIONS All files and directories should be readable and writable by owner and group (chmod 664) with owner 'uucp' and group 'root'. OPTIONS -D Turn on the debugging flag. Compatible with all options. The output is quite voluminous and should probably take and option to determine a debugging level. -A Run in interactive administrative mode to add systems, remove systems and edit the topology file as well as the 'files' and 'dir' files for the slave or master aspect of any system. -R Takes as argument the name of a mirror system. When mirror is run with any of the options M, m, U, or S, that mode operation is confined to only the specified system. This option is not compatible with the A option. -S Run in slave mode. -U Run in local update mode. turned on. The default is off. -m Run in master (distributing) mode. -M Run in complete master mode: do the work specified in both the -U and -m options. -N Do not actually queue files for transfer. Useful with The -M, -m modes together with the -D mode to get an output of what mirror would do if it were actually run. do the work specified in both the -U and -m options. -P Set the canonical permissions running in any of the modes M, m, S, U, other wise they are not set. The modes that are set are defined in the Params file. Inactive in -A mode. Page 4 (printed 11/11/97) MIRROR(LOCAL) XENIX System V (Lambda) MIRROR(LOCAL) SEE ALSO uucp(C), cron(C), rm(C), uuname(C), uuinstall(AD) gtfield(LOCAL), gtln(LOCAL) FILES /usr/lib/mirror/mirror the shell script /usr/lib/mirror/Params parameters for the mirror script. /usr/lib/mirror/systems/slaves list of slave systems /usr/lib/mirror/systems/masters list of master systems /usr/lib/mirror/slaves/*/files list of files mirrored to remote systems /usr/lib/mirror/slaves/*/dirs list of directories mirrored to remote systems /usr/lib/mirror/masters/*/files list of files mirrored from remote systems /usr/lib/mirror/masters/*/dirs list of directories mirrored from remote systems /usr/lib/mirror/slaves/*/sums transferred remote checksums from remote slaves /usr/lib/mirror/slaves/*/lsums local checksums of files that we mirror to remote slaves /usr/lib/mirror/masters/*/sums checksum files available for remote masters /usr/lib/mirror/work for temporary files NOTES The variables LOCALSYSTEM, SUMPROG, RMFPROG, UUDEMON and UUINSTALL of the 'mirror' shell script are system and unix flavor dependent and may have to be set by the System Administrator. Notes on mirrored network configuration: Reference: "Data Networks" - Gallagher & Bertsekas A given node should not have too many connections. This will swamp the node causing network failure. The maximum Page 5 (printed 11/11/97) MIRROR(LOCAL) XENIX System V (Lambda) MIRROR(LOCAL) number of connections should be determined by the power or capacity of the node which should depend on: machine speed available RAM available disk storage local activity An additional consideration is the mirror demands placed on the node. What is the expected throughput for each connection? This will be dependent on the centrality (if it can be defined) of the node in the network. A peripheral network node will presumably have fewer demands placed on it. For a star network, the central machine should have the maximal possible power, since it is exactly central and singularly so. Efficiency and manageability of a network is inversely proportional to the dimension of its topologizing graph. Page 6 (printed 11/11/97) File: /usr/lib/mirror/Notes ----------------------------------------------------------------------------- For Improvements and extensions or generalizations. ----------------------------------------------------------------------------- It is currently possible that in directory mirroring, a file may exist on the remote system that we do not have. This is treated as an error. The master system should be able to ignore such a file. For adding a system allow within the menu, modifications and intallation of the system to the uucp network files. - DONE Enable runs for a particular remote system: -R . - DONE Local and remote files to have different pathnames. - DONE SLAVE SYSTEM: Manufactures a 'sums' file for its master. (Uses its own path name) MASTER SYSTEM: Sends for slave's 'sums' file. Manufactures its own 'lsums' file for its slave. (Uses its own path name) Compares checksums of mirrored files by taking its own files one by one (line by line in 'lsums'). -------- To match pathnames, i.e., to find the correct checksum in the slave's sums file it needs to use the 'files' and 'dirs' files associated to the slave: Search 'files' first to find the line with its own pathname. If found, it will be the first field and the slave's pathname will be the second field. Quit search. Next, create two strings which are: (1. the local directory pathname of the file) and (2. the local filename) listed in the 'lsums file'. Search the 'dirs' file for the matching line. It should be found. Second field on that line is the pathname for the slave system. Get it. concat: (slave pathname) and (local file name) --------- Use the resulting full pathname to grep the slave's 'sums' file. Master may be adding a newfile to slave'sa directory and so if it is not found we queue for transmission. Otherwise do the checksum comparison. With no match, we queue for transmission. Replace queue for transmission by a direct uucp invocation: uucp $localpthname $system!$slavepath/$filename Add number of files transferred to each remote system to a log file. Send Timing to a log file. Operation warnings should still be captured by administrator's mail. : # # mirror # # UUCP network file and database mirroring system # # Runs in MASTER MODE - Update remote systems (local checksum update) # in master MODE - Update remote systems (without local checksum update) # in UPDATE MODE - (Only local checksum update) # in SLAVE MODE - Supplier to Master Systems that request # checksum files. # # and in interactive ADMINISTRATIVE MODE - Edit system files # # Run by cron in appropriate mode (other than administrative) at appropriate # times, on all mirror systems. # # Version 1.2 # # Bill Hammel - September 18, 1996 # October 7, 1996 - Added mirror administrative functions # + uucp administration. # October 8, 1996 - Option parsing & -R option, # & -D option. # October 9, 1996 - Split off of Params file. # - Files and directories to have different # paths on remote systems. # - uuq & uureq -> uucp directly. # October 10, 1996 - [-P] option # October 14, 1996 - [-N] Don't actually queue # #--------------------------------------------------------------------------# # trap "rm -f $SUMFILE $TMP/$$; exit 1" USAGE="Usage:\t$0 -[M | m | S | U] [-P] [-R ] [-D]\n\t$0 -N -[M | m] -[R ] [-D]\n\t$0 -A [-D]" # # Root of mirror system directories and files # exported to 'Params' when it is read. # See Params file for setting of parameters, constants, initializing, # pathnames, permission, owner and group defaults, etc. # MIRROOT="/usr/lib/mirror" #--------------------- ADMINISTRATION FUNCTIONS ---------------------------# # Editing of the system files: # systems/slaves, systems/masters # slaves/*/files, masters/*/files # slaves/*/dirs, masters/*/dirs # topology # admin_menu0() { clear echo "\n\n\n" echo "\t\t\tMIRROR ADMINISTRATION\n\n\n" echo "\t\t\ta. ADD A SYSTEM\n" echo "\t\t\tr. REMOVE A SYSTEM FROM MIRROR\n" echo "\t\t\tu. UPDATE A SYSTEM'S FILES\n" echo "\t\t\tt. SYSTEMS TOPOLOGY FILE\n" echo "\t\t\tc. CONVERT LOCAL FILE PATH TO REMOTE FILE PATH - check\n" echo "\t\t\tU. ADD and DELETE FROM, UPDATE UUCP FILES\n" echo "\t\t\te. EXIT\n\n" echo "\t\t\tENTER> \c" } admin_menu1() { clear echo "\n\n\n\n\n" echo "\t\t\tUPDATE MIRROR SYSTEM FILES\n\n\n" case $mode in s|S) echo "\t\t\tf. FILES MIRRORED TO SYSTEM: $system\n" echo "\t\t\td. DIRECTORIES MIRRORED TO SYSTEM: $system\n" ;; m|M) echo "\t\t\tf. FILES MIRRORED FROM SYSTEM: $system\n" echo "\t\t\td. DIRECTORIES MIRRORED FROM SYSTEM: $system\n" ;; esac echo "\t\t\te. EXIT\n\n" echo "\t\t\tENTER> \c" } fd_files() { # Enter here with the variables $system and $mode set case $mode in s|S) UPDIR=$S_RSYSD/$system ;; m|M) UPDIR=$M_RSYSD/$system ;; esac while true do admin_menu1 read ans junk case $ans in f) vi $UPDIR/files ;; d) vi $UPDIR/dirs ;; e) return ;; *) echo "\n\t\tUnrecognized response: $ans" sleep 5 continue ;; esac done } rm_sys() { clear echo "\n\n\n\n\n\n" echo "\t\tENTER NAME OF SYSTEM TO REMOVE> \c" read oldsys junk grep $oldsys $MASTER_LIST $SLAVE_LIST exstat=$? if [ $exstat -ne 0 ]; then echo "\n\t\tSystem: $oldsys is unknown" sleep 5 return fi # # edit to remove from slaves and masters lists use grep -v # rm -rf the oldsys subdirectories of the slave and master # directories. # cat $MASTER_LIST | grep -v $oldsys > $TMPD/$$ mv $TMPD/$$ $MASTER_LIST chmod 644 uucp root $MASTER_LIST cat $SLAVE_LIST | grep -v $oldsys > $TMPD/$$ mv $TMPD/$$ $SLAVE_LIST chmod 644 uucp root $SLAVE_LIST if [ ! -d $M_RSYSD/$oldsys ]; then rm -rf $M_RSYSD/$oldsys fi if [ ! -d $S_RSYSD/$oldsys ]; then rm -rf $S_RSYSD/$oldsys fi # NB We should display and check with user to remove. echo "\t\tSYSTEM $oldsys REMOVED FROM MIRROR" sleep 5 } add_sys() { clear echo "\n\n\n\n" echo "\t\tENTER NEW SYSTEM\'S NAME> \c" read newsys junk grep $newsys $MASTER_LIST $SLAVE_LIST exstat=$? if [ $exstat -eq 0 ]; then echo "\n\t\tSystem: $newsys is already known" sleep 5 return fi echo "\t\tSYSTEMS CAN BE BOTH MASTER AND SLAVE\n" echo "\n\t\tIS NEW SYSTEM A SLAVE (y/n)> \c" read s_ans junk echo "\n\t\tIS NEW SYSTEM A MASTER (y/n)> \c" read m_ans junk if [ "$s_ans" = "y" -o "$s_ans" = "Y" ]; then grep $newsys $SLAVE_LIST exstat=$? if [ $exstat -eq 0 ]; then echo "\n\t\tSystem: $newsys is already slave system" sleep 5 else echo "\n\t\tINSTALLING $newsys AS A SLAVE SYSTEM\n" echo $newsys >> $SLAVE_LIST if [ ! -d $S_RSYSD/$newsys ]; then mkdir $S_RSYSD/$newsys chmod 664 uucp root $S_RSYSD/$newsys fi fi fi if [ "$m_ans" = "y" -o "$m_ans" = "Y" ]; then grep $newsys $MASTER_LIST exstat=$? if [ $exstat -eq 0 ]; then echo "\t\tSystem: $newsys is already slave system" sleep 5 else echo "\n\t\tINSTALLING $newsys AS A MASTER SYSTEM\n" echo $newsys >> $MASTER_LIST if [ ! -d $M_RSYSD/$newsys ]; then mkdir $M_RSYSD/$newsys chmod 664 uucp root $M_RSYSD/$newsys fi fi fi } update_sys() { clear echo "\n\n\n" echo "\t\tENTER SYSTEM NAME AND MODE (name s|m)> \c" read system mode echo "\n\t\tChecking System: $system Current Mode\n" grep $system $MASTER_LIST $SLAVE_LIST exstat=$? if [ $exstat != 0 ]; then echo "\t\tUnknown system: $system" sleep 5 return else sleep 2 fi case $mode in s|S) grep $system $SLAVE_LIST exstat=$? if [ $exstat != 0 ]; then while true do echo "\t\tNot slave system: $system" echo "\t\tMake it one? (y/n) ENTER> \c" read ans junk case $ans in y|Y) echo $system >> $SLAVE_LIST if [ ! -d $S_RSYSD/$system ]; then mkdir $S_RSYSD/$system chmod 664 uucp root $S_RSYSD/$system fi echo "\n\n\t\t\t\tOK - DONE" sleep 5 ;; n|N) echo "\n\n\t\t\t\tOK" sleep 5 return ;; *) echo "\n\t\tUnrecognized response: $ans" sleep 5 continue ;; esac done else sleep 2 fi ;; m|M) grep $system $MASTER_LIST exstat=$? if [ $exstat != 0 ]; then while true do echo "\n\t\tNot master system: $system" echo "\n\t\tMake it one? (y/n) ENTER> \c" read ans junk case $ans in y|Y) echo $system >> $MASTER_LIST if [ ! -d $M_RSYSD/$system ]; then mkdir $M_RSYSD/$system chmod 664 uucp root $M_RSYSD/$system fi echo "\n\t\t\t\tOK - DONE" sleep 5 ;; n|N) echo "\n\t\t\t\tOK" sleep 5 return ;; *) echo "\n\t\tUnrecognized response: $ans" continue ;; esac done else sleep 2 fi ;; *) echo "\n\t\tUnrecognized mode: $mode" sleep 5 return ;; esac # Updating a system with $system name and with slave or master $mode fd_files } # Administrative mode entry point # admin_mirror() { while true do admin_menu0 read ans junk case $ans in a) add_sys continue ;; r) rm_sys continue ;; u) update_sys continue ;; t) vi $TOPOLOGY continue ;; c) echo "LOCAL FILE PATH> \c" read l_path echo "REMOTE SYSTEM> \c" read system path_conv echo "REMOTE FILE: $system!$r_path" echo "RETURN> \c" read system continue ;; U) clear $UUINSTALL continue ;; E|e|Q|q) clear exit 0 ;; *) echo "\n\t\tUnrecognized Response $ans" sleep 5 continue ;; esac done # Switch below does exiting after dropout - if it happens } #---------------- END ADMINISTRATION FUNCTIONS ---------------------------# # Sets and maintains permissions # Files Fchmog() { if [ $DOPERMS -eq 0 ]; then return fi if [ $MODE -eq $SLAVE ]; then chmog $S_F_MODE $S_F_OWNER $S_F_GROUP $l_path else chmog $M_F_MODE $M_F_OWNER $M_F_GROUP $l_path fi } # Sets and maintains permissions # Directories and their file and directory contents Dchmog() { if [ $DOPERMS -eq 0 ]; then return fi if [ $MODE -eq $SLAVE ]; then chmog $S_D_MODE $S_D_OWNER $S_D_GROUP $l_dir find $l_dir -type d -exec chmog $S_D_MODE $S_D_OWNER $S_D_GROUP {} \; find $l_dir -type f -exec chmog $S_F_MODE $S_F_OWNER $S_F_GROUP {} \; # NB This is really incomplete below one level else chmog $M_D_MODE $M_D_OWNER $M_D_GROUP $l_dir find $l_dir -type d -exec chmog $M_D_MODE $M_D_OWNER $M_D_GROUP {} \; find $l_dir -type f -exec chmog $M_F_MODE $M_F_OWNER $M_F_GROUP {} \; fi } #------------ FUNCTIONS THAT CHECK INTEGRITY OF CHECK FILES ---------------# # SUMFILE is externally set in the main process. # Function sets external flag depending on integrity check outcome. chk_sumfile() { # Extract checksum line for this 'sums' file rchksum=`gtln 1 $SUMFILE` if [ $DEBUG -eq 1 ]; then echo "$SUMFILE:\c" wc $SUMFILE echo "Editing" fi # Edit out the checksum line ed $SUMFILE <<- EOF > /dev/null 1d w q EOF if [ $DEBUG -eq 1 ]; then echo "Edited" echo "$SUMFILE:\c" wc $SUMFILE fi # Calculate the actual checksum output of the received file cchksum=`$SUMPROG $SUMFILE` # Get the actual checksum numbers from each line rsum=`gtfield -c 1 $rchksum` csum=`gtfield -c 1 $cchksum` if [ $DEBUG -eq 1 ]; then echo "\nReceived checksum: $rsum" echo "Computed checksum: $csum" fi # Compare checksums for file sent and file received and set # the GOODFILE flag accordingly if [ $rsum -eq $csum ]; then GOODFILE=1 else GOODFILE=0 fi if [ $DEBUG -eq 1 ]; then echo "chk_sumfile returning" fi } # Modify the simple list of checksums to include its own checksum mk_sumfile() { # Get checksum output for 'sums' file that is to be sent to a # master system cchksum=`$SUMPROG $SUMFILE` # Edit to add this line as the first line in the file ed $SUMFILE <<- EOF > /dev/null 0a $cchksum . w q EOF if [ $DEBUG -eq 1 ]; then echo "mk_sumfile returning" fi } #---------------------- FUNCTIONS THAT CREATE CHECKSUMS ----------------------# # Must be called before premakedsums # Called by S_mk_sums and M_mk_lsums. # premakefsums() { if [ $DEBUG -eq 1 ]; then echo "premakefsums begin" fi # Set loop parameters # Find number of lines (= number of files) in 'files' file. flns=`cat $RF_FILE | wc -l` flno=1 # Loop creating checksum lines in 'sums' file for the current file # master system for each file indicated in 'files' file. while [ $flno -le $flns ] do line=`gtln $flno $RF_FILE` l_path=`gtfield -c 1 $line` if [ "$l_path" = "#" ]; then # A comment line flno=`plus -i $flno 1` continue fi # Set proper m o g for current file # Function performs setting on $l_file Fchmog if [ $MODE -eq $SLAVE ]; then if [ -f $l_path ]; then echo "`$SUMPROG $l_path` $l_path" >> $R_SUMFILE else # File is not there; put in a parseable entry echo "0 0 $l_path" >> $R_SUMFILE fi else if [ -f $l_path ]; then echo "`$SUMPROG $l_path` $l_path" >> $R_SUMFILE else # File is not there; put in a parseable entry echo "WARNING: file $l_path does not exist for us to mirror to $system" flno=`plus -i $flno 1` continue fi fi flno=`plus -i $flno 1` done if [ $DEBUG -eq 1 ]; then echo "premakefsums returning" fi } # Must be called after premakefsums # Called by S_mk_sums and M_mk_lsums. # premakedsums() { if [ $DEBUG -eq 1 ]; then echo "premakedsums begin" fi # Set loop paprameters dlns=`cat $RD_FILE | wc -l` dlno=1 # Loop creating checksum lines in 'sums' file for the current system # for each file of each directory indicated. # Outer loop is loop on directories in the 'dirs' file for the # current remote master system. while [ $dlno -le $dlns ] do line=`gtln $dlno $RD_FILE` l_dir=`gtfield -c 1 $line` if [ "$l_dir" = "#" ]; then # A comment line dlno=`plus -i $dlno 1` continue fi if [ $DEBUG -eq 1 ]; then echo "Creating list of files in: $l_dir" fi # Set proper m o g for current directories # and files. # Function performs setting on $l_dir Dchmog # List all files of the current directory in a workfile. if [ $MODE -eq $SLAVE ]; then if [ -d $l_dir ]; then find $l_dir -type f -print > $TMPD/$$F else # Can't make entries; master system will # understand to send anyhow. # Increment loop counter dlno=`plus -i $dlno 1` continue fi else # A master mode if [ -d $l_dir ]; then find $l_dir -type f -print > $TMPD/$$F else # Can't make entries; master system will # understand to send anyhow. # Increment loop counter echo "WARNING: directory $l_dir does not exist to mirror to $system" dlno=`plus -i $dlno 1` continue fi fi if [ $DEBUG -eq 1 ]; then echo "Converting workfile list to Shell script" fi # Edit created list of files: transforming it into # an sh script that will generate the checksum lines # for all files listed. ed $TMPD/$$F <<- END > /dev/null 1,\$s/.*/echo \"\`$SUMPROG &\` &\"/ w q END if [ $DEBUG -eq 1 ]; then echo "Shell Script adding sums to lsums file" fi # Run created sh script appending output to the 'sums' file # being created. sh $TMPD/$$F >> $R_SUMFILE # Increment loop counter dlno=`plus -i $dlno 1` done rm -f $TMPD/$$F if [ $DEBUG -eq 1 ]; then echo "Clean up work directory" echo "premakedsums returning" fi } M_mk_lsums() { if [ $DEBUG -eq 1 ]; then echo "M_mk_lsums begin" fi # Reset the R_SYSS file to Remote System list of our slaves R_SYSS="$MIRROOT/systems/slaves" # Loop on the systems that our slaves. slns=`cat $R_SYSS | wc -l` slno=1 while [ $slno -le $slns ] do rsys=`gtln $slno $R_SYSS` if [ "$ONLYSYSTEM" != "" ]; then # The -R option has been used if [ "$ONLYSYSTEM" != "$rsys" ]; then slno=`plus -i $slno 1` continue fi fi R_SUMFILE="$S_RSYSD/$rsys/lsums" creat $R_SUMFILE # For the slave system's files RF_FILE="$S_RSYSD/$rsys/files" premakefsums # For the slave system's directories RD_FILE="$S_RSYSD/$rsys/dirs" premakedsums # We do not have to put a file checksum in since these # files aren't going anywhere. chmog $M_F_MODE $M_F_OWNER $M_F_GROUP $R_SUMFILE slno=`plus -i $slno 1` done if [ $DEBUG -eq 1 ]; then echo "M_mk_lsums returning" fi } S_mk_sums() { if [ $DEBUG -eq 1 ]; then echo "S_mk_sums begin" fi # Init loop parameters slns=`cat $R_SYSS | wc -l` slno=1 if [ $DEBUG -eq 1 ]; then echo "Entering master systems loop" fi # Loop on the systems that our masters. while [ $slno -le $slns ] do rsys=`gtln $slno $R_SYSS` if [ "$ONLYSYSTEM" != "" ]; then # The -R option has been used if [ "$ONLYSYSTEM" != "$rsys" ]; then slno=`plus -i $slno 1` continue fi fi R_SUMFILE="$M_RSYSD/$rsys/sums" if [ $DEBUG -eq 1 ]; then echo "Recreating: $R_SUMFILE" fi # This zeros out any old sum file contents creat $R_SUMFILE # For the master system's files RF_FILE="$M_RSYSD/$rsys/files" if [ $DEBUG -eq 1 ]; then echo "Checksum for entries in: $RF_FILE" fi premakefsums # For the master system's directories RD_FILE="$M_RSYSD/$rsys/dirs" if [ $DEBUG -eq 1 ]; then echo "Checksum for entries in: $RD_FILE" fi premakedsums # Put in the checksum of checksums. # (function mk_sumfile uses value of SUMFILE) SUMFILE=$R_SUMFILE mk_sumfile if [ $DEBUG -eq 1 ]; then echo "Checksum added to: $SUMFILE" echo "Adjusting Transmission Permissions" fi # Set proper permissions for transmission chmog $S_F_MODE $S_F_OWNER $S_F_GROUP $R_SUMFILE # Increment counter of master systems slno=`plus -i $slno 1` if [ $DEBUG -eq 1 ]; then echo "Next Master System: prepare sums file for request" fi done if [ $DEBUG -eq 1 ]; then echo "S_mk_sums returning" fi } # Convert the full pathaname of the local file to the # full pathname of the remote file that we are mirroring. # Given: l_path = local pathname of file to mirror out. # Set the variables: r_file = l_file = local file name # l_dir = local directory of file in question. # r_path = remote full pathname of file # r_dir = remote directory of file in question. # # Called only in -m and -M noninteractive mode # and in -A mode as a check # path_conv() { RSFILES="$S_RSYSD/$system/files" RSDIRS="$S_RSYSD/$system/dirs" l_dir=`dirname $l_path` l_file=`basename $l_path` if [ $DEBUG -eq 1 ]; then echo "path_conv begin" echo "l_path = $l_path" echo "l_file = $l_file" echo "l_dir = $l_dir" echo "\tConversion files:" echo "$RSFILES : $RSDIRS" fi # Assume first the pathname is a file t_r_line=`grep "$l_path " $RSFILES` grstat=$? if [ $DEBUG -eq 1 ]; then echo "path_conv: (files) t_r_line = $t_r_line" fi if [ $grstat -eq 0 ]; then # It is a file and we are finished r_path=`gtfield -c 2 $t_r_line` r_dir=`dirname $r_path` r_file=`basename $r_path` if [ $DEBUG -eq 1 ]; then echo "path_conv - files returning with conversion" fi return fi if [ $DEBUG -eq 1 ]; then echo "path_conv: Remote file not converted as file," echo " Searching dirs file." fi # ----------------------------------------------------- # while [ "$l_dir" != "." ] do # If here the pathanme is not a file and we look for # the pathname that is that of a mirrored directory. if [ $DEBUG -eq 1 ]; then echo "path_conv: dirs l_dir = $l_dir" fi t_r_line=`grep "$l_dir " $RSDIRS` grstat=$? if [ $grstat -eq 0 ]; then # We have the line with the remote pathname r_dir=`gtfield -c 2 $t_r_line` r_file=$l_file r_path=$r_dir/$r_file if [ $DEBUG -eq 1 ]; then echo "r_path = $r_path" echo "r_file = $r_file" echo "r_dir = $r_dir" echo "path_conv - dirs returning" fi return else # This event seems like a logical inconsistency; it is # the result 'find' working recursively and find files # in a subdirectory of the directory named in the 'dirs' # file. Transfer the subdirectory name to the file # name and search again for the directory. t_dir=$l_dir t_file=$l_file l_dir=`dirname $t_dir` subd_name=`basename $t_dir` r_file=$subd_name/$t_file l_file=$subd_name/$t_file if [ $DEBUG -eq 1 ]; then echo "SUBDIRECTORY" echo "l_dir = $l_dir" echo "l_file = $l_file" fi continue fi done } #---------------------- REREQUEST ABBERANT sums FILE ---------------------# rereqsums() { # Request immediate transfer of prepared checksum files # from our remote master directory to the local # slave's directory of the remote system. if [ $DEBUG -eq 1 ]; then echo "uucp -d -c -r $system!$M_RSYSD/$LOCALSYSTEM/sums $S_RSYSD/$system/sums" sleep 2 fi uucp -d -c -r $system!$M_RSYSD/$LOCALSYSTEM/sums $S_RSYSD/$system/sums # Increment counter of reqrequests for this file REREQS=`plus -i $REREQS 1` # Wait for new copy of remote sums file until [ -f $SUMFILE ] do sleep $CHECKTIME done } #--------------------- END FUNCTION DEFINITIONS --------------------------# #---------------- PARAMETERS, SWITCHES, VARIABLE INITITIALIZATIONS -------# #---------------- STRINGS, PATHNAMES, PROGRAMS, FILENAMES ----------------# # Params sets all other paramters . $MIRROOT/Params #------------------------ NONINTERACTIVE MODE OPTIONS ------------------------# # The MODE token values - Should never be any reason to change. NOTSET=-1 SLAVE=0 MASTER=1 UPDATE=2 master=3 ADMIN=4 MODE=$NOTSET # Same format as that for getopt in C OPTSTRING="R:ADMmNSU" # #------------ INTERACTIVE MODE or NONINTERACTIVE MODE OPTION SETUPS ----------# # First and only command line argument is the option determining MODE. # Don't need a default case since syntax check has already eliminated # all options that are not syntactically acceptable. #-------------------------- Option Syntax Check ---------------------------# if [ $# -eq 0 ]; then echo $USAGE exit 0 fi if [ $# -gt 4 ]; then echo "Too many options/paramters" echo $USAGE exit 1 fi set -- `getopt $OPTSTRING $*` if [ $? != 0 ]; then echo Options: $* echo Optstring: $OPTSTRING echo $USAGE exit 1 fi # for i in $* do case $i in -N) QUEUE=0 shift ;; -R) # Option takes systen name as argument # Cannot be used in Aministrative Mode. ONLYSYSTEM=$2 shift; shift ;; -P) # Option takes no argument DOPERMS=1 shift ;; -A) # Option takes no argument if [ $MODE -eq $NOTSET ]; then MODE=$ADMIN fi shift ;; -D) # Turn on the debugging flag DEBUG=1 shift ;; -M) if [ $MODE -eq $NOTSET ]; then # Set Mode switch MODE=$MASTER # Remote System list of our slaves R_SYSS="$MIRROOT/systems/slaves" fi shift ;; -m) if [ $MODE -eq $NOTSET ]; then # Set Mode switch MODE=$master # Remote System list of our slaves R_SYSS="$MIRROOT/systems/slaves" fi shift ;; -S) if [ $MODE -eq $NOTSET ]; then # Set Mode switch MODE=$SLAVE # Remote System list of our masters R_SYSS="$MIRROOT/systems/masters" fi shift ;; -U) if [ $MODE -eq $NOTSET ]; then # Set Mode switch MODE=$UPDATE # Remote System list of our masters R_SYSS="$MIRROOT/systems/slaves" fi shift ;; --) shift break ;; *) echo "Unknown Option: $1" echo $USAGE exit 1 ;; esac done if [ $MODE -eq $NOTSET ]; then echo "Improper Syntax" echo "One of options: -[AMmUS] must be set" echo $USAGE exit 1 fi if [ $DEBUG -eq 1 ]; then echo "End function definitions, constants, parameters and options" echo "Mode: $MODE" echo "Local system: $LOCALSYSTEM" echo "Remote System directories: $R_SYSS" sleep 2 fi if [ $MODE -eq $ADMIN ]; then # Only do administrative stuff. admin_mirror exit 0 fi #-------------------------- END SETUP SECTIONS ---------------------------# # This message will go to administrator's mail echo "$0 $1: Begin \c" date #------------------- BEGIN MAIN NONINTERACTIVE PROCESS -------------------# #-------------------------- Slave Mode Process ---------------------------# # Slave mode simply prepares the checksum files for retrieval by all # the master systems. if [ $MODE -eq $SLAVE ]; then S_mk_sums # This message will go to administrator's mail echo "$0 $1: local update run Complete: \c" date fi #-------------------------- Master Mode Process ---------------------------# # Examine by SUMPROG, the remote mirrored images of all the files # in 'files', and all the directories of 'dirs' # to see if they are conformed to the the cognate files on the # local system. # If the are not, update them by queueing for uucp transfer to the # specified systems. if [ $MODE -eq $MASTER -o $MODE -eq $master ]; then # # Remote system has presumably constructed its checksum files by running # in Slave Mode. # Request them from all remote slave systems that we mirror to. # mlns=`cat $R_SYSS | wc -l` mlno=1 while [ $mlno -le $mlns ] do system=`gtln $mlno $R_SYSS` if [ "$ONLYSYSTEM" != "" ]; then # The -R option has been used if [ "$ONLYSYSTEM" != "$system" ]; then mlno=`plus -i $mlno 1` continue fi fi # Remove the old sums file that may be present $RMFPROG $S_RSYSD/$system/sums # Request immediate transfer of prepared checksum files # from our remote master directory to the local # slave's directory on our local system. if [ $DEBUG -eq 1 ]; then echo "uucp -d -c -r $system!$M_RSYSD/$LOCALSYSTEM/sums $S_RSYSD/$system/sums" sleep 2 fi uucp -d -c -r $system!$M_RSYSD/$LOCALSYSTEM/sums $S_RSYSD/$system/sums # Increment counter of remote slave systems mlno=`plus -i $mlno 1` done # Start demon to send out queued requests for all sums files. $UUDEMON fi if [ $MODE -eq $MASTER -o $MODE -eq $UPDATE ]; then # # While we are waiting for the remote checksum files to arrive, # recreate the sums files for the local system files 'lsums' that # are to be distributed to all remote slave systems. We will need # them for the checksum comparisons. We initiate remote file # update if the remote checksum is not equal to our local checksum. # M_mk_lsums # This message will go to administrator's mail echo "$0 $1: Local Update Complete \c" date fi if [ $MODE -eq $MASTER -o $MODE -eq $master ]; then # Reset the R_SYSS file R_SYSS="$MIRROOT/systems/slaves" # Remote System list of our slaves # MASTER MODE # We must now await the arrival of all the 'sums' files from the remote # systems to which we mirror. # NB Do we really have to wait for ALL the checksum files to arrive? # I don't think so. Perhaps change the below! # # The number of remote slave systems slns=`cat $R_SYSS | wc -l` # The number of remote files that have arrived rfiles=0 # Unused arrival_list except in debugging arrival_list="" # The "waiting loop" for the arrival of the correct number of # sums files from the slave systems. while [ $rfiles -lt $slns ] do # Check periodically to see if all files are here # The "check on all arrivals" loop rfiles=0 slno=1 while [ $slno -le $slns ] do system=`gtln $slno $R_SYSS` if [ "$ONLYSYSTEM" != "" ]; then # The -R option has been used if [ "$ONLYSYSTEM" != "$system" ]; then # Don't actually check slno=`plus -i $slno 1` rfiles=`plus -i $rfiles 1` continue fi fi if [ -f $S_RSYSD/$system/sums ]; then arrival_list="$arrival_list $system" rfiles=`plus -i $rfiles 1` fi if [ $DEBUG -eq 1 ]; then echo "Arrivals: $arrival_list" echo "# of Systems: $slns" echo "# of Arrivals: $rfiles" echo "Checked system #: $slno" fi slno=`plus -i $slno 1` if [ $slno -ge $slns ]; then break fi done if [ $DEBUG -eq 1 ]; then echo "Out of system loop on sums files" fi if [ $rfiles -eq $slns ]; then if [ $DEBUG -eq 1 ]; then echo "All system sums files here." echo "Break waiting loop." fi break fi if [ $DEBUG -eq 1 ]; then echo "Waiting $CHECKTIME seconds." date fi sleep $CHECKTIME done # # All remote files are here and we begin the loop on each system # with the checksum comparisons of the sums and lsums files # themselves. # # Loop on all slave systems rsyss=`cat $R_SYSS | wc -l` rsysno=1 if [ $DEBUG -eq 1 ]; then echo "Entering slave system processing loop." fi while [ $rsysno -le $rsyss ] do system=`gtln $rsysno $R_SYSS` if [ "$ONLYSYSTEM" != "" ]; then # The -R option has been used if [ "$ONLYSYSTEM" != "$system" ]; then # Don't actually check rsysno=`plus -i $rsysno 1` continue fi fi SUMFILE="$S_RSYSD/$system/sums" chk_sumfile chmog $M_F_MODE $M_F_OWNER $M_F_GROUP $SUMFILE if [ $DEBUG -eq 1 ]; then echo "Checksum done of $SUMFILE" fi if [ $GOODFILE -eq 0 ]; then echo "Bad checksum ($0): $SUMFILE" echo "Saving remote file as: $SUMFILE.bad" echo "Rerequesting $SUMFILE" mv $SUMFILE $SUMFILE.bad if [ $REREQS -gt $MAXREREQS ]; then echo "The Number of Requests for our $SUMFILE" echo "due to bad xmission now exceeds" echo "($MAXREREQS), this is the set maximum." echo "$0 of system: $system will not be done." echo "$0 proceding to next system." echo "Admin should check transmission setup." # Set for next system rsysno=`plus -i $rsysno 1` # Reset the REREQS counter for the next system REREQS=0 # Increment number of systems skipped # Not Used. N_SYSSKIPPED=`plus -i $N_SYSSKIPPED 1` SYSSKIPPED="$SYSSKIPPED $system" fi # Implied else: # Do NOT go to the next system. # The requesting process will continue until we # get a copy that checks out, or MAXREREQS is # exceeded. # The function rereqsums has done the waiting # for the file to arrive. # Rerequest the sums file that is bad. # This function will use the currently set value # of the 'system' variable. rereqsums continue fi if [ $DEBUG -eq 1 ]; then echo "Good Checksum on $SUMFILE" fi # # If here: # Good checksum on received sum file and its own checksum # line has been removed. # We compare the sums and lsums files, assuming that the # we being the master of this process, know what files to # transfer. # LSUMFILE="$S_RSYSD/$system/lsums" rlns=`cat $SUMFILE | wc -l` llns=`cat $LSUMFILE | wc -l` if [ $rlns -ne $llns ]; then echo "Warning ($0):" echo "\twc -l $SUMFILE: $rlns" echo "\twc -l $LSUMFILE: $llns" fi # # Start reading our lsums file line by line, and searching # the sums file of the current remote system for the # pathename on our line. We already have 'llns' value. # if [ $DEBUG -eq 1 ]; then echo "System: $system -" echo " Entering main sums-lsums comparison loop" fi llno=1 while [ $llno -le $llns ] do # Sums File Line format: (checksum) (blocks) (pathname) l_line=`gtln $llno $LSUMFILE` l_chksum=`gtfield -c 1 $l_line` # Get full pathname of the file we mirror from l_path=`gtfield -c 3 $l_line` # # NB For a remote file to have a different pathname # we use our 'files' & 'dirs' to convert to the # correct remote pathname 'r_line' before grepping # the 'sums' arrived from the remote system. # The remote system will use its own pathname. # # Convert: l_path -> r_path, # Set variables: r_path, r_file, r_dir path_conv # Look for matching line with converted full path name # in sums file from remote system. r_line=`grep "$r_path\$" $SUMFILE` grepstatus=$? if [ $DEBUG -eq 1 ]; then echo "greped r_line: $r_line" fi if [ $grepstatus -ne 0 -o "$r_line" = "" ]; then # grep has failed to find a match in sums. # Assume this is starting a new file on the # remote system. if [ $DEBUG -eq 1 ]; then echo "$r_path not present on $system" fi r_chksum=0 else # Otherwise we have a matching line; get its # checksum field. r_chksum=`gtfield -c 1 $r_line` fi if [ $DEBUG -eq 1 ]; then echo "r_chksum: $r_chksum l_chksum: $l_chksum" fi if [ $r_chksum -ne $l_chksum ]; then # Update of the remote system file is necessary. # Queue the update do not let system do the # scheduling since this process is part of # the system scheduling # The list of the slave systems is the # order of priority. if [ $DEBUG -eq 1 ]; then echo "\t\t\tSENDING -" echo "uucp -d -c -r $l_path $system!$r_path" echo "Remote System: $system" echo "Local - File: $l_path\tDirectory: $l_dir" echo "Remote - File: $r_file\tDirectory: $r_dir" sleep 2 fi if [ $QUEUE -eq 1 ]; then uucp -d -c -r $l_path $system!$r_path else echo "\t\t\t\tNOT QUEUED" fi else if [ $DEBUG -eq 1 ]; then echo "\t\t\t\tCHECKSUMS MATCH\n" fi fi # Increment counter of lsums line llno=`plus -i $llno 1` done # End loop on system checksum comparisons rsysno=`plus -i $rsysno 1` # Reset the REREQS counter for the next system REREQS=0 rm -f $SUMFILE if [ $DEBUG -eq 1 ]; then echo "Remote $system 'sums' file removed - Next system" fi done # End loop on slave systems # Start Uucp Scheduler as background process. $UUDEMON & if [ $DEBUG -eq 1 ]; then echo "All slave system files queued for remote mirroring." echo "Uucp Scheduler started as background process." fi # This message will go to Administrator's mail echo "$0 $1: run Complete \c" date fi if [ $DEBUG -eq 1 ]; then echo "mirror - EXITING WITH SUCCESS." fi exit 0 : # /usr/lib/mirror/Params # # '.' read by mirror # # The value of MIRROOT is exported from mirror # #---------------- PARAMETERS, SWITCHES, VARIABLE INITITIALIZATIONS -------# #---------------- STRINGS, PATHNAMES, PROGRAMS, FILENAMES ----------------# # Switch for debugging output. # Normal setting should be 0; set to 1 for debug runs at the command line. # Option -D turns this switch on. DEBUG=0 # Set on: actually do the uucp queueing, off just show what you would do. # Option -N turns this switch off. QUEUE=1 # UNIX FLAVOR DEPENDENT - Change by unix masters only SUMPROG=sum UUDEMON=/usr/lib/uucp/uudemon.hour UUINSTALL="/etc/uuinstall 2" # UNIX FLAVOR DEPENDENT - Change by unix masters only # Check local manual - This should "remove forcefully" RMFPROG="rm -f" # UNIX FLAVOR DEPENDENT - Change by unix masters only # Check local manual - This should retrieve the local uucp system name # in /etc/systemid or a file of similar name and same use. LOCALSYSTEM=`uuname -l` # Time interval in seconds between checks for arrival of the sent for # remote system sums files. # Changeable with caution. CHECKTIME=150 # The number of requests allowed before a complaint is registered. # This number is adjustable. MAXREREQS=5 # Counter for the number of times a remote sums file is rerequested. # This is an initialization - DO NOT CHANGE. REREQS=0 # Counter for the number of systems skipped due to intolerable transmission, # as defined by MAXREREQS. # This is an initialization - DO NOT CHANGE. N_SYSSKIPPED=0 # List of the systems skipped due to intolerable transmission, # as defined by MAXREREQS. # This is an initialization - DO NOT CHANGE. SYSSKIPPED="" # For -R option in -M -S -U -m modes # This is an initialization - DO NOT CHANGE ONLYSYSTEM="" # Topology file of Mirror nodes TOPOLOGY=$MIRROOT/topology # System files of master and slave remote systems # Should never be changed. MASTER_LIST="$MIRROOT/systems/masters" SLAVE_LIST="$MIRROOT/systems/slaves" # Master system and Slave system Directories # Should never be changed. M_RSYSD="$MIRROOT/masters" S_RSYSD="$MIRROOT/slaves" # What it says it is. # Should never need to be changed. TMPD="$MIRROOT/work" # Without the -P option do not set the canonical permissions below DOPERMS=0 # Permissions, owner and group for files mirrored # Cf mirror: Fchmog(), Dchmog() # SLAVE Files S_F_MODE="+rw" S_F_OWNER=uucp S_F_GROUP=root # SLAVE Directories S_D_MODE=777 S_D_OWNER=uucp S_D_GROUP=root # MASTER Files M_F_MODE="+rw" M_F_OWNER=uucp M_F_GROUP=root # MASTER Directories M_F_MODE=777 M_F_OWNER=uucp M_F_GROUP=root # End Params

Return to Home Page
Return to Metayoga Page
Return to shell Page

The URL for this document is:
http://graham.main.nc.us/~bhammel/graham/SHELL/mirror.html
Created: 1997
Last Updated: May 28, 2000 Email me, Bill Hammel at
bhammel@graham.main.nc.us READ WARNING BEFORE SENDING E-MAIL