git and rsync folders tools round 2

This article is round 2 from a recent article I wrote at http://www.whiteboardcoder.com/2014/08/git-and-rsync-folders.html .

I am dealing with another folder that does not have a nice structure.  In this case I may have folders three levels deep and folders one level deep I want to rsync and not store in the .git repo.   Also, aside from the structure and shell scripts, I do not want to store anything in git,, except the folder structure up to the rsync script.   I like my round 2 solution better than my previous ones.  I think I will roll it out to my other folders where I have rsync needs.

Create the repository

> git init

Update .git/config  (not per say necessary)

> vi .git/config

Here is my config file

[core]         repositoryformatversion = 0         filemode = true         bare = false         logallrefupdates = true         ignorecase = true [core]         packedGitLimit = 128m         packedGitWindowSize = 128m [pack]         deltaCacheSize = 128m         packSizeLimit = 128m         windowMemory = 128m

Create .gitignore file

For this git repository I only want to see my scripts, and the folders they are in.

> vi .gitignore

Place the following in it.

#Ignore Everything * # But not scripts or folders !*.sh !*/

This first ignores everything.  Then it adds two exceptions. Folders and anything ending in *.sh (my scripts)

Create Default script

Create a default script that will be copied to each directory.

> vi .rsyncme-DEFAULT.sh

And place the following in it (replacing the highlighted area with your own information)

#!/bin/bash #Remote location url=git.example.com #Check for an override name name="" if [ $2 ] then   name="$2@" fi #======================================= # #loc is remote directory path loc="/not-git/rsync/my_main_folder/" folder=("XXXXX") # #=============================================== flags="-avzr" if [ "$1" == 'push' ] then   echo "Push it"   #Need to make the directories   #for folder   ssh "$name"$url mkdir -p $loc${folder}   rsync $flags * "$name"$url:$loc$folder else   echo "Pull it"   rsync $flags "$name"$url:$loc$folder/* . fi

Create Copy Script

After this is done create a script that will copy this script into each folder, rename it and replace the XXXXX with is parent folder name. 

This script will search for folders with a file named .rsyncme.sh

> vi makescripts.sh

And place the following in it.

#!/bin/bash for folder in `find $PWD -iname ".rsyncme.sh" -exec dirname {} \;`; do   cp .rsyncme-DEFAULT.sh $folder/.rsyncme.sh   pushd .     cd $folder     basename $folder | xargs -I '{}' sed -i 's/XXXXX/{}/g' .rsyncme.sh   popd done

This will find all folders that have a file named .rsyncme.sh  .  It will replace the file with the generic one.  It will then replace the XXXXX with the folder name.

Create Run All script

Create a run all script that will find all the .rsyncme.sh scripts and run them.  (to make one easy push)

> vi runallpush.sh

And place the following in it. (replace the highlighted with your username)

#!/bin/bash for folder in `find $PWD -iname ".rsyncme.sh" -exec dirname {} \;`; do   echo $folder   pushd .   cd $folder   bash ./.rsyncme.sh push patman   popd done

Rsync all the data

After you have set up the base directory on the server you run the runallpush.sh script to rsync everything.

> ./runallpush.sh

Commit the git

> git add . > git commit -m "initial commit"

Then on your git server init a bare repo that you can push your repository to.  In this example I am sudo'n to the git user (who does not have a normal shell, so you have to designate it)

> sudo su git -s /bin/bash > cd /git/repos > mkdir project.git > cd project.git > git --bare init

Finally push it up to a remote master server (adjust the url to your remote server and the path)

> git remote add origin git@example.com:/git/repos/project.git > git push origin master

Clone the repo and pull rsync

From another computer clone the repository

> git clone git@example.com:git/repos/project.git

After this head into one of the directories from the command line and pull your data (rsync) edit command for your username.

> ./.rsyncme.sh pull patman

I could have written on script to run all the scripts and pull (rsync) all the data down, but that is not my goal here.   My goal is to be able to download a folder's contents, remotely, in a pinch.  Otherwise I do not want to take up the hard drive space (Or in my specific case my thumb drive's space)

In conclusion

Now any new folder you want to rsync just run this command in the folder.  (create a .rsyncme.sh file)

> touch .rsyncme.sh

Then run the makescripts.sh command in the base directory to update the script

> ./makescript.sh

Then run the new .rsyncme.sh script to push pull the folder

> ./.rsyncme.sh push

> ./.rsyncme.sh pull

And of course make sure your .gitignore is ignoring the folders you intent to rsync and not the .rsyncme.sh scripts themselves.

Update  8/30/2014

I had an issue trying to set up .gitignore in  another folder.  Instead of fighting the nightmare that can be .gitignore I did the following.

Set up .gitignore to ignore the folders you want to ignore.  Then place .rsyncme.sh files in the folders you want to rsync.  Finally run this command line command to add all .rsyncme.sh to the git repo.

> find $PWD -name ".rsyncme.sh" | xargs -I "{}" git add -f {}

References