git and rsync folders tools round 2

Posted on Thursday, August 28, 2014

This article is round 2 from a recent article I wrote at .

I am dealing with another folder that does not have a nice structure.  In this case I may have folders three levels deep and folders one level deep I want to rsync and not store in the .git repo.   Also, aside from the structure and shell scripts, I do not want to store anything in git,, except the folder structure up to the rsync script.   I like my round 2 solution better than my previous ones.  I think I will roll it out to my other folders where I have rsync needs.

Create the repository

> git init

Update .git/config  (not per say necessary)

> vi .git/config

Here is my config file

        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
        ignorecase = true
        packedGitLimit = 128m
        packedGitWindowSize = 128m

        deltaCacheSize = 128m
        packSizeLimit = 128m
        windowMemory = 128m

Create .gitignore file

For this git repository I only want to see my scripts, and the folders they are in.

> vi .gitignore

Place the following in it.

#Ignore Everything

# But not scripts or folders

This first ignores everything.  Then it adds two exceptions. Folders and anything ending in *.sh (my scripts)

Create Default script

Create a default script that will be copied to each directory.

> vi

And place the following in it (replacing the highlighted area with your own information)


#Remote location

#Check for an override name
if [ $2 ]

#loc is remote directory path


if [ "$1" == 'push' ]
  echo "Push it"
  #Need to make the directories
  #for folder
  ssh "$name"$url mkdir -p $loc${folder}
  rsync $flags * "$name"$url:$loc$folder
  echo "Pull it"
  rsync $flags "$name"$url:$loc$folder/* .

Create Copy Script

After this is done create a script that will copy this script into each folder, rename it and replace the XXXXX with is parent folder name. 

This script will search for folders with a file named

> vi

And place the following in it.


for folder in `find $PWD -iname "" -exec dirname {} \;`; do
  cp $folder/
  pushd .
    cd $folder
    basename $folder | xargs -I '{}' sed -i 's/XXXXX/{}/g'

This will find all folders that have a file named  .  It will replace the file with the generic one.  It will then replace the XXXXX with the folder name.

Create Run All script

Create a run all script that will find all the scripts and run them.  (to make one easy push)

> vi

And place the following in it. (replace the highlighted with your username)


for folder in `find $PWD -iname "" -exec dirname {} \;`; do
  echo $folder
  pushd .
  cd $folder
  bash ./ push patman

Rsync all the data

After you have set up the base directory on the server you run the script to rsync everything.

> ./

Commit the git

> git add .
> git commit -m "initial commit"

Then on your git server init a bare repo that you can push your repository to.  In this example I am sudo'n to the git user (who does not have a normal shell, so you have to designate it)

> sudo su git -s /bin/bash
> cd /git/repos
> mkdir project.git
> cd project.git
> git --bare init

Finally push it up to a remote master server (adjust the url to your remote server and the path)

> git remote add origin
> git push origin master

Clone the repo and pull rsync

From another computer clone the repository

> git clone

After this head into one of the directories from the command line and pull your data (rsync) edit command for your username.

> ./ pull patman

I could have written on script to run all the scripts and pull (rsync) all the data down, but that is not my goal here.   My goal is to be able to download a folder's contents, remotely, in a pinch.  Otherwise I do not want to take up the hard drive space (Or in my specific case my thumb drive's space)

In conclusion

Now any new folder you want to rsync just run this command in the folder.  (create a file)

> touch

Then run the command in the base directory to update the script

> ./

Then run the new script to push pull the folder

> ./ push

> ./ pull

And of course make sure your .gitignore is ignoring the folders you intent to rsync and not the scripts themselves.

Update  8/30/2014

I had an issue trying to set up .gitignore in  another folder.  Instead of fighting the nightmare that can be .gitignore I did the following.

Set up .gitignore to ignore the folders you want to ignore.  Then place files in the folders you want to rsync.  Finally run this command line command to add all to the git repo.

> find $PWD -name "" | xargs -I "{}" git add -f {}


No comments:

Post a Comment