Sunday, April 16, 2023

GlusterFS Add a New Server to the Pool

Its set up roughly like this.

I want to add one new fresh Gluster Server to the Pool and update the two volumes.  I want the Replicated to be Distributed Replicated volume and the Dispersed to remain the same type but add one more brick on the new server to it. see [1] for details on volume types.

Also see this diagram for what I am trying to move to.


Create a new Machine from scratch

I am not going to go into any detail on the first creation of the server itself.   The server is an ubuntu 22.04 with a secondary drive attached to it and mounted.  Now I just need to install glusterfs like I did in this post [2]

So let’s get it set up

OK run the following commands to trust this and get it updated


 > sudo add-apt-repository ppa:gluster/glusterfs-10


Now update


 > sudo apt update


Check version


 > apt show glusterfs-server


10.4 I will take that!

Now install glusterfs


 > sudo apt install glusterfs-server



Start GlusterFS up

Start it up


 > sudo systemctl start glusterd


Check its status


 > sudo systemctl status glusterd


Now enable it so it starts after a reboot (You will notice in the above image it is currently disabled)


 > sudo systemctl enable glusterd


Check the status


 > sudo systemctl status glusterd


It is enabled, now reboot


 > sudo reboot


Check if its running after a reboot


 > sudo systemctl status glusterd


And we are happy now.



Add new GlusterFS server to pool and check status

My new server is at so run this command from on of my existing glusterfs servers


 > sudo gluster peer probe


Then check


 > sudo gluster peer status



Now checking out the current volume data


 > sudo gluster volume info




Expand the Dispersed Volume

Some good notes on this can be found at [3]

volume-two is my dispersed volume run this command to add an extra brick to it


 > sudo gluster volume add-brick volume-two



volume add-brick: failed: Incorrect number of bricks supplied 1 with count 3


OK that failed


Here is the original command I used to create this volume


 > sudo gluster volume create volume-two disperse 3 redundancy 1
  > sudo gluster volume start volume-two


I guess in the end I want it to be a disperse 4 with a redundancy 1.

So how do I do that?

Another try.


 > sudo gluster volume add-brick volume-two disperse 4 redundancy 1


No go on that either

Hmm doing some research



When expanding distributed replicated and distributed dispersed volumes, you need to add a number of bricks that is a multiple of the replica or disperse count. For example, to expand a distributed replicated volume with a replica count of 2, you need to add bricks in multiples of 2 (such as 4, 6, 8, etc.).


So I could expand it but I would have to add 3 bricks to it at a time since the original had 3 bricks.

I don’t want to do that so what choices do I have?

I think the only real viable answer is to create the new volume with a new name and have it set up how I want it.  Then mount it and copy the files over I want from the original volume.  I may have to delete files after I copy them if I don’t have enough overall space since the bricks of both are on the same servers.   Then once all that is done stop the first volume and rename it.  Then stop the new volume and rename it to the original volumes name.  Then go around and make sure everyone who has mounted the original volume remounts the new volume.  This is all doable but would have some downtimes.  I just wish I could add a brick and have it adjust itself… Oh well it can’t … in the manner I want it to…

Create New Volume


 > sudo gluster volume create volume-three disperse 4 redundancy 1
  > sudo gluster volume start volume-three



Now let me mount this volume from my client server


 > sudo vi /etc/fstab


And place the following in it /volume_three_client glusterfs defaults,_netdev 0 0



Make the mount point folder


> sudo mkdir /volume_three_client

 Now mount them


 > sudo mount -a


And check them out


 > df -h


Change the ownership of the mounts to my user


 > sudo chown $USER:$USER /volume_three_client/



Now check the volume status from one of the gluster servers


 > sudo gluster volume info volume-three




Copy Files

Now I do not have much data so I can easily copy everything from volume-two to volume-three.  I can just use this command


 > sudo rsync -a /volume_two_client/ /volume_three_client


This may take some time if there are lots of files.  Keep that in mind.


Change Volume names

 Stop the old volume and rename it


 > sudo gluster volume stop volume-two
 > sudo gluster volume rename volume-two temp-name


But we get a failure….

Poking around it looks like this CLI feature was removed at some point.


 > sudo gluster volume stop volume-two
 > sudo gluster volume set volume-two cluster.local-volume-name temp-name



 > sudo gluster volume info


OK that did not really work…

After poking around for a long time, I did not find an answer that worked.


Clean up

Now to clean up we need to remove the original volume-two.


 > sudo gluster volume delete volume-two




 > sudo gluster volume info



Volume Name: volume-one
Type: Replicate
Volume ID: f1a9c18b-0a8b-4e65-bfd7-047d246cfa2b
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
storage.fips-mode-rchecksum: on
cluster.granular-entry-heal: on

Volume Name: volume-three
Type: Disperse
Volume ID: a11ded3c-ef03-4e73-8d66-d5d251ae442c
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (3 + 1) = 4
Transport-type: tcp
Options Reconfigured:
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on


Here we can see the volume is no longer there

The folders on the glusterfs servers are still there

We need to remove the /data/brick2 from every gluster server that had it.


 > sudo rm -r /data/brick2



Expand the Replicated

I think we will be able to change the Replicate volume to a Distributed Replicated volume like this.

Run this command to see my before.


 > sudo gluster volume info volume-one





 > sudo gluster volume add-brick volume-one



Hey that worked


 > sudo gluster volume info volume-one


Hey that looks good.

Doing a few quick checks it has expanded and it looks all good 😊

So I guess I can expand a replicate volume easy enough, if I do it in 2x with my set up.


[1]       Replicated GlusterFS Volume
            Accessed 04/2023
[2]       Installing GlusterFS 10.3 on Ubuntu 22.04 and get it working
            Accessed 04/2023
[3]       Expanding Volumes
            Accessed 04/2023








No comments:

Post a Comment