Sunday, April 16, 2023

GlusterFS Add a New Server to the Pool

Its set up roughly like this.

I want to add one new fresh Gluster Server to the Pool and update the two volumes.  I want the Replicated to be Distributed Replicated volume and the Dispersed to remain the same type but add one more brick on the new server to it. see https://docs.gluster.org/en/latest/Quick-Start-Guide/Architecture/#types-of-volumes [1] for details on volume types.

Also see this diagram for what I am trying to move to.

 


Create a new Machine from scratch

I am not going to go into any detail on the first creation of the server itself.   The server is an ubuntu 22.04 with a secondary drive attached to it and mounted.  Now I just need to install glusterfs like I did in this post http://www.whiteboardcoder.com/2023/02/installing-glusterfs-103-on-ubuntu-2204.html [2]

So let’s get it set up

OK run the following commands to trust this and get it updated

 

 > sudo add-apt-repository ppa:gluster/glusterfs-10

 

Now update

 

 > sudo apt update

 

Check version

 

 > apt show glusterfs-server

 

10.4 I will take that!

Now install glusterfs

 

 > sudo apt install glusterfs-server

 


 

Start GlusterFS up

Start it up

 

 > sudo systemctl start glusterd

 

Check its status

 

 > sudo systemctl status glusterd

 

Now enable it so it starts after a reboot (You will notice in the above image it is currently disabled)

 

 > sudo systemctl enable glusterd

 

Check the status

 

 > sudo systemctl status glusterd

 

It is enabled, now reboot

 

 > sudo reboot

 

Check if its running after a reboot

 

 > sudo systemctl status glusterd

 

And we are happy now.

 

 

Add new GlusterFS server to pool and check status

My new server is at 192.168.0.203 so run this command from on of my existing glusterfs servers

 

 > sudo gluster peer probe 192.168.0.203

 

Then check

 

 > sudo gluster peer status

 

 

Now checking out the current volume data

 

 > sudo gluster volume info

 

 



 

Expand the Dispersed Volume

Some good notes on this can be found at https://docs.gluster.org/en/v3/Administrator%20Guide/Managing%20Volumes/#expanding-volumes [3]

volume-two is my dispersed volume run this command to add an extra brick to it

 

 > sudo gluster volume add-brick volume-two 192.168.0.203:/data/brick2

 

 

volume add-brick: failed: Incorrect number of bricks supplied 1 with count 3

 

OK that failed

Why?

Here is the original command I used to create this volume

 

 > sudo gluster volume create volume-two disperse 3 redundancy 1 192.168.0.200:/data/brick2 192.168.0.201:/data/brick2 192.168.0.202:/data/brick2
  > sudo gluster volume start volume-two

 

I guess in the end I want it to be a disperse 4 with a redundancy 1.

So how do I do that?

Another try.

 

 > sudo gluster volume add-brick volume-two disperse 4 redundancy 1 192.168.0.203:/data/brick2

 

No go on that either

Hmm doing some research

 

Note

When expanding distributed replicated and distributed dispersed volumes, you need to add a number of bricks that is a multiple of the replica or disperse count. For example, to expand a distributed replicated volume with a replica count of 2, you need to add bricks in multiples of 2 (such as 4, 6, 8, etc.).

 

So I could expand it but I would have to add 3 bricks to it at a time since the original had 3 bricks.

I don’t want to do that so what choices do I have?

I think the only real viable answer is to create the new volume with a new name and have it set up how I want it.  Then mount it and copy the files over I want from the original volume.  I may have to delete files after I copy them if I don’t have enough overall space since the bricks of both are on the same servers.   Then once all that is done stop the first volume and rename it.  Then stop the new volume and rename it to the original volumes name.  Then go around and make sure everyone who has mounted the original volume remounts the new volume.  This is all doable but would have some downtimes.  I just wish I could add a brick and have it adjust itself… Oh well it can’t … in the manner I want it to…


Create New Volume

 

 > sudo gluster volume create volume-three disperse 4 redundancy 1 192.168.0.200:/data/brick3 192.168.0.201:/data/brick3 192.168.0.202:/data/brick3 192.168.0.203:/data/brick3
  > sudo gluster volume start volume-three

 

 

Now let me mount this volume from my client server

 

 > sudo vi /etc/fstab

 

And place the following in it

 

192.168.0.200:/volume-three /volume_three_client glusterfs defaults,_netdev 0 0

 

 

Make the mount point folder

 

> sudo mkdir /volume_three_client

 Now mount them

 

 > sudo mount -a

 

And check them out

 

 > df -h

 

Change the ownership of the mounts to my user

 

 > sudo chown $USER:$USER /volume_three_client/

 

 

Now check the volume status from one of the gluster servers

 

 > sudo gluster volume info volume-three

 

 

 


Copy Files

Now I do not have much data so I can easily copy everything from volume-two to volume-three.  I can just use this command

 

 > sudo rsync -a /volume_two_client/ /volume_three_client

 

This may take some time if there are lots of files.  Keep that in mind.

 

Change Volume names

 Stop the old volume and rename it

 

 > sudo gluster volume stop volume-two
 > sudo gluster volume rename volume-two temp-name

 

But we get a failure….

Poking around it looks like this CLI feature was removed at some point.

 

 > sudo gluster volume stop volume-two
 > sudo gluster volume set volume-two cluster.local-volume-name temp-name

 

 

 > sudo gluster volume info

 

OK that did not really work…

After poking around for a long time, I did not find an answer that worked.

 

Clean up

Now to clean up we need to remove the original volume-two.

 

 > sudo gluster volume delete volume-two

 

 

 

 > sudo gluster volume info

 


 

Volume Name: volume-one
Type: Replicate
Volume ID: f1a9c18b-0a8b-4e65-bfd7-047d246cfa2b
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 192.168.0.200:/data/brick1
Brick2: 192.168.0.201:/data/brick1
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
storage.fips-mode-rchecksum: on
cluster.granular-entry-heal: on

Volume Name: volume-three
Type: Disperse
Volume ID: a11ded3c-ef03-4e73-8d66-d5d251ae442c
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (3 + 1) = 4
Transport-type: tcp
Bricks:
Brick1: 192.168.0.200:/data/brick3
Brick2: 192.168.0.201:/data/brick3
Brick3: 192.168.0.202:/data/brick3
Brick4: 192.168.0.203:/data/brick3
Options Reconfigured:
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on

 

Here we can see the volume is no longer there

The folders on the glusterfs servers are still there

We need to remove the /data/brick2 from every gluster server that had it.

 

 > sudo rm -r /data/brick2

 

 

Expand the Replicated

I think we will be able to change the Replicate volume to a Distributed Replicated volume like this.

Run this command to see my before.

 

 > sudo gluster volume info volume-one

 

 

 

 

 > sudo gluster volume add-brick volume-one 192.168.0.202:/data/brick1 192.168.0.203:/data/brick1

 

 

Hey that worked

 

 > sudo gluster volume info volume-one

 

Hey that looks good.

Doing a few quick checks it has expanded and it looks all good 😊

So I guess I can expand a replicate volume easy enough, if I do it in 2x with my set up.

References

[1]       Replicated GlusterFS Volume
              
https://docs.gluster.org/en/latest/Quick-Start-Guide/Architecture/#types-of-volumes
            Accessed 04/2023
[2]       Installing GlusterFS 10.3 on Ubuntu 22.04 and get it working
            http://www.whiteboardcoder.com/2023/02/installing-glusterfs-103-on-ubuntu-2204.html
            Accessed 04/2023
[3]       Expanding Volumes
              
https://docs.gluster.org/en/v3/Administrator%20Guide/Managing%20Volumes/#expanding-volumes
            Accessed 04/2023

 

 

 


 

 

 

 

No comments:

Post a Comment