I am a Veeam Vanguard 2021

I am a Veeam Vanguard 2021

This afternoon I received an email notifying me that my 2021 Veeam Vanguard Renewal Nomination has been approved! That means I am a Veeam Vanguard 2021!

I am a Veeam Vanguard 2021
Honored and proud to be a Veeam Vanguard again in 2021!

The Veeam Vanguard Program

You can read all about the Veeam Vanguard Program here. It is Veaam’s top-level influencer community. We share our insights, provide feedback, and help each other, as well as Veeam, succeed. The Veeam Vanguards are a varied group of people and combine a wide range of expertise in different verticals and in different disciplines. We all contribute to the community at large in different ways. What unites us is that we all share a passion for our industry and show technical thought leadership for the different technical communities in which Veeam exists.

Now I would not call myself a technical thought leader but I do know that via this program I get to stand on the shoulders of giants. Doing so gives me a majestic view over this industry segment, a view I would not have without it.

Thank you!

This is a very nice way to head into the weekend. I am both honored and proud to be a Veeam Vanguard. Veeam as a company has won my respect many years ago and it has only grown. They focus on customer value, deliver an excellent high-quality product, offer top-notch support, and finally offer great tools to the community. I can honestly say that I enjoy working with their products as well as the company and the individuals I know at Veeam.

I am looking forward to another year in this program and the opportunities this brings to us all for sharing, learning, and growing.

Thank you, we’ll make 2021 a good community year!

Change the service account on a Veeam hardened repository

Introduction

In Veeam Backup & Replication (VBR) v11 we have the new hardened repository host. I Have also shared some other findings in previous blogs. You add this to the Veeam managed servers via single-use credentials. These are not stored on the VBR server. That account only needs permission on the repository volumes after adding the repository server to the VBR managed servers. Let’s investigate how to change the service account on a Veeam hardened repository in case we ever need to. Warning, this early info and not official guidance.

Where is that account configured for use with Veeam?

Naturally, we created the account for adding the hardened repository to the Veeam managed servers. We also set the correct permission on the backup repository volumes. After adding the repository host to Veeam managed servers, with the single-use credential method, we removed this account from the sudo group and that’s it.

The VBR server itself does not hold the credentials. The credentials only live on the repository host. But where is that configured? Well under Under /opt/veeam/transport you will find a config file VeeamTransportConfig.

Change the service account on a Veeam hardened repository
Under /opt/veeam/transport you find a config file VeeamTransportConfig

Open it up in your favorite editor and take a look.

Change the service account on a Veeam hardened repository
ServiceUser is where the user account is configured

Change the service account on a Veeam hardened repository

From my testing, you can simply change the user account in the VeeamTransportConfig file if you ever need to. When you do, save the file and restart the veeamtransport service so it takes effect.

sudo service veeamtransport restart

When you change this service user you must also take care of the permissions on the repository folder(s).

We change the ownership recursively for the drive mount to that user.

sudo chown -R newveeamrepouser: newveeamrepouser /mnt/veeamxfsrepo01-02

The permissions for the user should still be what it needs to be (chmod 700)

Now, due to the immutability of the backup files changing the owner will fail on those files but that is OK.

Change the service account on a Veeam hardened repository
Don’t panic, you’ll be fine. Backups & restores will work.

For reading and restoring then tests show that this still works. Backup wise this will run just fine. It is all pretty transparent from the Veeam Backup & Replication side of things. I used to think I would need to run a full backup afterward but Veeam seems to handle this like a champ and I do not have to take care of that it ssems. We’ll try to figure out more when V11 is generally available.

Now, you still might want to give Veeam support a call when you want or need to do this. Remember, this is just informational and sharing what I learn in the lab.

Conclusion

Normally you will not need to change this service account. But it might happen, either because of a policy that mandates this, etc. That is the reason I wanted to find out how to do this and if you can do this without breaking the backup cycles. I also think this is why you can convert an existing Linux repository to a hardened one.

Immutability of Linux files on the Veeam hardened Linux repository

Introduction

Without any doubt, I find the hardened Linux repository Veeam introduces in Veeam Backup & Replication v11 one of the most fascinating new features to get my hands on. In the ever-escalating battle with ransomware and wipers, this is a very valuable option to have in your defensive arsenal. So, I grabbed the Beta 2 and got to work in the lab over the holidays to investigate and find out some details about the Immutability of Linux files on the Veeam hardened Linux repository.

Immutability of Linux files on the Veeam hardened Linux repository

I’s quite easy to find the file attribute “i” that marks a file as immutable.

attr -a -l attr -a -l 
Immutability of Linux files on the Veeam hardened Linux repository
lsattr -a also shows the hidden files

or

attr -a -l
Immutability of Linux files on the Veeam hardened Linux repository
lsattr -a -l list out the full name of the attribute.

Where is the information about the immutability actually stored? I mean, that “i” attribute is one thing but how do the Linux host and Veeam know from what time period this immutability is valid. In the end, the service has to clear it and know when to do this. Or is this only stored in the Veeam database or both?

How does it now from when till when a file must be immutable?

Digging around in the files and folders of the Veeam repository, I soon found the lock file “.veeam.x.lock” (see the green arrow in the image above) that is created by the veeamimmureposvc service. The owner is root, hence it is not created by the Veeam transport service. The veeamimmureposvc service is a local account with root access for managing the immutability. It only works locally and does not listen on any network port, hence it cannot be accessed remotely.

Immutability of Linux files on the Veeam hardened Linux repository
The veeamimmureposvc service controls the .veeam.x.lock file. the x is a number has increments with every backup job you run.

Let’s look inside to see if we can read something there?

cat .veeam.9.lock
the lock file is an XML file containing all the date/time stamps for every file in that backup job.

When you open that file you will find it to be an XML file. Inside you’ll see the date and time stamp for every file in the backup chains for that job. That’s cool.

But there is more. When we run to look for extend file attributes we find that every Veeam created file has a one called user.immutable.until.

The backup files all have an extended file attribute called user.immutable.until.

With that name, it is clear it can be of interest to us. If you look at what is in there, you’ll see it contains the date and time stamp for that file’s immutability period.

getfattr * -n user.immutable.unil 
Immutability of Linux files on the Veeam hardened Linux repository
The extended file attribute contains the timestamp until when that backup file is immutable!

That I find interesting. Veeam saves the information twice. Is that for redundancy or as some sort of checksum? Maybe it also has to do with the fact Veeam backup files are transportable and self-contained so that information is stored as an extended file attribute.

Conclusion

So there you have it. A small piece of information on where the immutability information is stored. The most surprising thing to me was that it is actual stored twice.

I hope you fund this interesting. Poking around to figure out the how and what of things always helps me tremendously to learn and understand the technologies I want to work with. That leads to better decisions in design and implementation. It leads to both trust and confidence, which helps me decide where and when to leverage it. Finally it also, almost without, it is invaluable when supporting the technology.

Extending a Veeam Repository XFS File System

Extending a Veeam Repository XFS File System

Since diving into the Veeam Backup & Replication v11 Linux hardened repository I have started to use XFS in bite-size deployments to gain experience with it. One of the things that will certainly come in handy is extending a Veeam Repository XFS File System. In this blog post, I show to do that.

Mind you that I am doing this with a virtual machine on Hyper-V (Windows Server 2019) in the lab. Not every permutation of hardware and storage controllers you can find. But still, the procedure here will not differ that much.

Determine the size of the current disk.

sudo slblk
Extending a Veeam Repository XFS File System
Ours is the 20 TB disk, sdd, a SCSI disk.

Now take note of the bytes and sectors

sudo fdisk -l 
We just notice the size, bytes and sectors to compare after we extended disk.

Expand the disk

In the virtual machine settings I extend the virtual disk I want to grow with the required capacity.

Extending a Veeam Repository XFS File System
Let’s add 20 TB and make it 30 TB in total.

In real life that might be you growing a raid controllers’ virtual disk by adding physical disks to the raid controller, you expanding the volume on the storage array or simply adding disks to the local server and adding them to the software-based raid solution you use.

The virtual machine will pick up the extra capacity right away. For our UBUNTU 20.04.1 OS to see it up we’ll need to rescan the SCSI busses for change. In a virtual machine, this can be done via rescan-scsi-bus.sh, available scsitools that will need to be installed if not there.

Use the -s options as that will really show the resized disks.

sudo apt-get install scsitools
sudo rescan-scsi-bus.sh -s
Yup, that’s our disk on SCSI controller 1, location 0.

Now let’s check the disk size again

Yes, lsbsk shows 30 TB.
fdisk -l confirms. Note the new bytes and sector values. It has gone up.

Extend the xfs volume to use the unallocated space

Now we need our xfs volume to use the unallocated capacity in this disk. We use -d as this will grow the file system to the largest possible size, 30 TB in our case.

Note: If you run the below command with -n instead of -d, this gives you the current information on your xfs volume with extending the filesystem yet.

sudo xfs_growfs -d /mnt/veeamxfsrepo-03
Extending a Veeam Repository XFS File System
Voila. We are done.

See Ubuntu Manpage: xfs_growfs – expand an XFS filesystem for more options

Note: What I did find is that if you just expand the disk and than extend the xfs file system, it also works. It seems to just work without rescanning the disk after extending it! The disks size in df -h will show this space then as well.

Conclusion

That was it. Short and sweet. There is not much to it once you know how to do it. One thing to remember is that you cannot shrink an XFS file system. So, as always, start smaller and grow when needed. Always leave spare capacity to work with when needed. Yes, even in 2021 this is advice to live by in the storage world. For Veeam this means that multiple smaller repositories or extents give you more wiggle room than fewer very large ones. Leave capacity in reserve, either in a spare repository/extend or unallocated. This, especially combined with a scale-out backup repository in Veeam will allow you to work your self out of most capacity pickles you might find your self in.