How To Fix "Item [Backup Job NAME] Is Locked By Running Session Backup Job NAME [Agent Backup]" In Veeam Endpoint

In the event that you encounter the error in the title while running Veeam Endpoint Backup (Veeam Agent for Microsoft Windows) it most likely means that your backups will cease to work, as mine have. The issue appears to be a flag set in the Veeam database which informs Veeam that a backup is running and the backup files for the job in question are locked. If the system has an unexpected shutdown during the backup window this flag can remain set. This leaves Veeam in a somewhat troubled state, the flag is set that backup files are locked, but no job is running, meaning that flag will never be removed by what I assume would be part of the job completion process.

There seem to be few options for resolving this, the highest Google result is a Veeam forum post that boils down to “reinstall and start a new backup chain”. I’ve done my fair share of uninstalls/reinstalls in my time, and I’m a little sick of them, so I did the only reasonable thing to avoid 15-20 minutes of mind numbing reinstall work, I spent an hour digging into Veeam Endpoint and successfully resolving the issue, leaving my precious backup chain intact and learning a few things along the way.

TL;DR

I’ll give a summary of the fix first as I want to go in to a bit of detail on how I got to the solution, there are several things I picked up during the process I think are worth sharing, but for those that just want their backups to work, here it is:

1. Download and install the sqlcmd utility
   • A link to the latest version is here
2. Stop the Veeam Agent for Microsoft Windows (VeeamEndpointBackupSvc) Service
   • net stop VeeamEndpointBackupSvc
3. Download PSExec, and open a cmd.exe prompt as NT AUTHORITY\SYSTEM
   • PsExec.exe -i -s cmd.exe
4. Back up the database files in C:\ProgramData\Veeam\EndpointData from the SYSTEM prompt
   • copy VeeamBackup.mdf VeeamBackup.mdf.BACKUP
   • copy VeeamBackup_log.ldf VeeamBackup_log.ldf.BACKUP
5. Run the following commands from the SYSTEM prompt

   sqlcmd -S (LocalDb)\VeeamEndPoint -d "VeeamBackup" -Q "delete from [Backup.TrackedActions.Leases]"
   sqlcmd -S (LocalDb)\VeeamEndPoint -d "VeeamBackup" -Q "delete from [Backup.TrackedActions.LockItems]"
   sqlcmd -S (LocalDb)\VeeamEndPoint -d "VeeamBackup" -Q "delete from [Backup.TrackedActions.Locks]" 

6. Start the Veeam Agent for Microsoft Windows (VeeamEndpointBackupSvc) Service
   • net start VeeamEndpointBackupSvc
7. Your backups should now work
8. Delete your SQL backup files

The Journey

Alternative title: What the hell is LocalDB?

The first clue was this Veeam Knowledge Base article, it gives us some SQL commands to run against the Veeam database, though it’s written for Veeam Backup & Replication rather than Veeam Endpoint Backup. I made a bit of an assumption here that Veeam would use much of the same internals in Endpoint Backup because why re-invent a wheel that you’ve already invented?

However, a quick look at the services on my local machine showed that no SQL services were present. I tried a telnet to port 1433 on localhost in case Veeam were doing some odd MSSQL Express embedding I hadn’t seen before, but this was also unsuccessful. SSMS showed no database instances running locally, and Add/Remove Programs showed no MSSQL install either.

I then started looking through the Veeam Endpoint install folders, hoping to either find some other database files (SQLite perhaps?), or some configuration files - anything that would point to where data for the backup jobs was being stored. First I checked in C:\Program Files\Veeam\Endpoint Backup to no avail, and then hit the jackpot in C:\ProgramData\Veeam\EndpointData.

This folder had 2 files: VeeamBackup.mdf, VeeamBackup_log.ldf.

MDF and LDF files are database files, typically (but not exclusively) used by Microsoft SQL Sever, the MDF (Main Data File) is the database, and the LDF (Log Data File) is the transaction log. This confirmed some kind of database was indeed being used, but I needed to find what and how.

I pulled out ProcessExplorer and searched for open handles on the VeeamBackup.mdf file. There was one result, a child process of Veeam.EndPoint.Service.exe, sqlservr.exe. Now we knew for sure MSSQL was the database, but it wasn’t the usual SQL Server service I was familiar with. Looking at the command line for the sqlservr.exe process revealed the next clue.

"C:\Program Files\Microsoft SQL Server\110\LocalDB\Binn\\sqlservr.exe"   -c -SMSSQL11E.LOCALDB -sLOCALDB#28578ABD -d"C:\Windows\System32\config\systemprofile\AppData\Local\Microsoft\Microsoft SQL Server Local DB\Instances\VeeamEndPoint\master.mdf" -l"C:\Windows\System32\config\systemprofile\AppData\Local\Microsoft\Microsoft SQL Server Local DB\Instances\VeeamEndPoint\mastlog.ldf" -e"C:\Windows\System32\config\systemprofile\AppData\Local\Microsoft\Microsoft SQL Server Local DB\Instances\VeeamEndPoint\error.log"

Microsoft SQL Server\110\LocalDB? I was loosely familiar with SQL Server paths and I was not used to seeing LocalDB - so what is LocalDB? A quick Google revealed it’s a minimal deployment of SQL Server Express, essentially an embedded instance.

The documentation also provided some instructions for how we can connect to a LocalDB instance, essentially (localdb)\<InstanceName>. Now we have to find the instance name.

I would love to say I looked at the command line above and pulled it right out of the path Microsoft SQL Server Local DB\Instances\VeeamEndPoint\master.mdf, but I didn’t. I skimmed through the command, and completely missed the relevant part. Goddamn it. Googling led me to the SqlLocalDB command line utilility used to create and manage LocalDB instances.

sqllocaldb i prints out all SQL Express LocalDB instances owned by the current user. I tried this under my own account context (with no success - zero instances were listed), but as the sqlservr.exe process was running as SYSTEM I assumed we needed to be in its context. PSExec came to our aid, and we had our instance name.

C:\WINDOWS\system32>sqllocaldb i
v11.0
VeeamEndPoint

C:\WINDOWS\system32>sqllocaldb i VeeamEndPoint
Name:               VeeamEndPoint
Version:            11.0.7001.0
Shared name:
Owner:              NT AUTHORITY\SYSTEM
Auto-create:        No
State:              Running
Last start time:    1/4/2020 12:49:08 AM
Instance pipe name: np:\\.\pipe\LOCALDB#28578ABD\tsql\query

I fired up SSMS, first trying under my own context again (I have an aversion to running things as SYSTEM) but without success. However, running as SYSTEM, and using (LocalDb)\VeeamEndPoint as the server worked!

At this point I stopped the Veeam Agent for Microsoft Windows service, took backups of the MDF/LDF files, and went to explore the database.

All three of the tables referenced in the previously linked Veeam KB article were present, and two had a single row of data, things were looking promising. I ran the SQL query from the KB article against the two tables with data, though doing it on all 3 would do no harm.

delete from [Backup.TrackedActions.Leases]
delete from [Backup.TrackedActions.LockItems]
delete from [Backup.TrackedActions.Locks]

Lastly I restarted the Veeam Agent for Microsoft Windows service, opened the console, and ran a successful backup.