I was setting up Azure File Sync tonight and ran into a weird issue that I found absolutely nothing online about. Whenever I tried to add this one server as a server endpoint, I kept getting an error stating the provisioning was canceled and in the progress area it said "Provisioning failed" with the Enabling Sync step saying "Server job with command name NewServerEndpointWithDatePolicy and id <some guid> failed with -2134347507."
Not very descriptive! And searching yielded no results, so it's apparently not a common error.
Well thankfully the Azure File Sync agent has very good logging. In the Event Viewer, you look under Applications and Services Logs, Microsoft, FileSync, Management, Operational for these types of errors.
The very last entry was the error, event ID 5103.
This is basically the same text that showed up in the portal, but with one vital extra clue: Sync is not supported on a volume using compression.
Oh, well that explains it! Except... The only problem is that this volume isn't compressed. I double checked this.
Yup, not compressed. But maybe some deep subfolder is compressed. So I reset compression by recursively disabling compression on the entire volume again. So now I knew none of the files were compressed.
Trying again led to the same error. This is where I hit a brick wall and decided to take a break.
I found later that the entire volume had been compressed at some point in the past. But this was years ago, and as shown above, compression is now disabled.
So it appears that disabling compression isn't enough for the validation step of Azure File Sync. Something from the past is still lurking around in the system. What could the agent possibly be looking at that made it think the volume was still compressed?
At this point, I put on my programmer's hat and tried to think how they developed the detection logic. Since nobody else I could find is having this issue, it must be something pretty unique or rare happening here. I checked WMI properties on the volume (there's a flag for compressed volume), but that also showed that the volume was not compressed. I dug a bit in the registry, but I couldn't find anything interesting. I looked up the Win32 API call to obtain volume properties and executed that, but it also the volume was uncompressed. So what is the agent looking at?
Then it dawned on me. NTFS metadata is stored in the system hidden "System Volume Information" folder that is at the root of every drive. Maybe there is a clue there. I went into the folder properties of File Explorer and disabled "Hide protected operating system files (recommended)."
Then I went back to the root and it all became very clear.
Compressed folders show up in blue because I have "Show encrypted or compressed NTFS files in color" enabled (this is disabled in Server 2016+ by default). But if you look at the properties of the folder, you can also clearly see that the System Volume Information folder itself is compressed.
So all I needed to do was uncheck that, apply recursively, and be on my way, right? Nope. System Volume Information is highly protected by the operating system due to the critical nature of the data stored in there. So to remove compression, we have to go through a lot of hoops first.
The easiest way to do this is to run a few commands from an elevated command prompt. The GUI just gets in the way.
First I took ownership of System Volume Information. (If you are going to repeat these steps, obviously change the commands to your admin username and file path where applicable.)
TAKEOWN /F "D:\System Volume Information" /R
A bunch of errors flashed by, but this is okay. This will get us far enough. Now I ran the following two commands.
ICACLS "D:\System Volume Information" /GRANT Administrator:F
ICACLS "D:\System Volume Information" /GRANT Administrator:F /T /C
More errors flashed by, also okay.
Now back in the GUI, I went to the properties and unchecked the compression option. I chose apply recursively and watched it go. It ran into a lot of errors (of course) and I just had to keep clicking through them until it finished. Curiously, when I was done, all the subfolders were no longer comrpessed, but the System Volume Information root folder still showed up as compressed. I repeated the steps but chose "this folder only" this time and that took care of that. Weird, but okay.
At this point, I rebooted the server. On startup, Windows will reset the permissions to System Volume Information, so we don't have to worry about undoing the permissions we added above.
Back in Azure, before I could try to add the server endpoint again, I had to delete the previous attempt. It's not intuitive (no surprise for Azure) but in this case, you right click and select delete.
Now I attempted to add the server endpoint back and waited a few minutes for provisioning to start.
Success!
From this we can deduce that the developers looked at the compression attribute on System Volume Information for the detection logic instead of the drive root or via the volume properties API call. That's different from the other ways I saw how to do it when I googled, but maybe it's the correct way for whatever reason compression has to be disabled for Azure File Sync? I really don't know. It may also be a bug.
In any case, the error message needs to be written a lot more clearly, this issue was not obvious at all!