If like me, you have been using Pat’s awesome Get-CSFeatures script for your Skype for Business builds. You may have noticed a feature that no longer works with 2019. “Misc Server Config > Set Fabric logging to circular” with the error “This server does not have Windows Fabric Installed”
This is no fault of Pat’s. The error message is correct. Windows fabric isn’t installed!
With the release of Skype for Business 2019 the backend fabric has been moved to Microsoft Azure Service Fabric.
With the move to a new app is a new path for log files.
Windows Fabric used to log to C:\programdata\Windows Fabric\Fabric\log\Traces
However, Azure Fabric logs to C:\ProgramData\Microsoft\SF\Log\Traces
I naively thought that with improvements to 2019 and Azure Service Fabric that logs couldn’t grow out of control like they used to. But within 2 months my logs were already up to 22GB (1 of 3 Enterprise Ed Servers, with 1326 users across the pool)
This also happens on Standard Edition Servers. Here’s one that’s only been up for 2 weeks with no users on it. It’s already at 10GB!
So why don’t you just update the logs to circular manually?
Tom and FlinchBot have written about this before, but I’ve never really needed to worry about it thanks to the feature in Set-CsFeatures
Unfortunately, the old “Logman update trace FabricLeaseLayerTraces -f bincirc –cnf” trick doesn’t work anymore. Even with “-ets” to force it to run straight on the scheduler.
If we look at Logman we can see the following traces are running (note the -ets flag)
And by default, these are all set to linear logging.
Additionally, it’s no longer just the trace’s in the folder that gets large.
But now we have PerfMon graphs to worry about!
A quick look at Logman and we can see that the performance counters are set to not overwrite or circularly record either.
Unfortunately, regardless of whatever voodoo magic I parse to Logman.exe I cant enable circular logging. (Here are some examples, I spent hours trawling through docs.microsoft and trying different combinations)
Digging further I can see something is “Cleaning up” these logs and archiving them anyway. But these Perfmon logs are what is growing out of control.
The real fix
I started writing a script that would automatically clean these logs and more specifically the perfmon data up and got stuck in feature creep like I always do… Automatic installers, self-updates.. blah blah
Then I got sick with pneumonia and stopped working on it. Because you know, breathing is a thing.
Little did I know Chris Hayward was also working on a solution to the fabric logging problem, see his script block over here for adjusting the config and cleaning up the actual Fabirc Logs.
What Chris’s solution doesn’t do yet, is to clean out the “Archived” perfmon data from these folders too.
I’m still working on my own self-installing script, but for now, you can use this script block to clean it up.
As with anything that fiddles with these settings, we aren’t sure if it’s supported by the product group but rest assured we have asked the right people so hopefully, there is an official fix in the works.
Hope this helps.
Added “C:\ProgramData\Microsoft\SF\Log\PerformanceCountersBinary” to the $FoldersToClean list. Had a 32 Gb buildup over 15 months.
Thanks Korbyn, I’ll update the Gist.
I’m running this on my 2019 SFB box and it returns to the PS command line with apparently no actions taken. Can’t figure out why. I have over 22Gb in the PerformanceCountersBinaryArchive folder alone. All the logfile paths match. Have you heard of this before? FYI, I’m no powershell guru!
Hmm, do you even get the “Space saved” output? if there’s nothing at all perhaps the script just isn’t running?
Anyway, you can always just delete the stuff manually and then use Chris’ script to force S4B to use circular logging.
I really need to get around to building an on-prem lab again.