There doesn't seem to be an easy way to just see how much space all your mailboxes are using in the Microsoft 365 Exchange Online portal.  Additionally, while the output from Get-MailboxStatistics has a value "TotalItemSize" it's a string and not a raw value, so it requires a little parsing to w0rk.  Instead of just giving you the number of bytes (or MB or GB) it gives you a string like this: 38.25 GB (41,065,726,825 bytes)

Well, with a little regex and the Measure-Object cmdlet, we can get to where we want.

(Get-Mailbox -ResultSize Unlimited | Get-MailboxStatistics).TotalItemSize.Value -replace '.*?\(|[^\d]*','' | Measure-Object -Sum

It will take a little while, but in the end you'll get the total.  :-)

Count    : 486
Average  :
Sum      : 2339040542238
Maximum  :
Minimum  :
Property :

That's 2.12TB of data in 486 mailboxes, in this case.

I've only seen this on DPM 2022 when using Modern Storage, but presumably this can happen on MABS as well since they share code base and may happen on older versions of DPM.  The problem is that after some time, you will look at storage consumption metrics of various protected resources and see the consumption is being shown as 0MB.  This is concerning, of course, because does it mean you lost your backups?  Well, if you are having the same problem as me, the answer is no.  I was able to see storage valid recovery points, could restore from them, and could also see storage space being used in the storage pools.  So the issue seems to only impact queries of storage consumption, not the validity of backups, or the free space in the storage pool.

I haven't been able to find a cause or a trigger for this, I'm going to guess this is a bug that doesn't affect a huge number of users.  However, when I opened a case with Microsoft, the technician immediately knew what was happening and led me to a solution (below).  He indicated that it should permanently resolve the issue, but if not, just keep running the same command again.  It happens frequently enough for me I just decided to run this weekly in a scheduled task.

Anyway, to fix, open the DPM (or MABS) PowerShell console, and run the below PowerShell script.  It is located in the "bin" folder for the installation, so something like C:\Program Files\Microsoft System Center 2022\DPM\DPM\bin.

.\Manage-DPMDSStorageSizeUpdate.ps1 -ManageStorageInfo UpdateSizeInfo

Macs (and other Apple devices, possibly) leave little turds in the file system (.DS_STORE, etc).  They are little metadata files for Apple devices only, but are also not critical files for Apple so are entirely safe to delete.  They are completely useless for Windows or Linux.  Get rid of them with this PowerShell one-liner.

Get-ChildItem .DS_STORE,__MACOSX -Recurse | Remove-Item -Force -Recurse

Back in mid-2022, as part of their alignment with the web version of Outlook UI, Microsoft moved the navigation bar in Outlook desktop from the bottom to the side.  This also allowed you to pin apps to this side bar.  I found the app pinning to be less than useful as they are just basically links to the apps I don't use often or at all (Yammer, Todo), or I already have pinned to my task bar (Excel, Word, OneNote, etc.).  I prefer having less wasted horizontal screen real estate and just have the calendar and contacts as links at the bottom.

The good news is that based on customer feedback, Microsoft has added the option to turn this off!  And it's easy, but not intuitive.

Go into Options, then Advanced, then uncheck "Show Apps in Outlook."  Screenshots below!

Microsoft has some even more radical changes for Outlook coming soon, so keep your eye out on that and provide feedback.  They do listen. 

It took a while, but Microsoft has finally released a preview instance of SCOM in the cloud: Azure Monitor SCOM Managed Instance.  Like all Microsoft products, it's a mouthful, but it's clear where they are going with this and I'm quite pleased with the approach.  As I mentioned in my previous article on this subject, OpsMgr (SCOM) in the Cloud? Finally, Azure Monitor was a fundamentally new monitoring tool and despite Microsoft themselves completely replacing their internal OpsMgr instances with it, this wasn't something that appealed to a lot of enterprises. 

Azure Monitor is very focused on log aggregation and custom alerting and monitoring.  While SCOM has a steep learning curve, once you learned it, new resources get added to your health model automatically and the amount of customization you can do to large groups of monitored objects is just unprecedented in the monitoring space.  To me, It didn't (and still doesn't) make sense to throw all that out, rewrite everything from scratch, and then roll that into what are basically templates that have to be redeployed through custom automation you have to create yourself, which is what Log Analytics and the monitoring in Azure Monitor based on it provide to admins.  Don't get me wrong, this approach has it's place.  It's very tuned into a specific workflow: CI/CD.  And for that it makes sense.  But the world of enterprise infrastructure is so much more than that.  If you are developing an application, this approach works great.  If you are back office and supporting infrastructure, I just don't think that works well. 

And the market has seemingly spoken as Microsoft was very surprised by how slowly (if at all) OpsMgr users actually migrated to Azure Monitor, continuing even to pay the very high amount of licensing for cloud-based monitoring (you can generally get better VM density on-premises which lowers the effective "per VM" fee for SCOM).  In addition, there are dozens of RMM products that essentially use similar monitoring models as what SCOM has perfected.

So this is great news this is public preview.  Right now it looks to be very IaaS-oriented.  You spin up a managed management server instance, managed SQL instance, and you are good to go.  I hope that as time goes on, the model shifts much more to PaaS, where I do not have to concern myself with knowing what the backend even looks like.  I also hope the licensing follows this model, giving a choice between using legacy licenses or simply doing a per-resource or per-VM licensing fee, removing the current compute infrastructure fee you pay for the managed instance.  If the per-VM model doesn't cover all the cloud hosting costs, they could do what they did with Log Analytics and charge for total storage, retention, and other metrics that aren't based around the underlying compute.  A fully cloud PaaS should not concern its customers with the implementation details, just charge a reasonable monthly fee for platform usage based on easy-to-understand and easy-to-predict fee structure.  But this is Microsoft we are talking about, so I fully expect the licensing to be ridiculous, but hopefully they listen.

I don't want to end this negative!  This is truly a good thing.  The best on-prem monitoring platform is now also the best cloud monitoring platform and the future is bright.  Moving more of the GUI into Azure's management portal (which they are working on), tighter integration with Log Analytics, so you can have both log-based and probe-based monitoring, and new true innovation on the platform is good for everybody in infrastructure.  I'm excited to see where this goes!

Here's a video put out by the great folks at SquaredUp in conjunction with Microsoft about this new preview:

Every week there are 10-20 Shazams of my music, mostly coming from Japan.  If you happen to stumble on this website after doing a Shazam, please reach out.  I'd love to know where you heard the songs.  I haven't released any music since 2008, so this is a surprise to see an uptick in 2021-2022.  :-)

Drop (formerly Mass Drop) makes some pretty nice looking, customizable, reasonably priced (for mechanical) keyboards.  Myself, I bought the CTRL, which is the "tenkeyless" style.  I got Kailh Box White switches.

There's just one really big problem: when you press a key, it sometimes registers two or more presses.  A lot.  Sometimes this is referred to as a "chatter problem."

There's a lot of discussion on this and how to remediate it and a lot of disappointed customers.  Some people look at it as a software problem -- there are suggestions to manipulate the debounce and wait times inside QMK (the embedded OS that these keyboards run).  I had limited success with this.  There are also suggestions to pop the switch out and "wipe off dust" or replace the switches entirely.  This didn't work for me and frankly it doesn't pass the sniff test.  Are really that many people getting dozens to hundreds of faulty switches or dusty connections?

So I tried a different approach.  Still thinking on the second suggestion that the problem might be the electrical connection between the switches and board,  I tried using DeoxIT Gold contact cleaner/lubricant.  This worked.  It completely solved the issue.  No more chatter!

I'm a huge fan of DeoxIT products.  It is expensive, but it absolutely works as advertised.  I've used this for years to help with scratchy faders or bad analog audio connections, it's no wonder it also works on mechanical keyboard switch connections.

So how do you use it?  It's easy, but tedious.  Buy the brush-on kind, pop out every single switch, and apply it to each metal connection pin.  Then stick it back in.  It is 100% fixed.  No double presses.

I used DeoxIT G100L (aka "Gold" or "Pro Gold") but the D100L solution probably works just as well, maybe better since it's designed for non-gold connectors.

After removing all the switches, ready to apply DeoxIT!

DeoxIT Gold:

Before application:

After application:

Microsoft publishes a page which contains all the keyboard shortcuts for Outlook (and other Office products, but I needed Outlook in this instance, so we'll stick with that). It's located here: https://support.microsoft.com/en-us/office/keyboard-shortcuts-for-outlook-3cdeb221-7ae5-4c1d-8c1d-9e63216c1efd.  The problem is that it's just HTML, so while it's good for a quick reference, it's not something you can easily copy and paste into another format, like say Excel, to whip up a nice handy table you can print out or format some other way.

PowerShell to the rescue!

Taking a quick look at the page source shows that all the shortcuts are in HTML tables.  So all we really need to do is some simple HTML parsing and we can extract all the data we need into a hashtable, then convert it to CSV or whatever else we want.

Long story short, here's the basic script.

$outlookShortcutsHelpDocURL = 'https://support.microsoft.com/en-us/office/keyboard-shortcuts-for-outlook-3cdeb221-7ae5-4c1d-8c1d-9e63216c1efd?ui=en-US&rs=en-US&ad=US'
$windowsShortcutsElementRegEx = '<div.*?id="PickTab-supTabControlContent-1".*?>.*?</div>.*?<div.*?id="PickTab-supTabControlContent'
$shortcutTableRegEx = '<table.*?>.*?</table>'
$shortcutItemRegEx = '<tr>.*?<td>(.*?)</td>.*?<td>(.*?)</td>.*?</tr>'
$regexMatchOptions = [System.Text.RegularExpressions.RegexOptions]::Singleline + [System.Text.RegularExpressions.RegexOptions]::IgnoreCase

# ---------------------------------------------------------------------------------

$page = Invoke-WebRequest -Uri $outlookShortcutsHelpDocURL -UseBasicParsing
$windowsShortcuts = [Regex]::Matches($page.Content, $windowsShortcutsElementRegEx, $regexMatchOptions)
$shortcutTables = [Regex]::Matches($windowsShortcuts[0], $shortcutTableRegEx, $regexMatchOptions)

$shortcuts = $shortcutTables | % {
  [Regex]::Matches($_, $shortcutItemRegEx, $regexMatchOptions) | % {
     if ($_.Captures -and $_.Captures.Groups -and $_.Captures.Groups.Count -ge 3) {
        $shortcut = (($_.Captures.Groups[2].Value -ireplace '<.+?>','') -ireplace '\s+',' ').Trim()
        $description = (($_.Captures.Groups[1].Value -ireplace '<.+?>','') -ireplace '\s+',' ').Trim()
        @{$shortcut=$description}
     }
  }
}
$shortcuts

So what's going on above?  Well, above the line (the #--------- part) are some variables I'm defining to hold the URL and Regular Expressions (RegEx) that are extracting the data from the HTML.  I decided to do it this way so it's easier to edit in the future when Microsoft changes the page formatting.  The next big part is the Invoke-WebRequest.  That fetches the live version from Microsoft.  Notice the "UseBasicParsing" flag.  That tells PowerShell NOT to use the Internet Explorer parser.  IE parsing is a bit of a mess nowadays and we don't need a full parser anyway since regular expressions will be doing all the heavy lifting. 

There are three sequences of RegEx happening on the page output.  First, using $shortcutTableRegEx, we're looking for the section that contains Windows shortcuts (Outlook for Windows Desktop).  Second, we're extracting all the tables inside that section with $shortcutTableRegEx.  Then finally we are getting individual shortcut items with $shortcutItemRegEx.  Then we do a replace to remove any extra HTML or whitespace and then Trim() to remove leading or trailing whitespace.  Finally, the item pair is output into a hashtable with the @{} section.  When applied over the "for-each" loop, you get one big hashtable array, like below.

Now, while that's easy to read on the screen, it's not a great format for trying to export elsewhere, such as to CSV.  PowerShell doesn't really handle converting arrays of hashtables very well for exports.  While it unrolls nicely to the screen, it's actually a pretty complex object.  So to export it, we need to help it along a bit by extracting the data from the embedded arrays in the array we created, cast that to a PSCustomObject, then send it to the export cmdlet. 

$shortcuts | % { [PSCustomObject]@{"Shortcut"=[String]$_.Keys;"Description"=[String]$_.Values} } | Export-Csv shortcuts.csv -NoTypeInformation

You'll notice I'm actually still using a hashtable which is what is being cast to PSCustomObject.  This hashtable doesn't format to the screen as nicely (by default), but does work well for exporting.  The reason it doesn't look nice is because of how Format-Table works.  If you sent this to Format-Table, it will try to show the first column's entire width.  The columns are very wide, so it ends up just showing one column.  It does this even with the AutoSize parameter set.  We'd have to customize Format-Table's output to get it to look nice, which is too much work.  ;-) 

So anyway, since we generated a CSV now, this is what it looks like if we open the CSV in Excel.

According to unnamed sources in some reliable publications, Microsoft Systems Center Operations Manager (OpsMgr aka SCOM) is going to the cloud.  It's called Project Aquila (at least internally).  Finally.  I've been wanting this for years.  Of course Microsoft's official stance has been that System Center remains their "on-prem" (aka private cloud, aka public cloud but not managed) solution, but it was clear they were heavily pushing Azure Monitor (previously known as Log Analytics and OMS).  However, I don't think adoption has been great as a forklift replacement.  There are parts of Azure Monitor I like, but it doesn't replace OpsMgr.  They've also reimagined the product multiple times, changed its name multiple times, refactored the UX multiple times, etc.  You just don't do that with a successful new product.  And if you've used both Azure Monitor and OpsMgr, it's pretty clear why.

Both have steep learning curves, so let's get that out of the way.  One may seem easier to you than the other, but it's only because you've dedicated yourself to learning it.  That isn't a problem.  So I won't comment on Azure Monitor's learning curve except that they didn't make anything easier to help ease transition.

The problem, to me, is more the approach to monitoring.  Azure Monitor is DevOps focused, which is a good thing, but in my opinion it neglects some basic features as a result of this focus.  For example, there is very little in the way of deploying a baseline of monitors without custom scripts and lots of upkeep.  There is no concept of automatic discovery which is the best part of OpsMgr.  If you have a ton of DevOps labor resources, this is less of a problem, but it's also something OpsMgr does automatically and natively.  If you want to create a monitor that touches everything of a defined class across all entities, you can do it easily -- that's a core functionality in OpsMgr.  You can then set all the parameters and thresholds across the management group through overrides.  You do it once, and it is continuously pushed out to new resources automatically.   You can't easily do this in Azure Monitor without a lot of upkeep and auditing.  And if your monitoring parameters change, you have to manually (or by script) copy it across the organization.

On the other hand, Azure Monitor makes it insanely easy for anybody on any team or any scope to create custom monitoring to exactly suit their needs.  OpsMgr severely lacks this.  Even with the wizard driven monitors (which give you only a fraction of what is possible to monitor), scoping it to subgroups of classes is not trivial.  And creating new classes inside the management console is impossible.  The only way to get the full functionality out of it is to create a management pack, using Visual Studio Authoring Extensions or a third party like Silect.  It's not a great experience at all, debugging is very difficult, and documentation is lacking.

Visualizations and analytic intelligence is vastly superior in Azure Monitor.  Not a surprise since it's using the vast log collection to get insights.  And any new intelligence MS creates is immediately available and can retroactively provide insight.  It's the most impressive part of Azure Monitor.  You can see how this was/is a desperate need for OpsMgr because SquaredUp built their business on top of building a better OpsMgr UI on the web which MS has neglected.  They've since moved onto adding visualizations to other products, but for years their key focus was only OpsMgr.

The key difference in approach is that Azure Monitor works exclusively on log feeds.  There is no probe-based monitoring.  You can still simulate probes in some ways, such as heartbeating, by alerting on a lack of a periodic log entry being treated like a missed heartbeat, but fundamentally, it's not the same.  If you actually need a probe, you have to make it some other way and feed the output of that into the log stream.  In contrast, OpsMgr natively supports probe based monitoring, both from the agent or agentless.  Of course, on the flip side, while OpsMgr can react to logs, it doesn't aggregate logs like how Azure Monitor does with Log Analytics (we'll just ignore ACS since it's not that great and was always separate from OpsMgr anyway).  So root cause analysis of issues can be more difficult with OpsMgr.   

As you can see, they each do something different well.  My personal preference, tainted by years of working with SCOM, is that I prefer SCOM's shortcomings to Azure Monitor's shortcomings.   I feel what Azure Monitor lacks is easily filled in by other products, but the reverse isn't necessarily true.  But, all that said, why should we even have to choose?  MS makes both products, they can give us the features of both!

Aquila could be what changes this.  First of all, the rumors are that they are refactoring SCOM into containers.  This gives really terrific portability and seamless updating, making the product ready to run on the modern cloud.  Powered by Azure SQL (or replicated to/from MS-SQL) and with multiple regions and datacenters, SCOM will have HA options like it's never had before, both in the cloud and on-prem (or self-managed VMs).  Because of how easy it is to update containers, whether the container runs inside MS managed instances or if you run it on-prem or in your own cloud VM, there will be feature parity and the product will stay up-to-date.  Secondly, this brings Azure Monitor and OpsMgr closer together for integration and rapid feature enhancement.  The best of both worlds would be having the ability to, from a single pane of glass, have all your SCOM management packs, recommended and tweaked baseline monitoring, organization standards, and so forth deployed, but then on top of that have the log consumption, analytics, visualizations, and ad hoc log alerting of Azure Monitoring.  Further, since we are again talking about containers now, OpsMgr could run in any cloud, anywhere.  You can get the same monitoring experience in AWS, GCS, and so on.  So you now have a bridge connecting your cloud monitoring.  All of it rolled up to SCOM + Azure Monitor.  That would truly set the entire MS monitoring picture above and beyond the competition by leaps and bounds.  And of course finally we can't ignore it would give a better pathway for ACS migration to Log Analytics, which actually is a part of SCOM that I think is pretty terrible but it seems there are a lot of organizations that haven't made the migration yet.

So, in conclusion, I'm hoping Aquila has great success, this could be a game changer.

I installed Exchange 2016 tools on my desktop machine back in 2017. Since then numerous CUs have come out and we migrated to Office 365. Years later we upgraded the last couple Exchange servers we have to 2019 for ongoing hybrid management. This old 2016 installation is just legacy baggage at this point.

When I tried to uninstall, however, I found that I couldn't. I suppose because the management tools were on an old CU, they weren't compatible with the current state of my desktop. The error I was getting was pretty non-descript, it was the old "An Incomplete Installation Was Detected" error. Well, this was not, in fact, an incomplete installation. It was just management tools and old.

The recommended guidance was to basically do a fresh install to the latest CU and try again but that just seemed silly, so I executed the uninstall from the command line to try to get more info. This is the cached setup.exe that exists in the Exchange installation, not the one off the installation ISO (which I no longer had lying around).

MpsSvc is the Windows Firewall. Not a surprise here that it can't muck around with that, but we can just temporarily adjust the ACL using sc sdshow/sdset, like below. Make sure to record the original ACL! It might change over time if you are viewing this article years later, so do a fresh sdshow to confirm.

I copied the ACL from a more permissive service picked at random. In this case, atashost. Using sdset, the permissions were applied.

Now I ran the installer again. I got further but still failed at a step.

This time it was AppIDSvc. The ACL looked fine, but maybe it's being protected somehow in a way I don't understand. I did find an article while searching that mentioned you can remove it (and others) from the uninstall script. So I did that.

https://social.technet.microsoft.com/Forums/en-US/e3c3174f-b2f8-4dfa-9db0-f34b0453bf03/exchange-2016-installation-failed-unable-to-uninstall-or-install-again-mailbox-role-and-others

I edited ServiceControl.ps1 and removed both the signature (since we were breaking that) and the AppIDSvc from it.

Delete both:

And here we go again!

Success!

Now return the firewall service to it's original state.

Exchange still showed up for me in Programs and Features, but clicking uninstall again indicates it did get removed and this is just Windows Installer leftover stuff it will clean up on its own.

The last bit of cleanup was just removing the Exchange Server program folders. Unfortunately, this didn't work right away as files were locked.

Guessing these might be locked until reboot, I rebooted.

And that did the trick!