I finally found a way to reliably return all members of a SCOM group, including members of subgroups (nested groups). I don't know why Microsoft made this so difficult. Anyway, other online solutions suggest using the GetRelatedMonitoringObjects() method on the group, but it was unreliable for and didn't work for all object types. The method below seems to work for everything. The main difference is that this PowerShell function recursively enumerates groups, but the trick is how do you reliably tell if a class is a group? Well, thankfully, you can just pipe the class instance into Get-SCOMGroup and if it returns $null, it's not a group! This function lets you enumerate groups by DisplayName or Class Instance object (from Get-SCOMGroup).
Function Get-SCOMGroupMembers($group) {
if ($group.GetType() -eq "".GetType()) {
$group = Get-SCOMGroup -DisplayName $group
}
$group | Get-SCOMClassInstance | % {
if (($_ | Get-SCOMGroup) -ne $null) {
Get-SCOMGroupMembers $_
} else {
$_
}
}
}