The first post in this series had a warning in the middle of it. Test-OrphanedSID tried several lookups, and if every one of them fell through, it ended on this:

1
2
3
# If all methods fail, consider it orphaned
$script:SIDValidationCache[$SID] = $true
return $true  # Orphaned

“Fell through” covered two very different things. A lookup that comes back with “no such object” is evidence. A lookup that times out, gets access denied, or can’t reach a domain controller is no evidence at all. Both got the same answer, and Remove-OrphanedSid acts on that answer.

I filed that as issue #9 the night 1.0.1 shipped and left the fix for 1.1.0. Two more bugs turned up in the same function before it shipped.

I wanted to wait rather than push another version to the Gallery within a couple hours. Embarrassing for me to push 1.0.1 so quickly, I wanted to verify 1.1.0 was more coherent.

Three answers

Every lookup in the old Test-OrphanedSID had a typed catch for “not found” and a generic one for everything else, and both did the same thing: move on to the next lookup. The outer catch around the whole function said so out loud:

1
2
3
4
5
catch {
    # Assume orphaned on error for safety
    $script:SIDValidationCache[$SID] = $true
    return $true
}

For a tool that removes access, assuming orphaned is the unsafe direction. And because the answer went into the cache, one failed lookup decided that SID for every remaining object in the run.

In 1.1.0 the function returns one of three answers where it used to return a [bool]: Orphaned, Exists or Unverified. Each failure that isn’t “not found” is collected, and the decision at the end only says orphaned when nothing failed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# Nothing found the SID, but a lookup that failed might have. Unverified is not cached, so
# a transient failure does not decide the answer for every later object in the run.
if ($lookupFailures.Count -gt 0) {
    # ...
    return New-SIDLookupResult -SID $SID -Status 'Unverified' -ErrorMessage $failureText
}

# Every lookup completed and none found the SID
$script:SIDValidationCache[$SID] = New-SIDLookupResult -SID $SID -Status 'Orphaned'
return $script:SIDValidationCache[$SID]

The outer catch returns Unverified now, with a comment that says why. An invalid SID format does too.

The first post covers what that looks like in your results, and how to filter down to confirmed orphans.

Unverified results aren’t cached, so the next object that carries the same SID looks it up again. If the domain controller comes back partway through a run, later objects get a real answer.

On the lab DC

I checked the fix on the lab domain against the same two seeded orphans the first post used. The baseline gave both as Orphaned. With Get-ADUser sent to a domain controller that doesn’t exist, both came back Unverified, carrying the AD error in VerificationError. A -WhatIf preview and then a real -Force removal both left them alone and named each one in a warning. With lookups restored in the same session, both came back Orphaned again, so the uncached result did its job.

If your forest has trusts, the same run has more for you. An account denied all access to a live trustee doesn’t produce an error at all: AD returns an empty search, and Translate still resolves the SID, so the trustee isn’t reported. A trustee from a trusted domain whose DC was shut down came back Unverified after about 40 seconds per lookup. With the trusted DC running it wasn’t reported, and after the trustee was deleted it came back Orphaned with LikelySource = Foreign Domain/Trust.

A deleted-objects search that finds live objects

The second bug was in the step that looks for tombstones. If the ordinary lookup didn’t find a SID, Test-OrphanedSID searched again with -IncludeDeletedObjects, and took any match as proof the object had been deleted.

-IncludeDeletedObjects doesn’t mean “only deleted objects”. It means “deleted objects too”. On the lab DC, that search for the built-in Administrator’s SID returned CN=Administrator,CN=Users,..., alive and well.

On its own that was harmless, because the ordinary lookup would’ve found Administrator first. The trouble was the path where the ordinary lookup failed, say with a timeout that outlasted its retries, and the deleted-objects search then succeeded. A live principal came back Orphaned, got cached, and Remove-OrphanedSid would take its access away. That’s #9 again by a different route, and the #9 fix made the route easier to reach, because it treats a tombstone as decisive even when another lookup failed.

The fix asks for isDeleted and only counts a match that has it:

1
2
3
4
5
6
$matchedObjects = @(Invoke-ADOperationWithRetry -ScriptBlock {
        Get-ADObject -Filter "objectSid -eq '$SID'" -IncludeDeletedObjects -Properties isDeleted -ErrorAction Stop
    } -MaxRetries $lookupAttempts -OperationName 'Get-ADObject-Deleted-BySID' -ObjectContext $SID -CorrelationId $CorrelationId)

$liveObject = $matchedObjects | Where-Object { $_ -and $_.isDeleted -ne $true } | Select-Object -First 1
$deletedObject = $matchedObjects | Where-Object { $_ -and $_.isDeleted -eq $true } | Select-Object -First 1

A live match is Exists. A tombstone is still Orphaned, and it settles the question even if an earlier lookup failed, because a tombstone is positive evidence of deletion. On the DC, the query returns a live object with no isDeleted value at all and a tombstone with isDeleted = True, so the -ne $true check covers both.

The integration test that guards this has a principal whose ordinary lookups always time out and whose only answer comes from the deleted-objects search. Removal runs against an object where it’s the only trustee, and the test checks that its entry, its ACL and the backup folder are all unchanged.

Three and a half minutes per SID

The third problem was speed. With an unreachable domain controller, every SID still needing a lookup was correctly reported Unverified. Each one took about 3.5 minutes to get there.

The AD cmdlets talk to AD Web Services, and a connection to a DC that doesn’t answer waits about 42 seconds before giving up. Test-OrphanedSID tried five directory lookups per SID, and each one caught its own failure and moved on to the next. That one waited another 42 seconds. I measured that on the lab domain with two DCs: Remove-OrphanedSid pointed at the second DC, which was powered off right after enumeration. The log shows the lookups 42 seconds apart, five per SID, two SIDs, about seven minutes of a 678-second run. At that rate, a search base with 50 SIDs needing lookups would spend about three hours telling you it couldn’t check any of them.

Once one lookup has found the server down, the other directory lookups for that SID would fail the same way. 1.1.0 stops asking:

1
2
3
if ($serverDown) {
    $skippedLookups.Add('Get-ADObject -IncludeDeletedObjects')
} else {

It only stops for the one failure that means “nobody’s there”. Test-ADServerDownError walks the exception chain looking for ADServerDownException by type name, so it works on a test runner without RSAT:

1
2
3
4
5
6
7
8
$exception = $ErrorRecord.Exception
while ($exception) {
    if ($exception.GetType().FullName -eq 'Microsoft.ActiveDirectory.Management.ADServerDownException') {
        return $true
    }
    $exception = $exception.InnerException
}
return $false

A timeout or an access-denied error still tries every lookup, because the next one might succeed. SecurityIdentifier.Translate also still runs between the directory lookups, because it goes through LSA, a separate path from AD Web Services, and a name from it proves the SID exists. The skipped lookups are named in the result so you can see what wasn’t tried:

1
Not attempted because the directory server could not be contacted: <lookups>

Same SID, same unreachable address, timed inside the module on the lab DC:

ModuleTimeStatus
1.0.1 code path (main before the fix)211 sUnverified, five failed lookups
1.1.0, Windows PowerShell 5.142 sUnverified, one failed, four not attempted
1.1.0, PowerShell 7.642 ssame

The end-to-end run with the second DC powered off two seconds after enumeration gave the same picture: one failed lookup per SID, 42 seconds apart, both SIDs Unverified with Low confidence, nothing removed, no backup written, and Unverified SIDs (lookup failed, not removed): 2 in the summary.

One SID in that run needed a lookup after the power-off and came out fine: RAS and IAS Servers, which sits on every user object. Its directory lookup timed out, Translate resolved it, and it was correctly left alone.

Each SID now pays for one timeout instead of five, but the next SID still pays its own. A run-wide circuit breaker that stops asking a dead DC for a while, then checks whether it’s back, is the other half of issue #82, and I’m not counting the first half as done yet.

Until I get more data on the fix I pushed for it, issue #82 will remain open. Which either means feedback from others, or I get another project that uses this. It improved on my lab environment, but I don’t want to rely just on that.