Conditional Access: Outcomes, Not Verdicts

I had a Conditional Access policy sitting in report-only in my tenant and exactly one question about it: if I turn this on, who stops being able to work?

The What If tool answers a different question. I gave it a user, an application and a device state, and it handed back thirteen rows, one per policy, each saying whether that policy matched. Nine enabled, four report-only. The answer I wanted was in there somewhere, and getting to it meant reading down the list, mentally discarding the report-only ones, collecting the grant controls off everything that was left, working out which of them the user could actually produce, and then doing the whole thing again for the next persona, and the one after that.

I did that by hand twice. The third time I wrote a module instead.

It is called CaOutcome, it lives in Powershell/Entra/CaOutcome in the companion repo, and it turns a What If response into the three things an administrator actually wants to know: does the sign-in succeed, what does the user have to do to make it succeed, and what changes if the policy being piloted goes live.

One response, two worlds

The trick that makes the promotion question answerable at all is that the response already carries each policy's state next to its verdict. Filter to enabled and you have what the tenant enforces right now. Add enabledForReportingButNotEnforced and you have what it would enforce if every report-only policy were switched on. Same response, two folds, no extra API calls.

That matters more than it sounds, because there is no way to ask Graph to evaluate a hypothetical policy. The request body takes a sign-in to simulate, not a policy set, so evaluation is always against what is really in the tenant. Staging a candidate as report-only and reading both worlds out of one response is the way to simulate a promotion without enforcing it on anybody.

$body = @{
    signInIdentity      = @{ '@odata.type' = '#microsoft.graph.userSignIn'; userId = $userId }
    signInContext       = @{ '@odata.type' = '#microsoft.graph.applicationContext'
                             includeApplications = @('00000003-0000-0ff1-ce00-000000000000') }
    signInConditions    = @{ devicePlatform = 'windows'; clientAppType = 'browser'
                             deviceInfo = @{ isCompliant = $false } }
    appliedPoliciesOnly = $false
}

$response = Invoke-MgGraphRequest -Method POST -OutputType Json `
    -Uri 'https://graph.microsoft.com/beta/identity/conditionalAccess/evaluate' `
    -Body ($body | ConvertTo-Json -Depth 10)

ConvertTo-CaOutcome -WhatIfResult $response -SignInCondition $body.signInConditions

Scenario           :
Current            : GrantedWithControls, requires authenticationStrength:Passwordless MFA
Projected          : GrantedWithControls, requires compliantDevice as well
Delta.Summary      : LOCKS OUT this sign-in; cannot satisfy GRANT - Compliant Windows
                     Devices (compliantDevice); now requires compliantDevice
ReportOnlyApplying : 1

appliedPoliciesOnly has to be $false. Without the non-applying policies, a report-only policy that does not apply looks identical to one that was never returned, and the projection quietly stops meaning anything. ReportOnlyApplying is surfaced on the output for the same reason: a zero there means the two worlds are identical by construction, not because the promotion is safe. It is the number that decides whether the rest of the output is worth reading.

It takes Maester's output directly too, since Test-MtConditionalAccessWhatIf -AllResults hands back the same collection already unwrapped:

Test-MtConditionalAccessWhatIf -UserId $u -IncludeApplications $app -AllResults |
    ConvertTo-CaOutcome

The input can be a JSON string, the OData envelope with its value property, or an already-unwrapped collection. Whichever shape it arrives in, it gets normalised on the way in, because in practice the response comes from three or four different places and none of them agree.

What comes out

The fold follows Entra's documented evaluation. Every matching policy is evaluated and the aggregate is the most restrictive combination, so one applying block ends it, and otherwise every policy's grant clause has to be satisfied.

Two design choices in that fold are worth naming because they are the ones a simpler implementation gets wrong.

OR clauses are not flattened. "MFA or compliant device" and "MFA and compliant device" are different policies, and a flat list of control names cannot tell them apart. Flatten them and every multi-option grant clause reads as a set of mandatory requirements, which overstates what the user has to do and produces lockout findings that are not real.

Authentication strengths count as requirements. A modern MFA policy has an empty builtInControls array with its entire requirement sitting in authenticationStrength. A reader that looks only at the array reports the tenant's main MFA policy as requiring nothing at all. This is not an edge case, it is the default shape of a current policy, and it is the single most likely thing to make a hand-rolled check silently useless.

Applies is not the same as satisfiable

This is the part I got wrong in the first version, and it is the reason -SignInCondition exists.

A report-only policy requiring a compliant device applies to a non-compliant device exactly as it applies to a compliant one, and the API reports it as applying either way. Read the response on its own and that shows up as a mild extra requirement. It is a lockout. Access reads GrantedWithControls in both worlds while the user is on the wrong side of the door in one of them, and the promotion diff says "now requires compliantDevice" as though the user could go and get one.

Hand the module the conditions you sent and it decides that question wherever a documented Microsoft constraint lets it:

Every row rests on something Microsoft documents, never on inference. There is one inference that is tempting and deliberately not made: a strength allowing only windowsHelloForBusiness looks unsatisfiable on iOS, and it is not, because that combination is documented as Windows Hello for Business or platform credential and now covers macOS Platform SSO. The name does not name a platform. No allowedCombinations value carries a documented platform restriction, so none is asserted, and there is a test pinning that this stays Unknown if somebody (me, in six months) decides it looks obvious.

Three restraints go with the table:

• A blocking rule beats a satisfying one. A compliant, hybrid joined device reached over Exchange ActiveSync still cannot satisfy compliantDevice, because the client type never passes the device state regardless of how well managed the device is.
• easSupported is not treated as legacy. It names the Exchange ActiveSync clients that do support modern authentication, which is exactly why Microsoft keeps it as a separate value.
• Unknown is the default and the common answer. Whether a user has registered a method or accepted terms of use is not in the request. compliantApplication on Windows is left unknown because app protection there is in preview for Edge and nothing in the request names the browser. A lockout this module fails to spot is a finding you do not get. An invented one teaches you to ignore the field, which is worse, and permanently.

An OR clause is only unsatisfiable when every alternative is. Each finding carries Reasons alongside Blockers, so the summary reads "cannot satisfy BLOCK - Corp devices only - a legacy authentication client passes no device state" rather than naming a control and leaving you to work out why it is a problem.

Session controls, and the disagreements between them

Session controls needed more care than I expected, for a reason that has nothing to do with Conditional Access semantics and everything to do with the wire format. Graph returns sessionControls fully populated with nulls. A policy that sets nothing but a persistent browser session still comes back carrying nine other properties set to null. Count those and every policy reports as configuring everything.

So the extraction keeps only what a policy really asserts, and it distinguishes two kinds of off. A null property is a control the policy never configured. A property present with isEnabled false is a control somebody configured and then switched off. Both are dropped, but only the second was a deliberate act, and dropping it is what stops a disabled sign-in frequency from appearing to compete with a live one set by another policy.

Two policies setting the same control to different values is common, easy to create by accident, and only sometimes resolvable. Microsoft documents the aggregate as most restrictive, which is applied for the two controls where restrictiveness has a defensible ordering: sign-in frequency, where a shorter interval is stricter and every time is strictest, and persistent browser, where never is stricter than always. Everything else gets no ranking, and an unranked disagreement comes back marked Resolved = $false rather than guessed at.

Each surviving control is rendered to a canonical string so two policies asserting the same thing compare equal by value and a conflict is detectable without walking object graphs, with the raw object carried alongside for anything that needs the detail.

Personas, not people

One sign-in is rarely the question. Conditional Access is a per-population problem, and the interesting failures live in combinations nobody thought to check by hand: the contractor on an unmanaged Mac, the break glass account from an unusual country, the service desk on a mobile client. Writing those out one at a time is how they get skipped.

Declare the axes as data and multiply them out:

$matrix = Import-PowerShellDataFile .\ca-matrix.psd1
$outcomes = Expand-CaScenario -Matrix $matrix | Invoke-CaScenarioMatrix -DelayMillisecond 100

# Who stops getting in if the pilot goes live
$outcomes | Where-Object { $_.Delta.BecomesEffectivelyBlocked } |
    Select-Object Scenario, @{n='Why';e={$_.Delta.Summary}}

The matrix itself is a .psd1 (Examples/ca-matrix.psd1 is a worked example), and holding it as data rather than code is deliberate: it can be reviewed and extended by someone who does not write PowerShell, it diffs cleanly next to the baseline it produces, and it is the same artefact whether it is driving an ad hoc check or a nightly run.

Conditions = @(
    @{ Name = 'managed-windows'
       DevicePlatform = 'windows'; ClientAppType = 'browser'
       SignInRiskLevel = 'low'; UserRiskLevel = 'low'; Country = 'US'
       DeviceInfo = @{ isCompliant = $true; trustType = 'azureAD' } }

    # Device code flow, which cannot pass device state to the device doing the
    # authentication. Note the deliberately compliant, hybrid joined device: the flow
    # overrides it.
    @{ Name = 'device-code-flow'
       DevicePlatform = 'windows'; ClientAppType = 'mobileAppsAndDesktopClients'
       AuthenticationFlow = @{ transferMethod = 'deviceCodeFlow' }
       DeviceInfo = @{ isCompliant = $true; trustType = 'serverAD' } }
)

Conditions are PascalCase in the file and camelCase on the wire, and Expand-CaScenario just lowercases the first letter. Unrecognised keys are passed through rather than rejected, so a property Microsoft adds to signInConditions works the day it ships instead of waiting for me to notice.

Every axis multiplies and every scenario is one API call, so Expand-CaScenario throws past -MaxScenarioCount (250 by default) rather than truncating. That is not politeness about API quota. A silently shortened matrix reports a clean run over a fraction of what you asked for, which is the worst available failure mode for a tool whose entire value is that its green means something.

The same logic runs through the failure handling. Throttling and transient failures are retried with exponential backoff, and where the server sends Retry-After its number beats anything I would invent. A scenario that fails anyway comes back with Failed set rather than vanishing from the pipeline, because a dropped scenario looks exactly like a removed one to the next baseline comparison.

Baselining outcomes rather than configuration

Microsoft365DSC and every policy export tool watch the policy document, and they tell you when its JSON changes. A Conditional Access outcome depends on a good deal more than that document: group membership, role assignment, named locations, the compliance state of a device. Somebody joins a group, and a policy that already required a compliant device now applies to them. No policy changed. Nothing drifted. A user is locked out on Monday who was not on Friday, and no config comparison tool will ever mention it.

A committed baseline of outcomes catches that, because it records what the tenant does rather than what it is configured to do.

# Once, after reviewing the outcomes
$outcomes | Export-CaBaseline -Path .\ca-baseline.json

# Nightly
Expand-CaScenario -Matrix $matrix | Invoke-CaScenarioMatrix |
    Compare-CaBaseline -Path .\ca-baseline.json | Where-Object HasChange

Each row carries Status (Unchanged, Changed, Added, Missing or Failed) plus a CurrentDelta and a ProjectedDelta. The two worlds stay apart because they answer different questions. Current drift means what the tenant enforces has moved. Projected drift means the pilot's blast radius has moved, which happens without anyone touching the pilot, because the population a report-only policy would hit is not fixed.

Three properties make the file worth committing:

• Deterministic. Arrays are sorted and no timestamp is written. A generated-on field would produce a diff on every run whether or not anything moved, which trains everyone to stop reading the diff. Git already records when the file changed and who changed it.
• Complete or refused. Export-CaBaseline throws if any scenario failed, and -Force omits the failure with a warning. Recording a failed scenario as absent would make the next comparison report it as removed: a change invented by a transient HTTP error, in the one artefact whose entire value is that its changes are real.
• Missing is not Removed. A scenario in the baseline with no fresh outcome is reported as a gap in the run, because the usual cause is a failed evaluation rather than a deliberate edit to the matrix, and those two have to look different.

Authentication strengths get watched too

A custom authentication strength is an editable tenant object, and editing one changes what every policy referencing it requires, without anyone editing a policy. The requirement still reads authenticationStrength:Contoso strong on both sides, so comparing requirement names sees nothing at all.

One caveat here I measured rather than assumed, and got wrong the first time I wrote it down. Graph inlines the whole strength object inside each referencing policy, allowedCombinations and modifiedDateTime included, so a policy export is not byte identical after the edit. An earlier version of my own README claimed it was. What a config export actually shows is N policies whose embedded blob moved. What this shows is one strength weakened, which combination was added, and which personas it reaches.

So the strength's allowedCombinations and its combination configuration count travel with the control and into the baseline, and widening one is reported as a weakening:

jeff/managed  Changed  enforced now: WEAKENED authentication strength
                       'Contoso strong' now also allows password,sms

with AccessChanged false, no added required controls and no added policies. The strength edit is the only signal. Narrowing one is reported the other way, as is dropping a FIDO2 AAGUID allowlist or a certificate issuer restriction, both of which widen what satisfies the strength without changing a single combination.

How it is put together

The module declares no RequiredModules, which looks wrong for something that sends Graph requests. It is deliberate. Invoke-CaScenarioMatrix takes a -RequestHandler scriptblock, and its default handler looks for Invoke-MgGraphRequest at run time and says so plainly if it is absent:

if (-not (Get-Command -Name 'Invoke-MgGraphRequest' -ErrorAction SilentlyContinue)) {
    throw ('Invoke-MgGraphRequest was not found. Install and import ' +
        'Microsoft.Graph.Authentication and run Connect-MgGraph, or pass your own ' +
        '-RequestHandler.')
}

Declaring the Graph SDK in the manifest would make the whole module unimportable, and its tests unrunnable, on a host that only ever needed the folding. Every function other than the matrix runner is a pure transform over a response somebody else fetched. That seam is also how you replay recorded responses, route through your own transport, or test the matrix and the assertions over it without a tenant at all.

The suite is 210 tests at 98.7% coverage. 204 of them need no tenant and run anywhere. The six under Tests/Integration exercise the one path a fixture cannot, the default request handler that calls Invoke-MgGraphRequest, and skip themselves unless the session is connected to Graph with a Conditional Access read scope.

The fixtures are genuine What If responses from a tenant carrying nine enabled and four report-only policies, with directory object ids rewritten to synthetic ones. Microsoft's own well-known ids are left alone, because rewriting them would make the fixture describe a tenant that cannot exist. Keeping them real mattered more here than anywhere else I have written tests: every shape that caused a defect during development came from the API rather than from my imagination. grantControls with an empty builtInControls array. sessionControls padded with nulls. Two policies disagreeing about persistent browser on the same sign-in. I would not have invented any of those.

Where this sits next to what already exists

Nothing here replaces the tools already in this space. It fills a gap between them.

Entra's own What If tool is where everybody starts, and it is genuinely good at what it does: one sign-in, one screen, a verdict per policy. What it does not do is aggregate, project a promotion, or run over a population, and it has no output you can commit to a repository and diff next week.

Maester's built-in Conditional Access tests assert against the policies enforced today, which is the right thing for them to assert. Its What If support is a cmdlet you build custom tests on, and the documented pattern is roughly $result.grantControls.builtInControls | Should -Contain "mfa". That works, and it has two blind spots this module exists to cover. It reads across whatever policies came back without separating enforced from report-only. And it finds nothing at all in a policy whose requirement lives in authenticationStrength with an empty builtInControls array, which for most tenants is the main MFA policy.

Jasper Baes has been the most useful voice in this area for a couple of years, and his tools sit on a different axis rather than a competing one. The Conditional Access Validator generates Maester tests automatically from your current policy configuration, one per policy, based on that policy's own configured properties, and wraps them in an HTML report with a flow chart and a persona view. The Conditional Access Impact Matrix answers which policies apply to which users. Both start from the policy and ask who it hits, which is the right way round for building desired state tests you did not want to hand write.

This starts from the sign-in and asks what it meets. The Validator's own documented limitations are the tell here: device properties and session controls are on its roadmap rather than in it, and those are precisely the inputs that decide whether a requirement can be satisfied and whether two policies disagree. If you are already running the Validator you have per policy desired state generated for you, which is a real amount of work you no longer have to do. What you do not have is the aggregate: what one persona experiences once thirteen policies have all had their say, and what changes when four of them get promoted.

The two compose. Generate your per policy tests with the Validator, and add a handful of outcome assertions over the personas you care about.

The Maester test

There is a drop-in custom test in Examples/ContosoCaOutcome.Tests.ps1.template. Copy it into Maester's Custom folder, drop the .template extension, rename Contoso to your organisation, and Invoke-Maester discovers it with everything else. Findings land in the existing Maester HTML report and GitHub Action, with no fork and no separate reporting to maintain.

It asserts three things the built-in tests do not: that promoting the report-only policies would lock no persona out, that the break glass account survives both worlds, and, the one that stops the other two passing vacuously, that a report-only policy applied to at least one persona in the first place.

That last one is the assertion I would most encourage you to steal even if you use none of the rest. The promotion tests compare the enforced policy set against the same set plus the report-only policies. The moment the last piloted policy is promoted or deleted, those two sets are identical and every promotion test goes green forever, for a reason that has nothing to do with safety.

A few things I learned wiring it up, which the documentation does not say out loud. Maester test names are "ID: Title" and it warns on every name without one. Findings need to go through Add-MtTestResultDetail rather than Write-Host, and the severity passed there is what fills the Severity column in the report; a Severity: tag alone shows up in the test detail and leaves that column empty. And the baseline path has to be set at file scope as well as in BeforeAll, because Pester runs a file twice and a -Skip: expression is evaluated during discovery, where anything assigned in BeforeAll does not exist yet.

The Mt prefix is Maester's, so these cmdlets deliberately do not use it.

One correction in that file is worth repeating because it was a real finding rather than a tidy-up. The session conflict test originally looked only at Current, and reported a clean bill of health against a tenant that had a genuine unresolved cloudAppSecurity disagreement, because the two policies causing it were report-only and the conflict lived only in Projected. Checking the enforced world alone misses every conflict that promoting a pilot would introduce, which is the one thing this module exists to see coming.

Two things to know before you run it

The API is beta. POST /beta/identity/conditionalAccess/evaluate may change, and Maester's own documentation carries the same warning: tests written against this API may need updating as it moves toward v1.0. Everything read here is documented for whatIfAnalysisResult, but a beta response is not a contract. That is why the transport lives in a single private function and why the request handler is a parameter. An API shape change costs you one file, not a rewrite, and that is the most honest guarantee I can offer about something built on a preview endpoint.

The matrix is meant to be edited. The shipped ca-matrix.psd1 and the Maester template that reads it are wired to each other. The template's $ShouldAllow list names conditions from that file, and the personas are synthetic ids from my lab tenant. Clone it, run it unchanged against your tenant, and you get a clean-looking run over three users who do not exist. Replace the persona ids with the shapes of user you genuinely care about, then reconcile $ShouldAllow with whatever you named your conditions. That reconciliation is the step people skip, and skipping it is how you end up with assertions that pass because they match nothing.

The $ShouldAllow split is doing real work, incidentally. "Nobody is locked out" is the wrong assertion over a matrix that deliberately includes legacy authentication and a high risk foreign sign-in, because those rows are lockouts by design and a test that flags them trains you to ignore the result. So paths that are supposed to work are asserted to work, and paths that are supposed to be shut are asserted to be shut. Deciding which is which is your job, and it is the only part of this that cannot be automated.

The limitation that matters most

A baseline is only as good as its matrix. Nothing in this module discovers the personas worth checking. A population left out of the matrix is a population no assertion covers, and the run still reports green, in exactly the same confident tone it uses when it has checked everything.

That is not a defect I can fix with more code. It is the reason the matrix is a reviewable data file sitting next to the tests rather than something buried in a script, and the reason it is the first thing this post tells you to edit.

Related reading

• Using an MCP agent to query Entra with natural language.
• Using Entra to log into an application without SAML/OAUTH/OIDC
• Setting up SSO with Entra and Proxmox (OIDC)