Conditional Access: Outcomes, Not Verdicts

I had a Conditional Access policy sitting in report-only in my tenant and exactly one question about it: if I turn this on, who stops being able to work?

The What If tool answers a different question. I gave it a user, an application and a device state, and it handed back thirteen rows, one per policy, each saying whether that policy matched. Nine enabled, four report-only. The answer I wanted was in there somewhere, and getting to it meant reading down the list, mentally discarding the report-only ones, collecting the grant controls off everything that was left, working out which of them the user could produce, and then doing the whole thing again for the next persona, and the one after that.

I did that by hand twice. The third time I wrote a module instead.

It is called CaOutcome, it lives in Powershell/Entra/CaOutcome in the companion repo, and it turns a What If response into what an administrator wants to know: does the sign-in succeed, what does the user have to do to make it succeed, and what changes if the policy being piloted goes live.

One response, two worlds

The promotion question is answerable at all because the response already carries each policy’s state next to its verdict. Filter to enabled and you have what the tenant enforces right now. Add enabledForReportingButNotEnforced and you have what it would enforce if every report-only policy were switched on. Same response, two folds, no extra API calls.

There is no way to ask Graph to evaluate a hypothetical policy. The request body takes a sign-in to simulate, not a policy set, so evaluation is always against what is really in the tenant. Staging a candidate as report-only and reading both worlds out of one response is how you simulate a promotion without enforcing it on anybody.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

$body = @{
    signInIdentity      = @{ '@odata.type' = '#microsoft.graph.userSignIn'; userId = $userId }
    signInContext       = @{ '@odata.type' = '#microsoft.graph.applicationContext'
                             includeApplications = @('00000003-0000-0ff1-ce00-000000000000') }
    signInConditions    = @{ devicePlatform = 'windows'; clientAppType = 'browser'
                             deviceInfo = @{ isCompliant = $false } }
    appliedPoliciesOnly = $false
}

$response = Invoke-MgGraphRequest -Method POST -OutputType Json `
    -Uri 'https://graph.microsoft.com/beta/identity/conditionalAccess/evaluate' `
    -Body ($body | ConvertTo-Json -Depth 10)

ConvertTo-CaOutcome -WhatIfResult $response -SignInCondition $body.signInConditions

1
2
3
4
5
6

Scenario           :
Current            : GrantedWithControls, requires authenticationStrength:Passwordless MFA
Projected          : GrantedWithControls, requires compliantDevice as well
Delta.Summary      : LOCKS OUT this sign-in; cannot satisfy GRANT - Compliant Windows
                     Devices (compliantDevice); now requires compliantDevice
ReportOnlyApplying : 1

appliedPoliciesOnly has to be $false. Without the non-applying policies, a report-only policy that does not apply looks identical to one that was never returned, and the projection stops meaning anything. ReportOnlyApplying is surfaced on the output for the same reason: a zero there means the two worlds are identical by construction, so a clean delta says nothing at all about whether the promotion is safe. It is the number that decides how much the rest of the output is telling you.

It takes Maester’s output directly too, since Test-MtConditionalAccessWhatIf -AllResults hands back the same collection already unwrapped:

1
2

Test-MtConditionalAccessWhatIf -UserId $u -IncludeApplications $app -AllResults |
    ConvertTo-CaOutcome

The input can be a JSON string, the OData envelope with its value property, or an already-unwrapped collection. Whichever shape it arrives in, it gets normalised on the way in, because in practice the response comes from three or four different places and none of them agree.

What comes out

The fold follows Entra’s documented evaluation. Every matching policy is evaluated and the aggregate is the most restrictive combination, so one applying block ends it, and otherwise every policy’s grant clause has to be satisfied.

OR clauses are kept whole. “MFA or compliant device” and “MFA and compliant device” are different policies, and a flat list of control names cannot tell them apart. Flatten them and every multi-option grant clause reads as a set of mandatory requirements, which overstates what the user has to do and produces lockout findings that are not real.

Authentication strengths count as requirements. A modern MFA policy has an empty builtInControls array with its entire requirement sitting in authenticationStrength. A reader that looks only at the array reports the tenant’s main MFA policy as requiring nothing at all. That is the default shape of a current policy, and it is where a hand-rolled check most often goes wrong.

Applies is not the same as satisfiable

I had this wrong in the first version, and -SignInCondition exists because of it.

A report-only policy requiring a compliant device applies to a non-compliant device exactly as it applies to a compliant one, and the API reports it as applying either way. Read the response on its own and that shows up as a mild extra requirement. It is a lockout. Access reads GrantedWithControls in both worlds while the user is on the wrong side of the door in one of them, and the promotion diff says “now requires compliantDevice” as though the user could go and get one.

Hand the module the conditions you sent and it decides that question wherever a documented Microsoft constraint lets it:

Every row rests on something Microsoft documents, never on inference. One inference is tempting and deliberately not made: a strength allowing only windowsHelloForBusiness looks unsatisfiable on iOS. It is satisfiable, because that combination is documented as Windows Hello for Business or platform credential and now covers macOS Platform SSO. The name carries no platform. No allowedCombinations value carries a documented platform restriction, so none is asserted, and a test pins the answer to Unknown in case somebody (me, in six months) decides it looks obvious.

The table comes with restraints:

• A blocking rule beats a satisfying one. A compliant, hybrid joined device reached over Exchange ActiveSync still cannot satisfy compliantDevice, because the client type never passes the device state however well managed the device is.
• easSupported names the Exchange ActiveSync clients that do support modern authentication. Microsoft keeps it as a separate value for that reason, and the satisfiability rules treat it separately too.
• Unknown is the default and the common answer. Whether a user has registered a method or accepted terms of use is not in the request. compliantApplication on Windows stays unknown because app protection there is in preview for Edge and nothing in the request names the browser. A lockout this module fails to spot is a finding you do not get. An invented one teaches you to ignore the field.

An OR clause is only unsatisfiable when every alternative is. Each finding carries Reasons alongside Blockers, so the summary names the reason with the control: “cannot satisfy BLOCK - Corp devices only - a legacy authentication client passes no device state”.

Session controls arrive full of nulls

Session controls needed more care than I expected, for a reason that has nothing to do with Conditional Access semantics and everything to do with the wire format. Graph returns sessionControls fully populated with nulls. A policy that sets nothing but a persistent browser session still comes back carrying nine other properties set to null. Count those and every policy reports as configuring everything.

The extraction therefore keeps only what a policy asserts, and it distinguishes two kinds of off. A null property is a control the policy never configured. A property present with isEnabled false is a control somebody configured and then switched off. Both are dropped, but only the second was a deliberate act, and dropping it stops a disabled sign-in frequency from appearing to compete with a live one set by another policy.

Two policies setting the same control to different values is common, easy to create by accident, and only sometimes resolvable. Microsoft documents the aggregate as most restrictive, which is applied for the two controls where restrictiveness has a defensible ordering: sign-in frequency, where a shorter interval is stricter and every time is strictest, and persistent browser, where never is stricter than always. Everything else gets no ranking, and an unranked disagreement comes back marked Resolved = $false with both values carried.

Each surviving control is rendered to a canonical string so two policies asserting the same thing compare equal by value and a conflict is detectable without walking object graphs, with the raw object carried alongside for anything that needs the detail.

Personas, not people

One sign-in is rarely the question. Conditional Access is a per-population problem, and the interesting failures live in combinations nobody thought to check by hand: the contractor on an unmanaged Mac, the break glass account from an unusual country, the service desk on a mobile client. Writing those out one at a time is how they get skipped.

Declare the axes as data and multiply them out:

1
2
3
4
5
6

$matrix = Import-PowerShellDataFile .\ca-matrix.psd1
$outcomes = Expand-CaScenario -Matrix $matrix | Invoke-CaScenarioMatrix -DelayMillisecond 100

# Who stops getting in if the pilot goes live
$outcomes | Where-Object { $_.Delta.BecomesEffectivelyBlocked } |
    Select-Object Scenario, @{n='Why';e={$_.Delta.Summary}}

The matrix is a .psd1 (Examples/ca-matrix.psd1 is a worked example), and holding it as data is a deliberate choice: it can be reviewed and extended by someone who does not write PowerShell, it diffs cleanly next to the baseline it produces, and it is the same artefact whether it is driving an ad hoc check or a nightly run.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

Conditions = @(
    @{ Name = 'managed-windows'
       DevicePlatform = 'windows'; ClientAppType = 'browser'
       SignInRiskLevel = 'low'; UserRiskLevel = 'low'; Country = 'US'
       DeviceInfo = @{ isCompliant = $true; trustType = 'azureAD' } }

    # Device code flow, which cannot pass device state to the device doing the
    # authentication - so any policy requiring a compliant or hybrid joined device is
    # unsatisfiable here no matter how well managed the device is. Note the deliberately
    # compliant, hybrid joined device below: the flow overrides it.
    @{ Name = 'device-code-flow'
       DevicePlatform = 'windows'; ClientAppType = 'mobileAppsAndDesktopClients'
       SignInRiskLevel = 'low'; UserRiskLevel = 'low'; Country = 'US'
       AuthenticationFlow = @{ transferMethod = 'deviceCodeFlow' }
       DeviceInfo = @{ isCompliant = $true; trustType = 'serverAD' } }
)

Conditions are PascalCase in the file and camelCase on the wire, and Expand-CaScenario lowercases the first letter. Unrecognised keys are passed through, so a property Microsoft adds to signInConditions works the day it ships.

Every axis multiplies and every scenario is one API call, so Expand-CaScenario throws past -MaxScenarioCount (250 by default) instead of truncating. The cap is a parameter, so widening it stays an explicit decision somebody made.

The same reasoning runs through the failure handling. Throttling and transient failures are retried with exponential backoff, and where the server sends Retry-After its number beats anything I would invent. A scenario that fails anyway comes back with Failed set, because a scenario that vanished from the pipeline would read to the next baseline comparison as one somebody deleted from the matrix.

Baselining outcomes

Microsoft365DSC and every policy export tool watch the policy document, and they tell you when its JSON changes. A Conditional Access outcome depends on a good deal more: group membership, role assignment, named locations, the compliance state of a device. Somebody joins a group, and a policy that already required a compliant device now applies to them. No policy changed. A user is locked out on Monday who was not on Friday, and no config comparison tool will ever mention it.

A committed baseline of outcomes catches that, because it records what the tenant does.

1
2
3
4
5
6

# Once, after reviewing the outcomes
$outcomes | Export-CaBaseline -Path .\ca-baseline.json

# Nightly
Expand-CaScenario -Matrix $matrix | Invoke-CaScenarioMatrix |
    Compare-CaBaseline -Path .\ca-baseline.json | Where-Object HasChange

Each row carries Status (Unchanged, Changed, Added, Missing or Failed) plus a CurrentDelta and a ProjectedDelta. The two worlds stay apart because they answer different questions. Current drift means what the tenant enforces has moved. Projected drift means the pilot’s blast radius has moved, which happens without anyone touching the pilot, because the population a report-only policy would hit is not fixed.

The format is built for git:

• Deterministic. Arrays are sorted and no timestamp is written. A generated-on field would produce a diff on every run whether or not anything moved, and a diff that always shows something is a diff nobody reads. Git already records when the file changed and who changed it.
• Complete or refused. Export-CaBaseline throws if any scenario failed, and -Force omits the failure with a warning. Recording a failed scenario as absent would make the next comparison report it as removed: a change invented by a transient HTTP error, inside the one artefact whose changes are supposed to be real.
• Missing is not Removed. A scenario in the baseline with no fresh outcome is reported as a gap in the run, because the usual cause is a failed evaluation, and a deliberate edit to the matrix has to look different.

Authentication strengths get watched too

A custom authentication strength is an editable tenant object, and editing one changes what every policy referencing it requires, without anyone editing a policy. The requirement still reads authenticationStrength:Contoso strong on both sides, so comparing requirement names sees nothing at all.

One caveat I measured after getting it wrong on paper. Graph inlines the whole strength object inside each referencing policy, allowedCombinations and modifiedDateTime included, so a policy export is not byte identical after the edit. An earlier version of my own README claimed it was. A config export shows N policies whose embedded blob moved. This shows one strength weakened, which combination was added, and which personas it reaches.

The strength’s allowedCombinations and its combination configuration therefore travel with the control and into the baseline, and widening one is reported as a weakening:

1
2

jeff/managed  Changed  enforced now: WEAKENED authentication strength
                       'Contoso strong' now also allows password,sms

with AccessChanged false, no added required controls and no added policies. The strength edit is the only signal. Narrowing one is reported the other way, as is dropping a FIDO2 AAGUID allowlist or a certificate issuer restriction, both of which widen what satisfies the strength without changing a single combination.

How it is put together

The module declares no RequiredModules, which looks wrong for something that sends Graph requests. Invoke-CaScenarioMatrix takes a -RequestHandler scriptblock, and its default handler looks for Invoke-MgGraphRequest at run time and says so plainly if it is absent:

1
2
3
4
5

if (-not (Get-Command -Name 'Invoke-MgGraphRequest' -ErrorAction SilentlyContinue)) {
    throw ('Invoke-MgGraphRequest was not found. Install and import ' +
        'Microsoft.Graph.Authentication and run Connect-MgGraph, or pass your own ' +
        '-RequestHandler.')
}

Declaring the Graph SDK in the manifest would make the whole module unimportable, and its tests unrunnable, on a host that only ever needed the folding. Every function other than the matrix runner is a pure transform over a response somebody else fetched. That seam is also how you replay recorded responses, route through your own transport, or test the matrix and the assertions over it without a tenant at all.

The suite is 210 tests at 98.7% coverage. 204 of them need no tenant and run anywhere. The six under Tests/Integration exercise the one path a fixture cannot, the default request handler that calls Invoke-MgGraphRequest, and skip themselves unless the session is connected to Graph with a Conditional Access read scope.

The fixtures are genuine What If responses from a tenant carrying nine enabled and four report-only policies, with directory object ids rewritten to synthetic ones. Microsoft’s own well-known ids are left alone, because rewriting them would make the fixture describe a tenant that cannot exist. Every shape that caused a defect during development came from the API: grantControls with an empty builtInControls array, sessionControls padded with nulls, two policies disagreeing about persistent browser on the same sign-in. I would not have invented any of those.

Where this sits next to what already exists

This fills a gap between tools that already exist.

Entra’s own What If tool is where everybody starts, and it does that job well: one sign-in, one screen, a verdict per policy. It does not aggregate, project a promotion, or run over a population, and it has no output you can commit to a repository and diff next week.

Maester’s built-in Conditional Access tests assert against the policies enforced today, which is the right scope for them. Its What If support is a cmdlet you build custom tests on, and the documented pattern is roughly $result.grantControls.builtInControls | Should -Contain "mfa". That works. It also reads across whatever policies came back without separating enforced from report-only, and it finds nothing at all in a policy whose requirement lives in authenticationStrength with an empty builtInControls array, which for most tenants is the main MFA policy.

Jasper Baes has been the most useful voice in this area for a couple of years, and his tools sit on a different axis. The Conditional Access Validator generates Maester tests automatically from your current policy configuration, one per policy, based on that policy’s own configured properties, and wraps them in an HTML report with a flow chart and a persona view. The Conditional Access Impact Matrix answers which policies apply to which users. Both start from the policy and ask who it hits, which is the right way round for building desired state tests you did not want to hand write.

This starts from the sign-in and asks what it meets. The Validator’s own documented limitations are the tell: device properties and session controls sit on its roadmap, and those are the inputs that decide whether a requirement can be satisfied and whether two policies disagree. If you are already running the Validator you have per policy desired state generated for you, which is a real amount of work you no longer have to do. What you do not have is the aggregate: what one persona experiences once thirteen policies have all had their say, and what changes when four of them get promoted.

The two compose. Generate your per policy tests with the Validator, and add a handful of outcome assertions over the personas you care about.

The Maester test

There is a drop-in custom test in Examples/ContosoCaOutcome.Tests.ps1.template. Copy it into Maester’s Custom folder, drop the .template extension, rename Contoso to your organisation, and Invoke-Maester discovers it with everything else. Findings land in the existing Maester HTML report and GitHub Action, with no fork and no separate reporting to maintain.

The assertions it adds: promoting the report-only policies would lock no persona out, the break glass account survives both worlds, and a report-only policy applied to at least one persona in the first place.

That last one keeps the other two from passing vacuously, and I would steal it even if you use none of the rest. The promotion tests compare the enforced policy set against the same set plus the report-only policies. The moment the last piloted policy is promoted or deleted, those two sets are identical and every promotion test passes forever, for a reason that has nothing to do with safety.

Some of the wiring the Maester documentation does not say out loud. Test names are “ID: Title” and it warns on every name without one. Findings go through Add-MtTestResultDetail, and the severity passed to that function is what fills the Severity column in the report; a Severity: tag alone shows up in the test detail and leaves the column empty. And the baseline path has to be set at file scope as well as in BeforeAll, because Pester runs a file twice and a -Skip: expression is evaluated during discovery, where anything assigned in BeforeAll does not exist yet.

The Mt prefix is Maester’s, so these cmdlets do not use it.

One correction in that file came out of a real finding. The session conflict test originally looked only at Current, and passed against a tenant that had an unresolved cloudAppSecurity disagreement, because the two policies causing it were report-only and the conflict lived only in Projected. Checking the enforced world alone misses every conflict a promotion would introduce.

Before you point it at a real tenant

The API is beta. POST /beta/identity/conditionalAccess/evaluate may change, and Maester’s own documentation carries the same warning: tests written against this API may need updating as it moves toward v1.0. Everything read here is documented for whatIfAnalysisResult, but a beta response is not a contract. The transport lives in a single private function and the request handler is a parameter, so an API shape change costs one file. Nothing stronger than that is available for something built on a preview endpoint.

The matrix is meant to be edited. The shipped ca-matrix.psd1 and the Maester template that reads it are wired to each other. The template’s $ShouldAllow list names conditions from that file, and the personas are synthetic ids from my lab tenant. Clone it, run it unchanged against your tenant, and you get a clean-looking run over three users who do not exist. Replace the persona ids with the shapes of user you care about, then reconcile $ShouldAllow with whatever you named your conditions. That reconciliation is the step people skip, and skipping it produces assertions that pass because they match nothing.

The $ShouldAllow split is doing real work. “Nobody is locked out” is the wrong assertion over a matrix that deliberately includes legacy authentication and a high risk foreign sign-in, because those rows are lockouts by design and a test that flags them trains you to ignore the result. Paths that are supposed to work are asserted to work, and paths that are supposed to be shut are asserted to be shut. Deciding which is which is your job, and the only part of this that cannot be automated.

A baseline is only as good as its matrix

Nothing in this module discovers which personas to check. A population left out of the matrix is a population no assertion covers, and the run reports green in exactly the same confident tone it uses when it has checked everything.

More code does not fix that. It is why the matrix sits beside the tests as a reviewable data file, and why the quality of the answer is bounded by the quality of that file.

Related reading

• Using an MCP agent to query Entra with natural language.
• Using Entra to log into an application without SAML/OAUTH/OIDC
• Setting up SSO with Entra and Proxmox (OIDC)