- Conditional access policies can be complicated to implement and test.
- Microsoft offers a small set of tools that may be able to help design and debug conditional access policies.
- Most of these tools are freely available, but some require additional tools and costs.
Conditional access (CA) policies within Microsoft 365 are complex to create and troubleshoot. A small set of free tools are available from Microsoft to help customers troubleshoot and report on their CA policies. The options range from tools to report on use and coverage of CA to tools designed to help optimize CA policies and report on their use. While these tools are all free, at a minimum, users must be licensed (at additional costs) for Entra ID P1, Entra ID P2, or Entra ID Governance and Intune Plan 1 to take advantage of CA, and some tools also require Log Analytics or Security Copilot. This report provides an overview of these tools for security professionals responsible for designing, establishing, and maintaining CA policies within their organization, with a summary of the information each provides, use scenarios, and limitations.
Conditional Access Fundamentals
Conditional access provides the ability to impose additional security and governance prerequisites, beyond simply requiring a username and password when users connect to Entra-ID integrated applications such as Microsoft 365. For example, it can be used to ensure that even authenticated users can access services only from Intune-managed devices. CA is a process that occurs after the user’s credentials are provided to Entra ID, but before that user is granted access to the requested online service or application. CA becomes increasingly important as cloud- and mobile-based computing renders traditional perimeter access security methods, such as corporate firewalls, insufficient absent supplementation with additional protections.
Conditional Access Tools
A range of tools are available from Microsoft to help customers troubleshoot and report on CA policies. These tools include capabilities for reporting, troubleshooting, and review of raw logs that can help when CA policies are not working as expected.
These tools can each fill a specific gap for CA creation and troubleshooting, but none of them offers end-to-end coverage from design to implementation and post-deployment troubleshooting. Customers will have to select tools from this “grab bag” to match their needs.
The key CA tools available from Microsoft include:
- The What If tool
- Report-only mode
- The conditional access optimization agent
- The conditional access insights and reporting workbook
- Entra ID sign-in logs.
Most of these tools are relatively mature and predate Entra ID, other than the optimization agent, which is new and continues to have new features made available.
What If Tool
The What If tool is a stand-alone Web application that offers analysis of the policies that will apply to a specific identity, given a specific target resource (application) and one or more conditions. This can help administrators understand whether a newly designed (or modified) CA policy will apply to a specific user.
What questions does this answer: What policies will apply to a specific user, given a set of conditions?
Most useful for: Initial design and implementation, lab environments.
Results returned to the administrator:
- The policies that will apply to the specific identity and their status (report only or on)
- The policies that will not apply to the specific identity and their status
- Any applicable grant controls (grant or block access)
- Any applicable session controls (which gate or control features available when connecting to compatible Web applications).
The returned results also include an indicator showing whether any classic policies exist in your environment. Although this typically doesn’t directly apply to the scenario the administrator is trying to analyze, classic policies are outdated and should be replaced with modern, Entra ID-compatible CA policies.
Limitations: Administrator must provide the specific App ID (also known as client ID) of an application, not a bundle of applications like Microsoft 365 Apps for enterprise.
The results from the What If tool are visible within the admin center, as shown below (fig. 1):
Report-Only Mode
Report-only mode is a run-time mode for any CA policy, designed into the Entra ID admin center. Report-only mode allows an admin to gain a sense of when a policy would or would not be applied before activating the policy in production. The results can also show when or whether the policy may have required user interactivity with a grant control or session control, so could not be fully evaluated using report-only mode. Note that a policy can either be report-only, or on. There is no way both capabilities can be applied concurrently (for example, report-only for one specific group of users, on for another group of users.
What questions does this answer: If I implement this CA policy as created, will it be effective? Does it impede users from their work? Does it adequately ensure unauthorized users cannot bypass it?
Most useful for: Initial design and implementation, lab environments, pre-production testing within test or production environments.
Results returned to administrator:
- A comprehensive graph showing success or failure of the given policy, and when it was not applied, over time
- Specific logged examples where the policy did not apply (what user, when, while accessing which application).
Limitations: This capability is largely limited to Windows clients. Administrators should exclude Apple and Android devices because they can create infinite loops of prompts on user devices, even in report-only mode.
The results from report-only mode are visible within the admin center, as shown below (fig. 2):
Conditional Access Optimization Agent
The conditional access optimization agent is a Security Copilot agent that is available within the Entra ID admin center. It finds gaps in CA policies as applied to users and apps added within the last 24 hours and makes recommendations on how to address them. This capability can be performed on-demand or automatically every 24 hours. The optimization agent can also pre-create new report-only CA policies that are intended to address any discovered shortfalls. (Note that many features of the optimization agent remain in preview.)
What questions does this answer: What newly added users and applications currently do not have CA policies applied to them? How should I best address these gaps? Are there policies we have implemented that overlap with each other and should be consolidated to ease management?
Most useful for: Production environments, possibly labs.
Results returned to administrator:
- The number of new unprotected users and applications
- Policies with functional overlap that could possibly be consolidated
- New policy suggestions intended to address unprotected users or unprotected applications.
The agent can optionally create suggested new policies automatically in report-only mode, in phases based on potential impact. Results also provide the total number of sign-ins where CA was applied, and the Security Copilot capacity that has been used by the optimization agent over time. (The latter is only marginally useful since it is returned in security compute units [SCUs] consumed, not the actual USD or other currency that has been spent.)
Limitations: A licensed instance of Security Copilot is required, and the optimization can only achieve the intended result of comprehensive policy application if it is used as a Security Copilot agent that is running all the time and returning results each day. Administrators should also examine policies before applying them to ensure the agent has not made a poor design choice in the suggested policies or the report-only policies that have been created.
The optimization agent currently has a significant number of internal limitations, including that the agent can only run automatically every 24 hours, and agent suggestions cannot be customized before they are implemented. The agent can also address up to only 300 users and 150 applications in a single run and should not use an identity that requires Privileged Identity Management because it will fail to run.
The results from the CA optimization agent are visible within the admin center, as shown below (fig. 3):
Conditional Access Insights and Reporting Workbook
The conditional access insights and reporting workbook is a stand-alone Web application that offers reporting on CA policies over a range of time.
The workbook shows how CA policies are being applied practically across the organization in production, and where issues may require deeper investigation, including users who are making it through sign-in without a policy applying to them.
The workbook predates the optimization agent described above and offers similar results in some cases. However, unlike the optimization agent, the workbook does not offer any policy suggestions for addressing scenarios where CA was not applied, or suggestions for when policies could be consolidated.
What questions does this answer: What newly added users and applications currently do not have CA policies applied to them? How should I best address these gaps? (Although similar to the optimization agent, the workbook can be used to report on almost any arbitrary timeframe and offers much deeper results. However, the workbook can only be accessed on-demand.)
Most useful for: Production environments, possibly labs.
Results returned to administrator:
- Total success and failure, scenarios where no policy applied, and when user action is required before success or failure can be assessed.
- Each result can be further broken down to show how policies applied, including breakdowns to specific devices or device states, client applications, or locations.
The results can also be downloaded as an Excel document.
Limitations: A licensed instance of Log Analytics is required.
The results from the CA insights and reporting are visible within the online workbook, as shown below (fig. 4):
Entra ID Sign-in Logs
Raw Entra ID sign-in logs are available for organizations that want to create their own reports and integrate sign-in information (including CA) with their existing security incident and event management (SIEM) tools, such as Microsoft Sentinel or Splunk. Logs can also be exported to other locations as needed to address longer-term governance, compliance, or security assessment needs.
The logs are available directly within the Entra ID admin center, can be queried using Microsoft Graph, or exported as needed.
The logs are likely not the most useful tool for assessing or addressing CA needs due to the sheer volume of information provided, but they may help administrators discover how users are signing in, from where, what devices are being used, when risky sign-ins occurred, etc. For example, an administrator can research a specific user’s sign-in activity or application’s usage and see how CA was or was not applied.
The logs also show when specific types of sign-ins occurred, including interactive user sign-ins (typical users signing in directly) and noninteractive sign-ins where the user signed-in via an application or Windows component. The sign-ins also include service principles, managed identities, and Copilot agents.
What questions does this answer: What users have signed in to my Entra ID and when? How did these users access Entra ID and the target resource (directly or through a specific application?) What resources were accessed by this user? When did sign-ins succeed or fail? Numerous other questions could be answered depending on where sign-in logs were exported to for further reporting or analysis.
Most useful for: Production environments, comprehensive security or governance requirements, or deeper reporting than available using the tools discussed previously.
Results returned to administrator:
No direct results are returned to administrators, but logs can be consumed and analyzed using several different approaches and tools, including the Entra ID admin center, Log Analytics, a SIEM like Splunk or Microsoft Sentinel, and custom scripts, applications, or reports.
Limitations: The logs are largely raw data and require comprehension and typically processing in some form to be useful.
Licensing
At a minimum, CA requires that every user must be licensed for Entra ID P1 as well as Intune Plan 1. Some popular features exposed through CA, like Entra ID Protection, require Entra ID P2 for all users. Additional licenses for Microsoft Defender services, Entra ID Governance, or other services may also be required depending on their use within CA policies.
While these tools are all free, some tools require paid instances of Log Analytics or Security Copilot.
Directions Recommends
Design and test policies using report-only mode prior to deployment. Report-only offers the best—but not a complete—answer for assessing policies prior to deployment or for debugging troublesome policies.
Use the insights and reporting workbook for regular reporting on CA policies. While limited in scope, the CA insights and reporting workbook is the most complete reporting solution Microsoft offers for reporting on CA.
Consider using the CA optimization agent to bridge policy gaps. The newly enabled CA agent offers useful potential for a price but has limitations. In addition, policies should be reviewed prior to implementation as with any set of results returned from an AI-based service or tool.
Export logs to existing SIEMs or custom reports if deeper analysis is required. Microsoft’s range of freely available tools are limited in scope. Organizations with more comprehensive analysis or reporting needs will need to build custom applications or license third-party applications in order to achieve their desired detail.
Resources
Suggested steps for implementing conditional access are described at
Plan a Conditional Access deployment (Microsoft).
The conditional access insights and reporting workbook is described at
Conditional Access insights and reporting (Microsoft).
The conditional access optimization agent is described at
Microsoft Entra Conditional Access optimization agent (Microsoft).
Entra ID sign-in logs are described at
What are Microsoft Entra sign-in logs? (Microsoft).
Report-only mode is described at
Analyze Conditional Access Policy Impact (Microsoft).
The What If tool is described at
Troubleshoot Conditional Access Policies with the What If Tool (Microsoft).
Troubleshooting Entra ID sign-ins and conditional access is described at
Troubleshoot sign-in problems with Conditional Access (Microsoft).