Test builder

A new Maester check takes the next free MT. number. Reserve it on issue 697 before you open the pull request, so two contributors do not pick the same one.

The product where the fix is made. One value.

• Use the product whose admin surface holds the setting, not every product the test reads from.
• Tests in a product folder (tests/maester/entra, tests/cisa/exchange) use that folder's product.
• A product the test only reads from, or a licence it needs, is not a second product. Service and License already say that.

Search the list first. A product is a deliberate addition, because every key becomes a filter in the report. If nothing fits, choose Not in the list above, and add the entry to powershell/assets/MaesterTaxonomy.psd1 in the same pull request as the first test that needs it.

A product entry has a Key, a Name, Aliases and its Features. If the test also needs a connection Maester does not have yet, that is a larger change to powershell/assets/MaesterServiceRegistry.psd1: open an issue first.

The part of the product the test is about. One key from the product's Features list.

• Choose the feature whose settings page or object type the remediation sends the admin to.
• When a test spans two features, choose the one that fails most directly. A Conditional Access policy that requires a compliant device is conditional-access, not devices.
• Tests that check the same thing get the same feature, whatever folder or suite they are in.
• A feature that would hold one test only is usually too narrow. Prefer an existing feature.

Search the list first. A feature is a deliberate addition, because every key becomes a filter in the report. If nothing fits, choose Not in the list above, and add the entry to powershell/assets/MaesterTaxonomy.psd1 in the same pull request as the first test that needs it.

Add one line to the product's Features list. A feature that would hold one test only is usually too narrow, so prefer an existing one.

Other products the test also checks or depends on. Optional list, never the same key as Product.

• Add one when an admin of that product would expect to find the test under it. An Exchange Online test about an anti-phishing policy that is managed in the Defender portal has RelatedProduct = 'defender'.
• Add one when the fix is made in that product's portal although the test lives in another product's folder.
• Do not add a product only because its API is called.

The security theme, whatever the product. One key.

When more than one area fits, choose by what the control is for, in this order:

1. It protects administrators or admin rights: privileged-access.
2. It decides how sign-in is proven: authentication.
3. It is about an application, service principal or agent as the subject: applications, or ai for agents.
4. It is about people or data outside the organization: external-access.
5. Otherwise the area that names the mechanism: access-control, devices, email-protection, data-protection, logging-and-detection, identity-lifecycle, engineering-systems, infrastructure.

A product never decides the area on its own. An Azure DevOps test about who is a Project Collection Administrator is privileged-access, not engineering-systems.

Inventory and health tests with no security theme of their own are infrastructure. When no area fits, choose the closest and say so in the pull request, so the list can be reviewed.

Search the list first. An area is a deliberate addition, because every key becomes a filter in the report. If nothing fits, choose Not in the list above, and add the entry to powershell/assets/MaesterTaxonomy.psd1 in the same pull request as the first test that needs it.

The thirteen areas are meant to cover every product. When none fits, choose the closest and say so in the pull request, or propose a new one with its description and a row in the rubric's Area table.

What a failing result points at. One key.

• Tenant when the test evaluates one tenant-, organization-, forest- or domain-wide setting. The result is about the environment.
• Tenant when the test checks that at least one policy of a kind exists, even when it lists the policies that satisfy it. A failing result has nothing to list.
• Tenant when the test evaluates the single default or global policy of a kind, because there is one of it and it applies to everyone.
• Otherwise the kind of object a failing result lists or counts: User, Application, Device, ConditionalAccessPolicy, GroupPolicy and so on. A test that evaluates every policy and lists the ones that fail uses the policy type.
• A test that counts objects for review (an inventory test) uses the type it counts. When no key names that kind of object and it is a one-off, use Tenant.
• A family test, which yields one result per instance, uses the type of the instance.

Search the list first. A resource type is a deliberate addition, because every key becomes a filter in the report. If nothing fits, choose Not in the list above, and add the entry to powershell/assets/MaesterTaxonomy.psd1 in the same pull request as the first test that needs it.

Add one line to ResourceTypes with a one-line description. For a one-off inventory with no matching type, the rubric says to use Tenant.

The risk to the organization while the test fails. Not the effort to fix it and not the disruption of fixing it.

Decide in this order:

1. Reach. Privileged identities or the whole tenant move the value up. A single workload or a few objects move it down.
2. Directness. Usable from the internet or by any signed-in user moves it up. Needing an earlier compromise moves it down. A password taken by phishing, spray or stuffing is not an earlier compromise: those are the common techniques High already assumes.
3. Role of the control. The main control for a threat rates higher than a second layer behind another control.

Critical is rare. If the description cannot say in one sentence how an attacker gets from the failing state to admin rights or exposed secrets, the value is High or lower. A benchmark's own wording does not set the value: a SHALL policy can be Medium.

Five cases come up often:

• A test that passes whenever its data can be read is Info, whatever the data is about. If the fact deserves a severity, the test should fail on it.
• A failure that is only an outage or untidiness, such as replication health, an expiring token or an unused object, is Low. It is Medium or higher only when the outage switches off a security control, and then it is rated as the loss of that control.
• Alerts and notifications are a second layer behind the control they watch, so they are Medium at most.
• The same check in two suites gets the same value in both.
• The main control against an attack that starts from a stolen password is High. A second layer behind that control is Medium. Number matching against MFA fatigue is High; showing the app name and location in the prompt is Medium.

Anchors from existing tests:

Microsoft's Zero Trust Assessment rates many of the same Microsoft Entra and Intune settings. Where it has a test for the same setting, compare your value with its risk level and give the reason in the pull request if the two differ.

The five values are fixed. The report, the scores and the -Severity selector depend on them.

How much the recommended fix disrupts end users. Administrators and the owners of applications are not end users here. Developers are the end users of Azure DevOps and GitHub.

Omit UserImpact when Severity is Info.

How much administrator work the recommended fix takes.

Clean-up that one administrator can finish alone in a sitting is Low. Clean-up that needs the owner of each object is Medium. Turning on a plan that is billed by use is Medium: the click is easy, the budget approval is not. Count the work of the fix, including exclusions and rollout, but not the time to decide.

Omit ImplementationCost when Severity is Info.

The MITRE ATT&CK techniques that the checked control prevents or detects. A list of technique IDs, or None.

• IDs only, in the form T1110 or T1110.003. Tactic names are looked up from the pinned ATT&CK release (powershell/assets/MaesterAttackTechniques.json) and are never typed.
• Use a sub-technique when the control addresses only that sub-technique. When three or more sub-techniques of one technique apply, list the technique instead. Do not list a technique together with one of its own sub-techniques.
• List a technique only if the test's description explains, or plainly implies, how the control stops or reveals it. At most five.
• A test that implements a benchmark which publishes a mapping, such as CISA SCuBA, starts from that mapping and then applies these rules. An ID the pinned release has retired is replaced by its successor in that release, found by name.
• The description means the comment-based help and the Markdown file together.
• Use None for a test with no attacker behaviour behind it, such as replication health or an inventory count. An inventory test lists a technique only when the fact it reports is itself the precondition of that technique, such as accounts that do not require Kerberos pre-authentication.
• A retired or unknown ID fails the unit tests.

The list is the ATT&CK release Maester pins, in powershell/assets/MaesterAttackTechniques.json. A technique that is missing was retired, or is newer than the pinned release. Use its successor, or update the pinned release with ./build/attack/Update-MtAttackTechniques.ps1 in a pull request of its own.

Requirements of frameworks the test implements or supports, written <framework-key>:<requirement-id>, for example cis-m365:5.1.5.1, cisa-scuba:MS.AAD.3.1 or nist-800-53-r5:IA-2(1). Optional list. Any test can use it: one shipped with Maester or one of your own.

• Control is the only way a test is mapped to a framework. A CIS test carries Control = 'cis-m365:5.1.5.1' and a CISA test carries Control = 'cisa-scuba:MS.AAD.3.1'. Every test in tests/cis and tests/cisa must name a requirement of its suite's framework. The unit tests check this.
• The first value that names a requirement in a framework file becomes the Benchmark of the result. Put the requirement the test is written for first, and any other requirement it supports after it.
• Use the same attribute for a test that exists for its own reasons and also supports a requirement: a Maester test that checks the setting a CISA policy asks for, or a CIS test that also supports a NIST control.
• When the key is a framework shipped with Maester (cis-m365, cis-github, cisa-scuba), the requirement must exist in that framework's file. If it is new, add it to the file with its facts. The module build and the unit tests fail otherwise.
• Do not repeat the Related controls of a requirement. The framework file lists them and Maester adds them to the result.
• List a requirement only when a passing result is evidence for it. A test that checks one part of a requirement may list it; a test that is merely on the same topic may not.

A reference says the test supports the requirement. It never says the organization complies with it.

NIST SP 800-53

A test may list the NIST SP 800-53 Rev. 5 controls its setting implements, as nist-800-53-r5:<control>.

• Use an ID from the pinned list, powershell/assets/MaesterNistControls.json: a base control (AC-2) or an enhancement (IA-2(1)). The unit tests reject anything else.
• List a control only when a passing result is direct evidence that it is implemented for what the test checks.
• Prefer the enhancement when the test checks exactly what it says: MFA for privileged accounts is IA-2(1). Do not list a base control together with one of its own enhancements.
• At most three, best fit first. Never the policy-and-procedures controls (AC-1, IA-1 and so on).
• A test that only reports a count, or checks health, licensing or tidiness, lists none, unless the count is itself what a control asks to be tracked: dormant accounts for AC-2(3).
• A test that implements a CISA SCuBA policy does not list NIST controls. CISA publishes that mapping, the framework file carries it as Related, and the result gets it from there.

The lookup lists the frameworks Maester ships a file for: tests/cis/frameworks/cis-github.json, tests/cis/frameworks/cis-m365.json, tests/cisa/frameworks/cisa-scuba.json. For any other framework, type the reference yourself with a lowercase key, for example nist-800-53-r5:IA-2(1) or iso-27001-2022:A.5.17, and use the same key every time.

The framework file is a catalogue. It holds what a key:id reference cannot: each requirement's level, profile or criticality, the controls it relates to, and the full list of requirements, so typos and gaps are found. A new CIS or CISA requirement is added to that file; the mapping itself is always this attribute.

Services are registered in powershell/assets/MaesterServiceRegistry.psd1. A new service also needs a connection probe in the module, so open an issue before the pull request.

The list is Microsoft's product names and service plan identifiers page, refreshed every day. You can also type a name that is not listed yet: it is matched as written against the tenant's service plan names.

For the plans in powershell/assets/MaesterLicenseTable.psd1 Maester also counts government, education and volume variants as the same licence. Add a plan there when its variants should count too.

• Preview: new and still changing. Not run unless previews are included.
• Stable: the default.
• Deprecated: kept so history stays readable. Give ReplacedBy when another test took its place.