Quarantine and rollout
Two safeguards keep detection rules in Intelligence › Detections from flooding your team: an automated rollout that moves new and changed rules forward one stage at a time, and quarantine, which holds back a rule that fires far too often.
Quarantine and rollout are part of Intelligence, which is in beta. Access is by request, so it may not be enabled for your organization.
Rollout
New and changed rules can reach every organization through an automated rollout, one stage at a time:
| Stage | What it means |
|---|---|
| Draft | The rule compiled. |
| Tested | Its fixture tests passed. It needs at least one must-match and one must-not-match test. |
| Backtested | Its estimated daily volume over recent telemetry is within the limit. |
| Shadow | It runs in a small set of organizations, recording to the digest only. |
| Canary | It takes its real action in a slightly larger set of organizations. Your own (Custom) rules skip this stage. |
| On | It runs everywhere, as published. |
The Rollout tab lists the stages with each one's status (Passed, In progress, Blocked, Failed, Skipped or Pending), when it was entered and left, and the evidence recorded for it: test results, the backtest estimate, fires per day in shadow, precision and breaker trips. Evidence is always an aggregate: it never names other organizations. The tab also says when the rollout is next evaluated, and why it is blocked if it is, for example it has no fixture tests yet or the circuit breaker tripped during shadow or canary.
A rule that trips the circuit breaker during shadow or canary goes back to shadow. If it trips a second time, the rollout fails.
Platform administrators also get Advance, Roll back, Pause/Resume, Edit plan, Evaluate now and Abort, plus Start rollout for a rule that has none. Advancing, rolling back and aborting need a reason, which is recorded.
Quarantine
A rule that fires far above its limit in an organization is quarantined there automatically. A quarantined rule runs as if it were in Shadow: its matches go to the daily digest, and it doesn't notify anyone or open hunts until it's released. Nothing is thrown away. Past a cap, matches are counted instead of recorded, and the banner shows how many were held.
The rule's Overview shows a banner with:
- how many times it fired against its limit, and over what window;
- how many matches were held while it was quarantined;
- when it will be released automatically, or Manual only after repeated trips;
- when it was quarantined, and how many times it has tripped.
An organization administrator can select Release from quarantine, give a reason and confirm. The reason is kept in the rule's audit history. After a release, the rule's limit is higher for a while, so that it doesn't trip straight back.
If a rule trips in several organizations, it's quarantined in every organization. Only a Huntbase platform administrator can release that, with Release globally.
Organization administrators are notified when one of their rules is quarantined. The notification opens the rule's Tuning tab on the Watchers page.
Next steps
- Detections: test and measure rules
- Watchers: switch rules on and off, and tune them