Skip to main content

Investigate endpoints

The Investigate tab in Endpoints gives you forensic investigation and response on the hosts in an Endpoint Control fleet. It runs alongside your EDR rather than replacing it. Use it to check whether a fleet is compromised, look back through each endpoint's recent history, and keep a tamper-evident record of what happened.

Investigate is not an EDR. It does not block anything before it runs, and it never acts on its own. YARA matches are scored in shadow mode, and every change to an endpoint, such as quarantining a file or placing a legal hold, is a request that goes through your organization's approval rules.

Beta

This feature is currently rolling out and may not be enabled for your organization. The Investigate tab appears only where it is enabled.

Open Investigate​

  1. Click Endpoints in the navigation menu and open the Investigate tab.
  2. Choose a fleet in the fleet switcher. Investigate works on one fleet at a time, so with All fleets selected the tab asks you to choose one.
  3. Pick a section from the strip at the top.
SectionWhat you do there
Compromise assessmentAdd YARA rulesets, run hunts across the fleet and review matches.
IOC look-backSearch every endpoint's stored history for one indicator or a whole IOC list.
DetectionsSee live YARA detections across the fleet.
Evidence integrityCheck each endpoint's event store and signed digests, and place or release a legal hold.
ApprovalsApprove, reject or withdraw quarantine, legal-hold and settings requests. A badge shows how many are waiting.
SettingsTurn the event store, file scanning and live detection on or off, set retention, and copy the flagfile lines endpoints need.

This page refers to a section by its place in the app, for example Endpoints › Investigate › Approvals.

On an endpoint's page​

Work on one endpoint happens on that endpoint's page, in these tabs:

TabWhat you do there
Activity mapFollow what happened on the endpoint over time and through its process tree, live if you like. See Activity map.
Event timelineRead the endpoint's stored events for a time range. See Event timeline.
DetectionsLive YARA detections on this endpoint.
Evidence integrityThe endpoint's store, signed digests and legal hold, with Verify now.
Look-backCheck this endpoint's history for one indicator. IOC lists are looked back for from Endpoints › Investigate › IOC look-back.

Every endpoint named in the fleet sections (hunt progress and matches, look-back results, detections and the evidence integrity table) links to that endpoint's page, on the matching tab. Timeline and Activity map buttons open the endpoint's tab at the right moment. Endpoints › Investigate › Timeline and Activity map links still work: they ask you to choose an endpoint, then open its tab.

Compromise assessment​

A compromise assessment scans files on your endpoints with YARA-X rules. The endpoint runs the scan and reports only the files that match.

Add a ruleset​

  1. On Endpoints › Investigate › Compromise assessment, under Rulesets, click Add ruleset.
  2. Give it a Name, then paste your rules into YARA source or click Choose file to load a .yar file (up to 2 MB).
  3. Click Compile.

Huntbase compiles the rules with the same YARA-X version the endpoints run. You see Compiled or Compile failed, with any Errors and Warnings, the number of rules and a Perf check: how many milliseconds the rules took per MB of a small test scan. Fix the source and click Compile again if it failed. Only compiled rulesets can be used in a hunt.

Adding a ruleset with a name you already used creates a new version of it. The Rulesets table lists every version with its Version, Status, Rules, Warnings, ms / MB and SHA-256.

Start a hunt​

Click Start hunt under Hunts, or Start a hunt after a ruleset compiles.

FieldWhat it does
RulesetThe compiled ruleset to scan with.
TargetsAll online endpoints in the fleet, or Selected endpoints. Endpoints that are offline when the hunt starts are skipped.
ScopeDefaults for Linux, macOS or Windows fill in paths where attacker tooling and persistence usually land, such as temp, user, startup and service folders.
Paths to scanOne path or glob per line. ** matches any depth, for example /home/*/Downloads/**.
ExcludePaths or globs to skip.
Max file size (MB)Larger files are skipped. 64 MB by default.
Time budget per endpoint (minutes)The scan stops when it runs out. 15 minutes by default.

The endpoint always skips system pseudo-filesystems (such as /proc, /sys and /dev on Linux), network shares and its own cache. It doesn't follow symbolic links.

Hunts go out in waves of endpoints, not to the whole fleet at once, and each endpoint runs one scan at a time. A file that hasn't changed since the last scan with the same ruleset is not read again; its earlier match is reported for the new hunt.

Approve a hunt​

A hunt is a low-risk action under your organization's approval rules. If low-risk actions need approval, the hunt shows Needs approval and Waiting for approval, and nothing is sent until someone clicks Approve hunt. The page tells you who may approve: under four-eyes, a different person must approve it, so you can't approve your own hunt.

Follow a hunt and read matches​

Click a hunt in the Hunts table to open it. While it runs the page updates by itself.

Endpoints shows each endpoint's progress, Files scanned, Matches and any Error. A large hunt lists 500 endpoints at a time: use Previous and Next, or filter by status. The progress bar always counts every endpoint.

StatusMeaning
Pending, Dispatched, Queued, RunningWaiting for its wave, sent, waiting behind another scan on the endpoint, or scanning.
DoneThe scan finished.
Budget exceededThe time budget ran out. Matches found before then are kept.
CancelledScanning was switched off on the endpoint (see Resource limits and the kill switch).
Timed out, Skipped, FailedThe endpoint didn't answer in time, was not scanned, or the scan failed. The error says why.

Matches lists each file that matched a rule: Host, Path, SHA-256, Rule, Tags, Versions (ruleset and scanner) and the Shadow decision. Timeline opens the endpoint's event timeline at the moment of the match.

Each match also becomes an insight titled "YARA match: rule name", with the endpoint, the file and any ATT&CK techniques from the rule. See Insights.

Shadow decisions​

For every match, the endpoint scores the rule from 0 to 100 and records what it would do: Would quarantine at a score of 70 or more, otherwise No action proposed. The cell is labelled Shadow · no action taken: the decision is recorded so you can judge it, and nothing happens on the endpoint.

The score comes from the rule's score meta, or else its severity meta (critical 95, high 80, medium 50, low 25, info 5), or 50 with neither. Malicious tags such as malware, ransomware or webshell add 10; test, benign, hunting or informational take off 20.

Request quarantine​

To act on a match, click Request quarantine, give a Reason and confirm. Quarantine moves the file into the endpoint's quarantine, where it can be restored.

Quarantine from a match is a high-risk request under your approval rules. Unless your rules let anyone approve, it waits for a second person (Waiting for approval) on Endpoints › Investigate › Approvals. Once it is sent, the button reads Quarantine requested and View action opens the endpoint's Actions tab, where you can follow it.

Rule-writing tips​

  • Set severity (critical, high, medium, low or info) or an integer score in each rule's meta. Without them every match scores 50.
  • Add mitre_attack = "T1059, T1105" to the meta to tag the insight with ATT&CK techniques.
  • Put a description in the meta. It becomes the insight's description.
  • Test a new ruleset against a harmless marker first: a file you create with a unique string, and a rule tagged test that matches only that string. Hunt one endpoint with a narrow path before you hunt the fleet.
  • Watch Warnings and ms / MB. A slow ruleset uses more of each endpoint's time budget.

IOC look-back​

Each endpoint keeps a local, tamper-evident event store. A look-back searches it for an indicator without moving the data off the endpoint.

  1. Open Endpoints › Investigate › IOC look-back.
  2. Under Look back for, choose Single indicator or IOC list.
    • Single indicator: choose the Indicator and enter the Value.
    • IOC list: choose one of your IOC lists. Every indicator in the list is matched against each endpoint's store.
  3. Set Days back (1 to 90, 7 by default).
  4. Pick the endpoints, or keep the whole fleet, and click Look back.
IndicatorMatches
SHA-256Exact file hash, from process starts and YARA matches.
IP addressRemote address of network connections.
PathExact path, or a prefix ending in *.
Command lineCase-insensitive substring of process command lines.
DomainSubstring of any stored hostname or DNS field (best effort).

Hits appear as endpoints answer, with each endpoint's status and hit count. For a list look-back, the Matched IOC column shows which indicator each hit matched. Select a hit to open that endpoint's event timeline at that moment.

To check one endpoint quickly, use the Look-back tab on its page: it searches only that endpoint, for a single indicator, and its History shows the look-backs that covered it. A look-back only finds what is still in the store, so see Retention for how far back that goes.

Each endpoint reports at most 5,000 hits, within a time budget. An endpoint that reached either limit is marked capped, and a note says "Not every hit is shown". Narrow the days, or split the list.

History, under the results, lists the fleet's earlier look-backs with their Indicator (or the list name and version), Value, Days, Endpoints, Hits, Status and when each was Run. Open one to see its results again.

IOC lists​

An IOC list is a set of indicators you look back for in one go. The IOC lists section on IOC look-back shows each list's Name, Version, Entries, counts By kind, how many lines were Rejected, and when it was Added.

  1. Click Add IOC list.
  2. Give the list a Name.
  3. Paste indicators, or click Choose file to load a .txt or .csv file.
  4. Click Upload.

Put one indicator per line, or use CSV. Lines starting with # are comments, and a kind,value header line is skipped. The kind of a bare value is detected; to set it yourself, write kind,value (a tab or ; also works).

KindExample lineMatches
SHA-256e3b0c442…b855 (64 hex characters)The exact file hash, from process starts and YARA matches.
MD5d41d8cd9…8427e (32 hex characters)The exact hash, where the endpoint has one.
IP203.0.113.7Remote address of network connections.
Domainbad.example.netThat domain and its subdomains: evil.com matches a.evil.com, not notevil.com. URLs and defanged values (hxxp://, [.]) are accepted; the host is used.
Path/tmp/.x/*An absolute path. A trailing * matches as a prefix.
Command linecmdline,-enc JABCase-insensitive substring of process command lines. Command lines are never detected: start the line with cmdline, (or command,).

Other accepted kind names: hostname, fqdn and url for domains; file and filepath for paths; ip-src and ip-dst for IPs.

Duplicates are dropped and values are normalised. A list holds at most 100,000 indicators and 16 MB. After the upload you see how many indicators of each kind were accepted, and the Rejected lines with the reason for each (the first 100 are shown). Uploading a list with a name you already used creates a new version.

Event timeline​

The Event timeline tab on an endpoint's page reads process, network, login and persistence events from the endpoint's local store.

  1. Open the endpoint's page and the Event timeline tab.
  2. Pick a range (1 h, 6 h, 24 h or 7 d) or set From and To, then click Apply.
  3. Filter by class: Process start, Process exit, Connect, Listen, Login, Logout, Persistence +, Persistence −, File change and Scout action (actions run on the endpoint, including legal holds and quarantines).

Click a process to open its Process tree and Parent chain, and Open activity map to see it on the activity map. A timeline shows up to 5,000 events. If the endpoint didn't answer in full or the limit was hit, a note says so; narrow the range or click Refresh.

Detections​

With live detection on, each endpoint scans the executable of every process as it starts. A match shows up within seconds on Endpoints › Investigate › Detections, and on the Detections tab of the endpoint's page. Both update every 10 seconds, and new rows are highlighted.

ColumnWhat it shows
DetectedWhen the endpoint flagged it.
EndpointThe endpoint, linked to its page (fleet view only).
RuleThe YARA rule and its tags.
ProcessThe program, its pid and full path.
SHA-256The executable's hash.
Exec → detectFrom the process starting to the match on the endpoint. The header shows the median.
→ HuntbaseFrom the endpoint to Huntbase.
Shadow decisionThe score and what the endpoint would have done, labelled Shadow · no action taken. See Shadow decisions.

Activity map opens the endpoint's activity map on the detected process. Detections also appear in the activity map's Detection lane and on the process itself.

Live detection blocks nothing. To act on a file, run a hunt and request quarantine from its match.

Evidence integrity​

The store on each endpoint is tamper-evident. Someone with full control of a host can still delete or rewrite it, but once Huntbase has collected a day's digest, a rewrite can't go unnoticed.

  • Hash chain. The endpoint writes events in small batches. Each batch's hash is chained to the one before, per day.
  • Signed daily digests. Each day's chain head, sequence range and event counts are signed with a key that is unique to the endpoint. The current day is Open (rolling); at midnight UTC the day is Closed and its digest can never change.
  • Host key pinned on first sight. Huntbase records each endpoint's public key the first time it sees it, and checks every digest against that key. A different key later is treated as tampering, never accepted quietly.

Endpoints push signed digests every 15 minutes, and Huntbase verifies each one as it arrives. An endpoint that goes quiet for 30 minutes shows a warning, "No integrity update since …". Read live reads every endpoint's store and digests now (up to 25 seconds). Verify now, on an expanded endpoint or on its Evidence integrity tab, reads and verifies that one endpoint. Expand an endpoint to see its Daily digests: each day's sequence range, Chain head and whether the signature and chain are OK. Click the endpoint's name to open its Evidence integrity tab.

IntegrityMeaning
VerifiedEvery digest verifies with the pinned key and the days follow on from each other.
WarningNothing is proven wrong, but something needs a look: events were dropped, the endpoint has no digests yet, or it hasn't sent an update for 30 minutes ("No integrity update since …").
BrokenEvidence can't be trusted as-is: a signature doesn't verify, the host key changed or is invalid, a closed day's digest changed or reopened, a closed day was deleted before its retention ran out, there is a gap in the sequence between days, the store was found corrupt, or the endpoint's own chain check failed.
UnknownThe store hasn't been read yet, or is off on that endpoint.

The issues behind a warning or a break are listed for the endpoint. A Broken endpoint also raises an insight, "Investigate: evidence integrity broken".

If the store on an endpoint is found corrupt, the endpoint keeps the damaged store as evidence, starts a fresh one, and shows Broken.

Clear a broken state​

Broken stays until an organization admin clears it, so evidence from that endpoint isn't trusted by mistake. Investigate first, then:

  1. Click Clear… on the endpoint, in the table or on its Evidence integrity tab.
  2. Give a Reason.
  3. Tick Re-baseline the host key only when you know why the key changed, for example because the endpoint software was reinstalled. It pins the endpoint's current signing key.
  4. Click Clear integrity state.

Clearing doesn't repair anything: digests are verified again from then on. The clear and your reason are written to the audit trail.

The table also shows each endpoint's store size, Oldest and Newest day, event and Drops counts, and Clock skew (how far the endpoint's clock is off; timestamps are never adjusted).

A legal hold stops the endpoint deleting anything from its store, so evidence survives past the retention window.

  1. On Endpoints › Investigate › Evidence integrity, switch the hold on in the endpoint's Legal hold column, or use the Legal hold switch on the endpoint's Evidence integrity tab.
  2. Enter the Case id and a Reason, then click Request hold.

Placing and releasing a hold are medium-risk requests under your approval rules, so they may wait for a second person on Approvals. While on hold the endpoint keeps every stored event up to the store's size cap; past the cap, new events are dropped and counted. Releasing the hold (Request release) resumes normal retention, and events older than the retention window are then deleted.

Approve requests​

Quarantine, legal-hold and some settings requests go through your organization's approval rules. Approvers act on them in Endpoints › Investigate › Approvals. A badge on Approvals shows how many requests are waiting.

Requests lists each request with the Request (Quarantine file, Place on hold, Release hold or Settings change), the Endpoint, the Reason, who it was Requested by, its Risk · policy and its Status. Show Pending approval or All.

ButtonWho sees itWhat it does
ApproveSomeone the policy allows to approveSends the request to the endpoint.
RejectSomeone the policy allows to approveTurns the request down.
WithdrawThe person who askedTakes back your own request.
  • Four eyes: you can't approve your own request; someone else has to. You can withdraw it.
  • Org admin: the request needs an organization admin, other than the requester, to approve it. A Settings change always needs another organization admin.
  • Any responder: anyone allowed to run endpoint actions can approve it, including the requester.

Pending requests lapse after 24 hours. Once a request is approved, follow it on the endpoint's Actions tab.

Retention​

WhatKept by default
High-value events: process starts, logins, persistence changes and actions run on the endpoint30 days
Routine events: process exits, network connections and listening ports, logouts and file changes7 days
Store size1 GB per endpoint; the oldest days are deleted first when it is full

The day's chain and digest stay when routine events are pruned, so pruned days still verify. A legal hold suspends all deletion.

Resource limits and the kill switch​

Scans run as a separate process at the lowest CPU priority, capped at half of one core by default and with a memory limit, and they stop when the time budget runs out. The event store is bounded by its size cap.

To stop scanning at once:

  • Safe mode on a fleet stops running scans and refuses new ones, along with every other endpoint action.
  • File scanning off, in Settings, is the fleet kill switch for YARA scans. It applies at once, with no approval: running scans stop within about a second of the endpoint receiving the change and end Cancelled, new hunts are refused, and live detection stops.

Settings​

Organization admins manage a fleet's Investigate settings on Endpoints › Investigate › Settings. Changes apply to every endpoint in the fleet at its next config refresh and are written to the endpoint audit trail.

Capabilities

SwitchWhat it does
Event storeEach endpoint keeps a local, tamper-evident store of process, network, login and persistence events. IOC look-back, the timeline and evidence integrity read from it.
File scanningLets endpoints run YARA hunts.

Turning either off asks you to confirm. Turn off scanning applies at once, and the page shows File scanning is off — fleet kill switch until you turn it back on. Turn off event store stops endpoints recording evidence, and needs another organization admin to approve it on Approvals.

Live detection​

The Live detection card turns on scanning of each new process's executable as it starts. Matches appear under Detections within seconds, and in the timeline and the activity map. Nothing is blocked: decisions stay in shadow mode.

  1. Choose a compiled Ruleset.
  2. Set the CPU cap (%): 5 to 100, 25 by default.
  3. Switch live detection on, or click Save after a change.

Turning it on or off applies at once. It can't run while File scanning is off. The card shows the ruleset that is running, when the signed scanner and ruleset next renew (daily), and the last error from an endpoint, if any.

Retention sets how long endpoints keep events: High-value events (days), Routine events (days) and Store size cap (MB). Click Save retention. Routine events can't be kept longer than high-value events. Lowering a retention or the size cap deletes evidence, so it also needs another organization admin's approval; raising them applies at once. While a change waits, the page says so, with Open Approvals.

Agent flagfile lines are the osquery flags the event store needs on each endpoint. They are not pushed automatically. Copy them into each endpoint's flagfile (see Requirements).

Feed queries lists the scheduled queries the fleet serves to fill the event store. They are read-only.

Platform notes​

PlatformProcess events
LinuxKernel (BPF) process events, which need the endpoint software to run as root with BPF allowed. In containers, where the default security profile blocks BPF, the endpoint falls back to process snapshots taken every 10 seconds. Short-lived processes can be missed.
WindowsProcess snapshots when the endpoint software runs without elevation, because Windows event tracing (ETW) needs elevation.
macOSEndpoint Security events. Grant Full Disk Access to the endpoint software through your MDM.

Network, login and persistence events come from periodic snapshots on every platform.

Permissions​

TaskWho can do it
See rulesets, hunts, matches, detections and evidence integrityAnyone with access to the fleet.
Run an IOC look-back, read a timeline or the activity map, or Verify nowPeople allowed to run commands on the fleet's endpoints: admins of the connection, or a Responder grant.
Add rulesets, start or approve hunts, request quarantine or a legal hold, approve or reject requestsPeople allowed to run endpoint actions: admins of the connection, or a Responder grant. Your organization's approval rules decide who may approve each request.
Change Investigate settingsOrganization admins who can run endpoint actions. Turning the event store off, or lowering retention or the size cap, needs a second organization admin's approval.
Add IOC lists, see look-back historyPeople allowed to run commands on the fleet's endpoints.

On a shared fleet Investigate is view-only. See Access and activity to grant the Responder role.

Requirements​

  • Event store and, for hunts, File scanning turned on in Settings.

  • Scout endpoint extension 0.6.0 or later on each endpoint.

  • The Agent flagfile lines from Settings in the osquery flagfile on each endpoint. Settings shows the lines for your fleet. By default Scout reads events from osquery directly, and the lines are only --disable_events=false plus the event flag for the platform:

    PlatformEvent flag
    Linux--enable_bpf_events=true
    macOS--disable_endpointsecurity=false
    Windows--enable_process_etw_events=true

    No logger changes are needed. Restart osquery after you add them.

  • If a reverse proxy sits in front of Endpoint Control, it must route /scout/live, /scout/config, /scout/detections and /scout/digests (for example the regex /scout/(live|config|detections|digests)), or live mode, detections and integrity updates don't arrive.

Next steps​