Setting Detection Expectations
When implementing an endpoint detection and response solution it’s very tempting to imagine all the visibility you’ll have over endpoints. This level of visibility quickly becomes problematic because you discover many of the processes and commands that adversaries use to conduct discovery and system functions are also used internally by an operating system process or your own system administrators to perform job functions. This leads us into the topic of what behaviors are visible to EDR solutions and which ones may be reliably detected with the context Red Canary knows from your organization.
Excellent Visibility at Scale
Endpoint detection and response solutions give customers excellent visibility across their environment. Each product differs from the competition a bit, and the minimum parity across all EDR sensors is to provide process telemetry and metadata but not content. This means that all sensors across all platforms should provide process information such as path, name, command line, and md5 hash. Depending on the client platform (Windows, macOS, Linux), some telemetry data such as file modifications or network connections may be unavailable.
EDR solutions typically steer clear of reporting the content of files. Our partner sensors will show process command lines, but they do not index and report the content of executed scripts. Commands that are “internal shell commands” such as ‘echo’ and ‘trap’ will not be visible to EDR tools unless they are executed in very specific ways. For many cases this isn’t a problem because our Detection Engineers can piece together the context of several processes to deduce what has happened during a script execution. In cases such as a pure PowerShell script or Bash shell scripts that leverage lots of internal commands, this becomes more problematic.
Please also keep in mind that not all adversary techniques are visible to EDR solutions. A good example of this is Pass The Hash (T1075). This technique allows an adversary to authenticate to a system in a nefarious way that is apparent in Windows Event Logs. From the EDR perspective, the user that authenticates with Pass The Hash is simply another logged on user executing things.
Visibility != Detection
Visibility isn’t enough for our CIRT to detect malicious activity. The CIRT has to mix visibility with an additional factor- context. Context allows our Detection Engineers to accurately judge whether a single event has malicious or benign intent. In the course of your relationship with Red Canary you may receive detections that are classified as “Suspicious”. These are cases where the context we received wasn’t enough for us to judge an event as outright malicious but we felt it should still be reviewed.
Context is also a double-edged sword. There are some adversary techniques that blend in so well with existing process execution data that we cannot tell the difference between a malicious actor and a system administrator. For example, picture an event describing the modification of “.bash_history” by the “bash” process. A malicious actor may “echo” values into this file to overwrite contents, but the “bash” process also writes to this file during every execution to save command history. We encounter this context issue across all platforms but it is especially troublesome on macOS and Linux due to processes piping data, limitations of partner sensors, and the amount of detail for each process we can display to our CIRT for effective investigations.
Our Detection Engineers rely on the context of events that occur around a particular event that has been brought to their attention. They use process telemetry from sibling and parent processes, username details, and binary reputation information as data points to make a decision whether an event is malicious or benign. For this context, we rely on the likelihood that adversaries behave in particular ways to achieve their goals. A good example of a high-context event is a PowerShell instance that downloads an executable where PowerShell has spawned from an instance of Microsoft Excel. From this event, we can infer that functionality of Excel (likely a document macro) has spawned PowerShell. We can also use the network connection data from PowerShell and the reputation of the downloaded binary to determine whether the instance of PowerShell dropped a malicious binary or was the update to an add-in. This results in a quick and easy win for the customer and the CIRT.
Here are a few examples of low-context events:
- A single double-click execution of a Meterpreter process from a logged on user. From the EDR perspective this appears as a single process making a network connection to another host.
- Linux “echo” piping input into a “crontab” process. Since “echo” is usually invisible to EDR, we’ll only see the process “crontab” with a blank command line spawning from “bash”.
- Execution of reconnaissance commands. Reconnaissance commands are used commonly across all platforms for discovery and inspection of system components by administrators, operating systems, and adversaries. The presence of reconnaissance commands is typically common enough to overwhelm our CIRT if we look for individual commands alone.
At Red Canary we view our relationship with customers as a partnership, and this means we’re open to suggestions and new data from customers. To provide the best security for your organization, you can complement EDR tools with additional data that may lead to findings we do not have the context to detect. Useful data sources would be Windows Event Logs, PowerShell logs, and even Sysmon configured to collect select WMI events. For the macOS and Linux platforms, discoveries you’ve made using osquery or similar tools are also helpful.
If you discover malicious activity in your environment that we did not spot, you can help improve Red Canary by submitting details that led you to discover the activity to Incident Handling or Customer Success personnel. Once that data is submitted, we can help investigate additional activity using EDR tools available and work with you to improve our coverage for your environment and countless others.