Canary Exporter Setup (Example: Collecting Logs in Local File) – Red Canary help

Issue

We're trying to use Canary Exporter to send logs a local file on our Server.

Environment

Red Canary

Resolution

Please reference the Red Canary Help article about Exporting telemetry from Red Canary.

It's important to understand what the Canary Exporter is and what it's used for. The Canary Exporter is a Docker Container (Basically it's an application) that allows customers to collect the same raw telemetry data that we receive from the EDR platform.

In order to use Canary Exporter please perform the following steps:

First, install Docker on a local Server (preferably a Linux Server). See Docker install instructions for your OS. NOTE: Windows VMs have not been tested by Docker. Since the Canary Exporter Docker Image is based on Linux, you should be using a Linux or Unix based machine. If you need to have Canary Exporter set up in Windows, please carefully follow the Docker installation instructions for Windows.

- Here are the Docker install instructions for Windows: https://docs.docker.com/docker-for-windows/install/
  - NOTE: In order for Docker to run successfully on Windows you will need to enable either Hyper-V Windows Features, or you will need to Install required Windows components for WSL 2 when prompted during the Docker for Windows installation. If you choose to go with WSL you will also need to install the "WSL2 Linux kernel update package for x64 machines" after running the Docker for Windows installer.
  - NOTE: Nested virtualization scenarios, such as running Docker Desktop on a VMWare or Parallels instance might work, but there are no guarantees. For more information, see Running Docker Desktop in nested virtualization scenarios.
  - If you are still having trouble getting Docker for Windows to run, please see the Docker Logs and Troubleshooting article.

- Here are the official Docker install instructions for CentOS 7: https://docs.docker.com/engine/install/centos/
  - NOTE: there are many other versions of Linux listed in the Docker installation guide. We chose CentOS as an example.

Once Docker is installed, create and configure the local config.yaml file (An example config.yaml file is located in Red Canary > User Profile > Canary Exporter).
- NOTE: The default location to save the config.yaml file in LInux is: $HOME/canary-exporter/config.yaml.
- NOTE: The default location to save the config.yaml file in Windows is: C\User\<username>\canary-exporter\config.yaml.
Once the config file is created, edit the configuration. Adjust the "Input" section. Specifically the "stream_name:" section. Decide whether you want to receive "native" or "standardized" formatted log data. "Native" data format is the raw pre-processed event telemetry as we receive it from the EDR product. "Standardized" data that is formatted telemetry data according to Red Canary's standardized format. This format tends to be easier to read and parse, and is product-agnostic.
- NOTE: Specify either native or standardized in your config.yaml file without the use of single quotation marks. This will produce a parse error when the container runs if not implemented correctly.
Next, input your AWS Access Key ID and AWS Secret Access Key. These need to be generated from your Red Canary > User Profile > Canary Exporter > Generate Credentials section.
You will also need to input your Red Canary API Key in the section titled "red_canary_authentication _token". Your API key can be found by clicking on your user icon (top right of Red Canary dashboard), selecting "View Profile," then scroll down to the "API Authentication Token" section. From here you will need to click on the "Generate New Authentication Token" button. You will be presented with your new API key only once, so be sure to save the key some place safe.
Then, you will need to adjust the "outputs" section. The configuration of this section depends on whether you are using "standardized" or "native" format.
- If you choose "native" format you may want to filter for specific Event Types. To do this you will need to use the EDR product's (i.e Carbon Black, CrowdStrike, etc.) Event Type filters schema.
  - The "native" events stream can be classified as:
    - Raw Endpoint Events (i.e Sensor generated event telemetry)
  - The Carbon Black EDR Event Types filters can be found here: https://developer.carbonblack.com/reference/enterprise-response/event-forwarder/event-schema/.
    - See the article section "Raw endpoint events" for all Event Types filters. The level of filtering is limited to these broader categories listed above.
- If you choose "standardized" format you will need to use the "Canary Exporter Event Types" filters. These are:

binary

child_process

endpoint

file_creation

file_deletion

file_modification

model_attributes

network_connection

process_handle_open

process_thread_open

registry_key_deletion

registry_value_deletion

registry_value_write

registry_key_creation

remote_thread_creation

module_load

process_end

process_start

If you use "native" format the output section should look similar to this:

NOTE: Notice the "/" in front of the "output" directory name. This ensures log data is saved to your local log directory. If the "/" is placed after the output directory name, this will cause Canary Exporter to save logs to the /apps/output directory inside the Docker image):

outputs:
  - file:
      directory: /output 
      max_size_bytes: 10000000
      max_files: 4
      export_only_where:
        type:
          - ingress.event.regmod
          - ingress.event.filemod
          - ingress.event.netconn
          - ingress.event.process 

#NOTE: if you comment out the "export_only_where" and below sections  
or you do NOT specify a "type," Canary Exporter will send ALL Event Types.

if you use the "standardized" format the output section should look similar to this:

outputs:
  - file:
      directory: /output
      max_size_bytes: 10000000
      max_files: 2
      export_only_where:
        event_type_cd:
          - process_start
          - child_process
          - binary   

#NOTE: if you comment out the "export_only_where" and below sections 
or you do NOT specify a "type," Canary Exporter will send ALL Event Types.

6. Last step is to run the Docker image and mount your local volume to the Docker image outputs directory. Executing the following command will automatically download the Canary Exporter Docker image, mount your local volume, and run the application.

Run the following command:

"docker run" command for Linux. NOTE: This will output the logs to default local directory ($HOME/canary-exporter/output):

~# docker run -it --volume $HOME/canary-exporter/config.yaml:/config.yaml  --volume $HOME/canary-exporter/output:/output  redcanary/canary-exporter-ruby

"docker run" command for Windows systems. NOTE: This will output the logs to the default local directory (%HOMEPATH%\canary-exporter\output):

docker run -it --volume C:\Users\"<User Name>"\canary-exporter\config.yaml:/config.yaml --volume C:\Users\"<User Name>"\canary-exporter\output:/output redcanary/canary-exporter-ruby

NOTE: the "-it" switch means: "interactive" + "Allocate a psuedo TTY" mode which basically turns the docker container into an interactive terminal session. The output of that session is printed to std out (i.e the terminal) and waits for any commands from the terminal.

You can also use the "-d" (detach) switch. This basically does the opposite of the "-it" switch; It will run the docker container silently in the background and will print the container ID to std out (i.e the terminal).

Example:

~# docker run -dit --volume $HOME/....

NOTE: You can change the location of the output directory by changing the second --volume value for /output as needed.

Example:

~# docker run -it --volume $HOME/canary-exporter/config.yaml:/config.yaml  --volume $HOME/Documents/custom_file/output:/output  redcanary/canary-exporter-ruby