Canary Exporter Setup (Example: Collecting Logs in Local File) – Red Canary help

Issue

Use Canary Exporter to send logs a local file on our Server.

Environment

Red Canary

Resolution

Please reference the Red Canary Help article Exporting telemetry from Red Canary.

What is Canary Exporter and what is it used for?

The Canary Exporter is a Docker Container (an application) that allows customers to collect and forward the same raw telemetry data we receive from the EDR platform.

To use Canary Exporter, please perform the following steps:

Install Docker on a local Server (preferably a Linux Server). See Docker install instructions for your OS (found below).
NOTE: Windows VMs have not been tested by Docker. Since the Canary Exporter Docker Image is based on Linux, you should use a Linux or Unix based machine. If you need to have Canary Exporter set up in Windows, please carefully follow the Docker installation instructions for Windows.

Docker install instructions for Windows: https://docs.docker.com/docker-for-windows/install/
- - NOTE: In order for Docker to run successfully on Windows, enable either Hyper-V Windows Features, or Install required Windows components for WSL 2 when prompted during the Docker for Windows installation. If you choose to go with WSL, install the "WSL2 Linux kernel update package for x64 machines" after running the Docker for Windows installer.
  - NOTE: Nested virtualization scenarios, such as running Docker Desktop on a VMWare or Parallels instance might work, but there are no guarantees. For more information, see Running Docker Desktop in nested virtualization scenarios.
  - If you are still having trouble getting Docker for Windows to run, please see the Docker Logs and Troubleshooting article.

Docker install instructions for CentOS 7: https://docs.docker.com/engine/install/centos/

NOTE: There are many other versions of Linux listed in the Docker installation guide. We chose CentOS as an example.

Once Docker is installed, create and configure the local config.yaml file (An example config.yaml file is located in Red Canary > User Profile > Canary Exporter).
- NOTE: The default location to save the config.yaml file in Linux is: $HOME/canary-exporter/config.yaml.
- NOTE: The default location to save the config.yaml file in Windows is: C\User\<username>\canary-exporter\config.yaml.

Once the config file is created, edit the configuration.
- Adjust the Input section, specifically the stream_name: section.
- Decide whether you want to receive native or standardized formatted log data. Native data format is the raw pre-processed event telemetry as we receive it from the EDR product. Standardized data is formatted telemetry data according to Red Canary's standardized format. This format tends to be easier to read and parse, and is product-agnostic.
  - NOTE: Please use double quotes for either the "native" or "standardized" values. Using single quotes or no quotes may cause parse errors while Canary Exporter is running.
Input your AWS Access Key ID and AWS Secret Access Key.
These are generated from your Red Canary > User Profile > Canary Exporter > Generate Credentials. Ensure there are double quotes around these values in the config.yaml file.
Input your Red Canary API Key in the section titled "red_canary_authentication _token".
Your API key can be found by clicking on your user icon (top right of Red Canary dashboard), selecting "View Profile," then scrolling down to the "Generate API Authentication Token" section. Click on the "Generate" button. You will be presented with your new API key only once, so be sure to save the key in a secure location.
Adjust the outputs section.
The configuration of this section depends on whether you are using "standardized" or "native" format.
- If you choose native format you may want to filter for specific Event Types. To do this use the EDR product's (i.e., Carbon Black, CrowdStrike, etc.) Event Type filters schema.
- The "native" events stream can be classified as:
  - Raw Endpoint Events (i.e., Sensor generated event telemetry)
- The Carbon Black EDR Event Types filters can be found here. See the article section "Raw endpoint events" for all Event Types filters. The level of filtering is limited to these broader categories listed above.
- If you choose "standardized" format you will need to use the "Canary Exporter Event Types" filters. These are:
  - - - binary
      - child_process
      - endpoint
      - file_creation
      - file_deletion
      - file_modification
      - model_attributes
      - network_connection
      - process_handle_open
      - process_thread_open
      - registry_key_deletion
      - registry_value_deletion
      - registry_value_write
      - registry_key_creation
      - remote_thread_creation
      - module_load
      - process_end
      - process_start

If you use "native" format the outputs and event type section should look similar to this:
- NOTE: Notice the "/" in front of the "output" directory name below. This ensures log data is saved to your local log directory. If the "/" is placed after the output directory name, this will cause Canary Exporter to save logs to the /apps/output directory inside the Docker image):

outputs:
  - file:
      directory: /output 
      max_size_bytes: 10000000
      max_files: 4
      export_only_where:
        type:
          - ingress.event.regmod
          - ingress.event.filemod
          - ingress.event.netconn
          - ingress.event.process
          - ingress.event.module
          - ingress.event.childproc
          - ingress.event.crossprocopen
 

#NOTE: if you comment out the "export_only_where" and below sections  
or you do NOT specify a "type," Canary Exporter will send ALL Event Types.

If you use the "standardized" format, the outputs and event type section should look similar to this:

outputs:
  - file:
      directory: /output
      max_size_bytes: 10000000
      max_files: 2
      export_only_where:
        event_type_cd:
          - process_start
          - child_process
          - binary   

#NOTE: if you comment out the "export_only_where" and below sections 
or you do NOT specify a "type," Canary Exporter will send ALL Event Types.

NOTE: If you are using Linux EDR, the available event type filters can be found here: Filtering Telemetry for Linux EDR.

Example of what the "config.yaml" file settings should look like:

7. Run the Docker image and mount your local volume to the Docker image outputs directory.
Execute the following command to automatically download the Canary Exporter Docker image, mount your local volume, and run the application.

"docker run" command for Linux. NOTE: This will output the logs to default local directory ($HOME/canary-exporter/output):

~# docker run -it --volume $HOME/canary-exporter/config.yaml:/config.yaml  --volume $HOME/canary-exporter/output:/output  redcanary/canary-exporter-ruby

"docker run" command for Windows systems. NOTE: This will output the logs to the default local directory (%HOMEPATH%\canary-exporter\output):

docker run -it --volume C:\Users\"<User Name>"\canary-exporter\config.yaml:/config.yaml --volume C:\Users\"<User Name>"\canary-exporter\output:/output redcanary/canary-exporter-ruby

- NOTE: the "-it" switch means: "interactive" + "Allocate a psuedo TTY" mode which turns the docker container into an interactive terminal session. The output of that session is printed to std out (i.e., the terminal) and waits for any commands from the terminal.
You can also use the "-d" (detach) switch. This does the opposite of the "-it" switch. It will run the docker container silently in the background and will print the container ID to std out (i.e., the terminal).

- Example:
  - ```
  ~# docker run -dit --volume $HOME/....
```

You can change the location of the output directory by changing the second --volume value for /output as needed.

Example:

~# docker run -it --volume $HOME/canary-exporter/config.yaml:/config.yaml  --volume $HOME/Documents/custom_file/output:/output  redcanary/canary-exporter-ruby