Getting-started with Sysmon for Linux

14 min readOct 24, 2021

Introduction

Who didn’t played with Sysinternals (by @Mark Russinovich ) tools in the old good days when troubleshooting Windows issues. I loved using Filemon and Regmon for troubleshooting Windows NT or 200[0,3] problems and identifying bottlenecks. Unfortunately later these famous Sysinternals tools where replaced in September 2014 by Process Explorer (aka procexp), but also launched Sysmon. After 12 years we can welcome Sysmon again, but now on Linux and powered by eBPF. That’s awesome !!!

I couldn’t wait to try-out this new tool, so started directly. Sysmon can be seen as a competitor in the Security Audit domain, with tools like Sysdig Falco. Question here is what are the strengths and weaknesses i can currently spot.

After reading the excellent README.md it was mentioned to start with building and installing the SysinternalsEBPF package, which can be found on the Sysinternals Github.

SysinternalsEBPF Library

Also here we found a well-explained README.md, so we quickly started with preparing the Clang/LLVM build environment and cloning the repository on my clean Linux VM running Ubuntu(Bionic 18.04 LTS).

sudo apt update
sudo apt install build-essential gcc g++ make cmake libelf-dev llvm clang libzstd1 git libjson-glib-dev

After these steps we go into the cloned directory (SysinternalsEBPF) and create a build directory. Let’s now run ‘cmake ..’

cmake ..
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1") 
-- Checking for one of the modules 'json-glib-1.0'
-- Configuring done
-- Generating done
-- Build files have been written to: /home/cloud_user/SysinternalsEBPF/build

Next step is to run ‘make’ to do the actual Build.

cloud_user@b7eee0a0e81c:~/SysinternalsEBPF/build$ make
Scanning dependencies of target libbpf
[  2%] Creating directories for 'libbpf'
[  4%] Performing download step (git clone) for 'libbpf'
Cloning into 'libbpf'...
Note: checking out 'v0.1.0'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
  git checkout -b <new-branch-name>
HEAD is now at e8547bd vmtests: fix selftests checkout script
[  7%] No patch step for 'libbpf'
[  9%] Performing update step for 'libbpf'
[ 12%] No configure step for 'libbpf'
[ 14%] Performing build step for 'libbpf'
[ 17%] No install step for 'libbpf'
[ 19%] Completed 'libbpf'
[ 19%] Built target libbpf
[ 21%] Extracting unameOffsets.c
[ 24%] Extracting sysinternalsEBPFoffsets.h
[ 26%] Extracting unameOffsets.h
Scanning dependencies of target sysinternalsEBPF
[ 29%] Building C object CMakeFiles/sysinternalsEBPF.dir/telemetryLoader.c.o
[ 31%] Building C object CMakeFiles/sysinternalsEBPF.dir/discoverOffsets.c.o
[ 34%] Building C object CMakeFiles/sysinternalsEBPF.dir/searchOffsets.c.o
[ 36%] Building C object CMakeFiles/sysinternalsEBPF.dir/unameOffsets.c.o
[ 39%] Building C object CMakeFiles/sysinternalsEBPF.dir/installer.c.o
[ 41%] Building C object CMakeFiles/sysinternalsEBPF.dir/hexdump.c.o
[ 43%] Linking C shared library libsysinternalsEBPF.so
Building EBPF object sysinternalsEBPFmemDump.o
Building EBPF object sysinternalsEBPFrawSock.o
[ 43%] Built target sysinternalsEBPF
[ 46%] Packing libsysinternalsEBPF.so into libsysinternalsEBPF.so.o
[ 48%] Packing libsysinternalsEBPF.h into libsysinternalsEBPF.h.o
[ 51%] Packing sysinternalsEBPFshared.h into sysinternalsEBPFshared.h.o
[ 53%] Packing sysinternalsEBPFoffsets.h into sysinternalsEBPFoffsets.h.o
[ 56%] Packing sysinternalsEBPFmemDump.o into sysinternalsEBPFmemDump.o.o
[ 58%] Packing sysinternalsEBPFrawSock.o into sysinternalsEBPFrawSock.o.o
[ 60%] Packing offsets/offsets.json into offsets-offsets.json.o
[ 63%] Packing getOffsets/LICENSE into getOffsets-LICENSE.o
[ 65%] Packing getOffsets/Makefile into getOffsets-Makefile.o
[ 68%] Packing getOffsets/README.md into getOffsets-README.md.o
[ 70%] Packing getOffsets/extractOffsets.c into getOffsets-extractOffsets.c.o
[ 73%] Packing getOffsets/getOffsets.c into getOffsets-getOffsets.c.o
[ 75%] Packing getOffsets/mount.h into getOffsets-mount.h.o
[ 78%] Packing ebpfKern/LICENSE into ebpfKern-LICENSE.o
[ 80%] Packing ebpfKern/sysinternalsEBPF_common.h into ebpfKern-sysinternalsEBPF_common.h.o
[ 82%] Packing ebpfKern/sysinternalsEBPF_helpers.c into ebpfKern-sysinternalsEBPF_helpers.c.o
[ 85%] Packing /home/cloud_user/SysinternalsEBPF/build/libbpf/src/libbpf/LICENSE.LPGL-2.1 into libbpf-LICENSE.LPGL-2.1.o
[ 87%] Packing /home/cloud_user/SysinternalsEBPF/build/libbpf/src/libbpf/src/bpf_helpers.h into libbpf-src-bpf_helpers.h.o
[ 90%] Packing /home/cloud_user/SysinternalsEBPF/build/libbpf/src/libbpf/src/bpf_helper_defs.h into libbpf-src-bpf_helper_defs.h.o
Scanning dependencies of target libsysinternalsEBPFinstaller
[ 92%] Building C object CMakeFiles/libsysinternalsEBPFinstaller.dir/libsysinternalsEBPFinstaller.c.o
[ 95%] Building C object CMakeFiles/libsysinternalsEBPFinstaller.dir/installer.c.o
[ 97%] Linking C executable libsysinternalsEBPFinstaller
[100%] Built target libsysinternalsEBPFinstaller

After some minutes the build will be completed and you are ready to install the actual lib-extension with cmake or using a portable installer called ‘libsysinternalsEBPFinstaller’. During the example below we use cmake for the actual installation.

cloud_user@b7eee0a0e81c:~/SysinternalsEBPF/build$ sudo make install 
[  2%] Performing update step for 'libbpf'
[  4%] No configure step for 'libbpf'
[  7%] Performing build step for 'libbpf'
[  9%] No install step for 'libbpf'
[ 12%] Completed 'libbpf'
[ 19%] Built target libbpf
[ 43%] Built target sysinternalsEBPF
[100%] Built target libsysinternalsEBPFinstaller
Install the project...
-- Install configuration: ""
-- Installing: /opt/sysinternalsEBPF/ebpfKern/LICENSE
-- Installing: /opt/sysinternalsEBPF/ebpfKern/sysinternalsEBPF_common.h
-- Installing: /opt/sysinternalsEBPF/ebpfKern/sysinternalsEBPF_helpers.c
-- Installing: /opt/sysinternalsEBPF/ebpfKern/sysinternalsEBPFshared.h
-- Installing: /opt/sysinternalsEBPF/ebpfKern/sysinternalsEBPFoffsets.h
-- Installing: /opt/sysinternalsEBPF/getOffsets/LICENSE
-- Installing: /opt/sysinternalsEBPF/getOffsets/Makefile
-- Installing: /opt/sysinternalsEBPF/getOffsets/README.md
-- Installing: /opt/sysinternalsEBPF/getOffsets/extractOffsets.c
-- Installing: /opt/sysinternalsEBPF/getOffsets/getOffsets.c
-- Installing: /opt/sysinternalsEBPF/getOffsets/mount.h
-- Installing: /opt/sysinternalsEBPF/libbpf/LICENSE.LPGL-2.1
-- Installing: /opt/sysinternalsEBPF/libbpf/bpf_helpers.h
-- Installing: /opt/sysinternalsEBPF/libbpf/bpf_helper_defs.h
-- Installing: /usr/local/lib/libsysinternalsEBPF.so
-- Installing: /usr/local/include/libsysinternalsEBPF.h
-- Installing: /opt/sysinternalsEBPF/offsets.json
-- Installing: /opt/sysinternalsEBPF/sysinternalsEBPFmemDump.o
-- Installing: /opt/sysinternalsEBPF/sysinternalsEBPFrawSock.o
-- Installing: /opt/sysinternalsEBPF/libsysinternalsEBPFinstaller

As we mentioned above, we need this lib to be available at ‘/opt/sysinternalsEBPF` to successfully compile Sysmon for Linux.

Optionally we can now build a distributable package, in our case a DEB package. Take into account that during installation we expect eBPF is enabled and is compatible with your build.

cloud_user@b7eee0a0e81c:~/SysinternalsEBPF/build$ make packages
[  2%] Performing update step for 'libbpf'
[  4%] No configure step for 'libbpf'
[  7%] Performing build step for 'libbpf'
[  9%] No install step for 'libbpf'
[ 12%] Completed 'libbpf'
[ 19%] Built target libbpf
[ 43%] Built target sysinternalsEBPF
[100%] Built target libsysinternalsEBPFinstaller
Scanning dependencies of target packages
dpkg-deb: building package 'sysinternalsebpf' in 'sysinternalsebpf_1.0.0-1_amd64.deb'.
No rpmbuild found
[100%] Built target packagescloud_user@b7eee0a0e81c:~/SysinternalsEBPF/build$ find . -name *.deb
./deb/sysinternalsebpf_1.0.0-1_amd64.deb

Now we have successfully build a deb file called ‘sysinternalsebpf_1.0.0–1_amd64.deb`.

Setup Sysmon for Linux

Now let’s start with Sysmon for Linux. As implemented above, we have one important prerequisite in-place. Here we also go further with the README.md.

Mono & Google Test dependencies

Also here we ensure that all required packages are installed to do a successful build. In this case we require certain Mono project (& MonoDevelop) dependencies.Mono is the OSS project for .NET. Next to this we need GoogleTest for testing and Mocking services. That’s why we are going to add an additional repo. See the commands below. It could be that some packages like gcc, llvm, clang and more where already installed in the previous step before.

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831E
echo "deb https://download.mono-project.com/repo/ubuntu vs-bionic main" | sudo tee /etc/apt/sources.list.d/mono-official-vs.list
sudo apt update
sudo apt install build-essential gcc g++ make cmake libelf-dev llvm clang libxml2 libxml2-dev libzstd1 git libgtest-dev apt-transport-https dirmngr monodevelop googletest google-mock libjson-glib-devF

During the actual installation of the dependency packages you might see that certain Mono packages are precompiled before installation. I’ve noticed that this can take more time (up to 10 minutes) then expected, especially on 1 vCPU VMs. Below the expected output during installation.

......
Certificate added: C=RO, O=CERTSIGN SA, OU=certSIGN ROOT CA G
Certificate added: C=HU, L=Budapest, O=Microsec Ltd., OID.2.5.4.97=VATHU-23584497, CN=e-Szigno Root CA 2017
Certificate added: C=US, OU=emSign PKI, O=eMudhra Inc, CN=emSign ECC Root CA - C3
Certificate added: C=IN, OU=emSign PKI, O=eMudhra Technologies Limited, CN=emSign ECC Root CA - G3
Certificate added: C=US, OU=emSign PKI, O=eMudhra Inc, CN=emSign Root CA - C1
Certificate added: C=IN, OU=emSign PKI, O=eMudhra Technologies Limited, CN=emSign Root CA - G1
129 new root certificates were added to your trust store.
Import process completed.
Done
done.
Processing triggers for mime-support (3.60ubuntu1) ...
Processing triggers for ureadahead (0.100.0-21) ...
Setting up mono-devel (6.12.0.122-0xamarin1+ubuntu1804b1) ...
update-alternatives: using /usr/bin/mono-csc to provide /usr/bin/cli-csc (c-sharp-compiler) in auto mode
update-alternatives: using /usr/bin/resgen to provide /usr/bin/cli-resgen (resource-file-generator) in auto mode
update-alternatives: using /usr/bin/al to provide /usr/bin/cli-al (assembly-linker) in auto mode
update-alternatives: using /usr/bin/sn to provide /usr/bin/cli-sn (strong-name-tool) in auto mode
Setting up mono-roslyn (6.12.0.122-0xamarin1+ubuntu1804b1) ...
Mono precompiling /usr/lib/mono/4.5/csc.exe for amd64 (trying with LLVM, this might take a few minutes)...
Mono precompiling /usr/lib/mono/4.5/vbc.exe for amd64 (trying with LLVM, this might take a few minutes)...
Mono precompiling /usr/lib/mono/4.5/VBCSCompiler.exe for amd64 (trying with LLVM, this might take a few minutes)...
Mono precompiling /usr/lib/mono/4.5/Microsoft.CodeAnalysis.CSharp.dll for amd64 (trying with LLVM, this might take a few minutes)...
Mono precompiling /usr/lib/mono/4.5/Microsoft.CodeAnalysis.VisualBasic.dll for amd64 (trying with LLVM, this might take a few minutes)...
Mono precompiling /usr/lib/mono/4.5/Microsoft.CodeAnalysis.dll for amd64 (trying with LLVM, this might take a few minutes)...
Mono precompiling /usr/lib/mono/4.5/System.Collections.Immutable.dll for amd64 (trying with LLVM, this might take a few minutes)...
Mono precompiling /usr/lib/mono/4.5/System.Reflection.Metadata.dll for amd64 (trying with LLVM, this might take a few minutes)...
Setting up fsharp (5.0.0.0-0xamarin15+ubuntu1804b1) ...
* Installing 2 assemblies from fsharp into Mono framework paths
Setting up monodevelop (7.8.4.1-0xamarin6+ubuntu1804b1) ...
Processing triggers for libgdk-pixbuf2.0-0:amd64 (2.36.11-2) ...
ubuntu@ip-172-31-26-179:~$ cd /usr/src/googletest

After waiting for some minutes we can go into the ‘/usr/src/googletest’ directory, creating the build directory (with correct permissions) and running build commands again like ‘cmake’ and ‘make’.

ubuntu@ip-172-31-26-179:/usr/src/googletest/build$ cmake .
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: /usr/bin/python (found version "2.7.17") 
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Configuring done
-- Generating done
-- Build files have been written to: /usr/src/googletest/build.

Now run our ‘make’.

ubuntu@ip-172-31-26-179:/usr/src/googletest/build$ make
Scanning dependencies of target gmock_main
[  9%] Building CXX object googlemock/CMakeFiles/gmock_main.dir/__/googletest/src/gtest-all.cc.o
[ 18%] Building CXX object googlemock/CMakeFiles/gmock_main.dir/src/gmock-all.cc.o
[ 27%] Building CXX object googlemock/CMakeFiles/gmock_main.dir/src/gmock_main.cc.o
[ 36%] Linking CXX static library libgmock_main.a
[ 36%] Built target gmock_main
Scanning dependencies of target gmock
[ 45%] Building CXX object googlemock/CMakeFiles/gmock.dir/__/googletest/src/gtest-all.cc.o
[ 54%] Building CXX object googlemock/CMakeFiles/gmock.dir/src/gmock-all.cc.o
[ 63%] Linking CXX static library libgmock.a
[ 63%] Built target gmock
Scanning dependencies of target gtest
[ 72%] Building CXX object googlemock/gtest/CMakeFiles/gtest.dir/src/gtest-all.cc.o
[ 81%] Linking CXX static library libgtest.a
[ 81%] Built target gtest
Scanning dependencies of target gtest_main
[ 90%] Building CXX object googlemock/gtest/CMakeFiles/gtest_main.dir/src/gtest_main.cc.o
[100%] Linking CXX static library libgtest_main.a
[100%] Built target gtest_main

At this moment we almost have everything set ready to build Sysmon for Linux. First we will install the build above with ‘sudo make install’.

Sysmon for Linux build

Now that we have all dependencies we can build Sysmon for Linux. This can be easily done by completing our known step by cloning the repository ( we need to include — recurse-submodules), creating a build directory, run ‘cmake’ and do the actual build with ‘make’.

we can eventually run a series of unit tests by executing ‘./sysmonUnitTests’. Below a snippet of the end result, passing 19 tests for the 4 available test cases.

</EventData></Event>'
[       OK ] Events.DispatchEvent (13 ms)
[ RUN      ] Events.TranslateNULLSid
[       OK ] Events.TranslateNULLSid (0 ms)
[----------] 2 tests from Events (13 ms total)
[----------] Global test environment tear-down
[==========] 19 tests from 4 test cases ran. (719 ms total)
[  PASSED  ] 19 tests.0

Using Sysmon for Linux

First glance

Great. So we are now ready to install Sysmon for Linux on our local Ubuntu Instance. Let’s first look into the usage instructions.

ubuntu@ip-172-31-26-179:~/SysmonForLinux/build$ sudo ./sysmon -?
Sysmon v1.0.0 - Monitors system events
Sysinternals - www.sysinternals.com
By Mark Russinovich, Thomas Garnier and Kevin Sheldrake
Copyright (C) 2014-2021 Microsoft Corporation
Using libxml2. libxml2 is Copyright (C) 1998-2012 Daniel Veillard. All Rights Reserved.
Usage:
Install:                 sysmon -i [<configfile>]
Update configuration:    sysmon -c [<configfile>]
Print schema:            sysmon -s
Uninstall:               sysmon -u [force]
  -c   Update configuration of an installed Sysmon driver or dump the
       current configuration if no other argument is provided. Optionally
       take a configuration file.
  -i   Install service and driver. Optionally take a configuration file.
  -s   Print configuration schema definition of the specified version.
       Specify 'all' to dump all schema versions (default is latest)).
  -u   Uninstall service and driver. Adding force causes uninstall to proceed
       even when some components are not installed.
The service logs events immediately and the driver installs as a boot-start driver to capture activity from early in the boot that the service will write to the event log when it starts.
On Linux, events are stored in the Syslog, often found at /var/log/syslog.
Use the '-? config' command for configuration file documentation. More examples are available on the Sysinternals website.
Specify -accepteula to automatically accept the EULA on installation.
Neither install nor uninstall requires a reboot.

Take extra notice of the reboot message, yes for Linux we don’t need this off course :)

Here we learn that we can install Sysmon as a service with ‘-i’ flag, so it logs everything by default to ‘/var/log/syslog’. Also it’s recommended to automatically accept the EULA with ‘-accepteula’ flag.

Setup sysmon service

During our spike we can use the default configuration, so let’s do the actual installation. The installation creates a Systemd service, which will run Sysmon as daemon in the background.

sudo ./sysmon -accepteula
sudo ./sysmon -i

Now let’s check if the service is running correctly and look if our ‘/var/log/syslog’ is being populated.

ubuntu@ip-172-31-26-179:~$ systemctl status sysmon
● sysmon.service - Sysmon event logger
   Loaded: loaded (/etc/systemd/system/sysmon.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2021-10-15 18:48:26 UTC; 3h 3min ago
 Main PID: 16677 (sysmon)
    Tasks: 1 (limit: 4686)
   CGroup: /system.slice/sysmon.service
           └─16677 /opt/sysmon/sysmon -i /opt/sysmon/config.xml -service
Oct 15 21:49:07 ip-172-31-26-179 sysmon[16677]: <Event><System><Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/><EventID>5</EventID><Version>3</Version><Level>4</Level><Task
Oct 15 21:50:08 ip-172-31-26-179 sysmon[16677]: <Event><System><Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/><EventID>1</EventID><Version>5</Version><Level>4</Level><Task
Oct 15 21:50:08 ip-172-31-26-179 sysmon[16677]: <Event><System><Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/><EventID>5</EventID><Version>3</Version><Level>4</Level><Task
Oct 15 21:51:09 ip-172-31-26-179 sysmon[16677]: <Event><System><Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/><EventID>1</EventID><Version>5</Version><Level>4</Level><Task
Oct 15 21:51:09 ip-172-31-26-179 sysmon[16677]: <Event><System><Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/><EventID>5</EventID><Version>3</Version><Level>4</Level><Task
Oct 15 21:51:51 ip-172-31-26-179 sysmon[16677]: <Event><System><Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/><EventID>5</EventID><Version>3</Version><Level>4</Level><Task
Oct 15 21:52:02 ip-172-31-26-179 sysmon[16677]: <Event><System><Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/><EventID>1</EventID><Version>5</Version><Level>4</Level><Task
Oct 15 21:52:02 ip-172-31-26-179 sysmon[16677]: <Event><System><Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/><EventID>5</EventID><Version>3</Version><Level>4</Level><Task
Oct 15 21:52:09 ip-172-31-26-179 sysmon[16677]: <Event><System><Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/><EventID>1</EventID><Version>5</Version><Level>4</Level><Task
Oct 15 21:52:09 ip-172-31-26-179 sysmon[16677]: <Event><System><Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/><EventID>1</EventID><Version>5</Version><Level>4</Level><Task

The service is running as expected.

Validate auditing of Sysmon

Let’s try to trigger the auditing part by doing these simple commands below for creating a file and setting certain permissions.

cd /tmp
touch help
chmod 400 help
grep help /var/log/syslog

Now grep for help in ‘/var/log/syslog’ and we directly see our related log entries created by Sysmon.

Oct 15 18:50:27 ip-172-31-26-179 sysmon: <Event><System><Provider> Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/><EventID>1</EventID><Version>5</Version><Level>4</Level><Task>1</Task><Opcode>0</Opcode><Keywords>0x8000000000000000</Keywords><TimeCreated SystemTime="2021-10-15T18:50:27.552794000Z"/><EventRecordID>43</EventRecordID><Correlation/><Execution ProcessID="16677" ThreadID="16677"/><Channel>Linux-Sysmon/Operational</Channel><Computer>ip-172-31-26-179</Computer><Security UserId="0"/></System><EventData><Data Name="RuleName">-</Data><Data Name="UtcTime">2021-10-15 18:50:27.562</Data><Data Name="ProcessGuid">{4007bb44-cd73-6169-1090-8063b4550000}</Data><Data Name="ProcessId">16706</Data><Data Name="Image">/bin/touch</Data><Data Name="FileVersion">-</Data><Data Name="Description">-</Data><Data Name="Product">-</Data><Data Name="Company">-</Data><Data Name="OriginalFileName">-</Data><Data Name="CommandLine">touch help</Data><Data Name="CurrentDirectory">/tmp</Data><Data Name="User">ubuntu</Data><Data Name="LogonGuid">{4007bb44-c8ba-6169-e803-000000000000}</Data><Data Name="LogonId">1000</Data><Data Name="TerminalSessionId">1</Data><Data Name="IntegrityLevel">no level</Data><Data Name="Hashes">-</Data><Data Name="ParentProcessGuid">{00000000-0000-0000-0000-000000000000}</Data><Data Name="ParentProcessId">2035</Data><Data Name="ParentImage">-</Data><Data Name="ParentCommandLine">-</Data><Data Name="ParentUser">-</Data></EventData></Event
Oct 15 18:50:36 ip-172-31-26-179 sysmon: <Event><System><Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/><EventID>1</EventID><Version>5</Version><Level>4</Level><Task>1</Task><Opcode>0</Opcode><Keywords>0x8000000000000000</Keywords><TimeCreated SystemTime="2021-10-15T18:50:36.098585000Z"/><EventRecordID>47</EventRecordID><Correlation/><Execution ProcessID="16677" ThreadID="16677"/><Channel>Linux-Sysmon/Operational</Channel><Computer>ip-172-31-26-179</Computer><Security UserId="0"/></System><EventData><Data Name="RuleName">-</Data><Data Name="UtcTime">2021-10-15 18:50:36.107</Data><Data Name="ProcessGuid">{4007bb44-cd7c-6169-e0f1-c09b16560000}</Data><Data Name="ProcessId">16709</Data><Data Name="Image">/bin/chmod</Data><Data Name="FileVersion">-</Data><Data Name="Description">-</Data><Data Name="Product">-</Data><Data Name="Company">-</Data><Data Name="OriginalFileName">-</Data><Data Name="CommandLine">chmod 400 help</Data><Data Name="CurrentDirectory">/tmp</Data><Data Name="User">ubuntu</Data><Data Name="LogonGuid">{4007bb44-c8ba-6169-e803-000000000000}</Data><Data Name="LogonId">1000</Data><Data Name="TerminalSessionId">1</Data><Data Name="IntegrityLevel">no level</Data><Data Name="Hashes">-</Data><Data Name="ParentProcessGuid">{00000000-0000-0000-0000-000000000000}</Data><Data Name="ParentProcessId">2035</Data><Data Name="ParentImage">-</Data><Data Name="ParentCommandLine">-</Data><Data Name="ParentUser">-</Data></EventData></Event>
Oct 15 18:51:10 ip-172-31-26-179 sysmon: <Event><System><Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/><EventID>1</EventID><Version>5</Version><Level>4</Level><Task>1</Task><Opcode>0</Opcode><Keywords>0x8000000000000000</Keywords><TimeCreated SystemTime="2021-10-15T18:51:10.736735000Z"/><EventRecordID>56</EventRecordID><Correlation/><Execution ProcessID="16677" ThreadID="16677"/><Channel>Linux-Sysmon/Operational</Channel><Computer>ip-172-31-26-179</Computer><Security UserId="0"/></System><EventData><Data Name="RuleName">-</Data><Data Name="UtcTime">2021-10-15 18:51:10.746</Data><Data Name="ProcessGuid">{4007bb44-cd9e-6169-509c-790e2a560000}</Data><Data Name="ProcessId">16713</Data><Data Name="Image">/bin/grep</Data><Data Name="FileVersion">-</Data><Data Name="Description">-</Data><Data Name="Product">-</Data><Data Name="Company">-</Data><Data Name="OriginalFileName">-</Data><Data Name="CommandLine">grep --color=auto help /var/log/syslog</Data><Data Name="CurrentDirectory">/tmp</Data><Data Name="User">ubuntu</Data><Data Name="LogonGuid">{4007bb44-c8ba-6169-e803-000000000000}</Data><Data Name="LogonId">1000</Data><Data Name="TerminalSessionId">1</Data><Data Name="IntegrityLevel">no level</Data><Data Name="Hashes">-</Data><Data Name="ParentProcessGuid">{00000000-0000-0000-0000-000000000000}</Data><Data Name="ParentProcessId">2035</Data><Data Name="ParentImage">-</Data><Data Name="ParentCommandLine">-</Data><Data Name="ParentUser">-</Data></EventData></Event>

Looking through these XML parsed entries isn’t really Linux admin friendly. To overcome this Sysmon bundles with a tool called ‘sysmonLogView’ to parse these entries more user friendly. Let’s dive into that later.

Sysmon configuration

Seems that everything works, but what default config?

First impression was that we could use the “-s [all]” to print the active or full configuration, but this isn’t the case. Exporting this config wasn’t directly usable as input XML.

Inside the ‘/opt/sysmon’ directory we finally found a config.xml, it actually seems pretty empty.

Example default config.xml<Sysmon schemaversion="4.22"
<EventFiltering>
</EventFiltering>
</Sysmon>>

In the config we can define EventFilters, we can find some general documentation here.

Luckily I also found another article on the Microsoft Tech Community, which is a complete-collect-all configuration example.

all-in-one.xml<Sysmon schemaversion="4.70">
	  <EventFiltering>
	    <!-- Event ID 1 == ProcessCreate. Log all newly created processes -->
	    <RuleGroup name="" groupRelation="or">
	      <ProcessCreate onmatch="exclude"/>
	    </RuleGroup>
	    <!-- Event ID 3 == NetworkConnect Detected. Log all network connections -->
	    <RuleGroup name="" groupRelation="or">
	      <NetworkConnect onmatch="exclude"/>
	    </RuleGroup>
	    <!-- Event ID 5 == ProcessTerminate. Log all processes terminated -->
	    <RuleGroup name="" groupRelation="or">
	      <ProcessTerminate onmatch="exclude"/>
	    </RuleGroup>
	    <!-- Event ID 9 == RawAccessRead. Log all raw access read -->
	    <RuleGroup name="" groupRelation="or">
	      <RawAccessRead onmatch="exclude"/>
	    </RuleGroup>
	    <!-- Event ID 10 == ProcessAccess. Log all open process operations -->
	    <RuleGroup name="" groupRelation="or">
	      <ProcessAccess onmatch="exclude"/>
	    </RuleGroup>
	    <!-- Event ID 11 == FileCreate. Log every file creation -->
	    <RuleGroup name="" groupRelation="or">
	      <FileCreate onmatch="exclude"/>
	    </RuleGroup>
	    <!--Event ID 23 == FileDelete. Log all files being deleted -->
	    <RuleGroup name="" groupRelation="or">
	      <FileDelete onmatch="exclude"/>
	    </RuleGroup>
	  </EventFiltering>
	</Sysmon>

Credits to @Cyb3rWard0g

We can add this configuration easily loading this ‘all-in-one.xml’ by applying ‘sysmon -c all-in-one.xml’ on the CLI.

sudo ./sysmon -c all-in-one.xml

Hopefully in the coming months we can expect some updates in the well-known SwiftOnSecurity example Sysmon configuration, currently tailed for Sysmon on Windows.

Using sysmonLogView

As seen above reading XML messages isn’t that user friendly. To overcome this issue they have bundled Sysmon with sysmonLogView, which is the actual log viewer.

Let’s try out sysmonLogView by the following command.

sudo tail -f /var/log/syslog | sudo /opt/sysmon/sysmonLogView
Event SYSMONEVENT_PROCESS_TERMINATE
	RuleName: -
	UtcTime: 2021-10-15 20:13:10.015
	ProcessGuid: {4007bb44-e0d6-6169-0024-b3c5d0550000}
	ProcessId: 16952
	Image: /usr/bin/tail
	User: root
Event SYSMONEVENT_PROCESS_TERMINATE
	RuleName: -
	UtcTime: 2021-10-15 20:13:10.016
	ProcessGuid: {4007bb44-e0d6-6169-083e-6a8207560000}
	ProcessId: 16950
	Image: /usr/bin/sudo
	User: root
Event SYSMONEVENT_CREATE_PROCESS
	RuleName: -
	UtcTime: 2021-10-15 20:13:15.554
	ProcessGuid: {4007bb44-e0db-6169-080e-5f7a33560000}
	ProcessId: 16953
	Image: /usr/bin/sudo
	FileVersion: -
	Description: -
	Product: -
	Company: -
	OriginalFileName: -
	CommandLine: sudo tail -f /var/log/syslog
	CurrentDirectory: /home/ubuntu/SysmonForLinux/build
	User: ubuntu
	LogonGuid: {4007bb44-c8ba-6169-e803-000000000000}
	LogonId: 1000
	TerminalSessionId: 1
	IntegrityLevel: no level
	Hashes: -
	ParentProcessGuid: {00000000-0000-0000-0000-000000000000}
	ParentProcessId: 2035
	ParentImage: -
	ParentCommandLine: -
	ParentUser: -
Event SYSMONEVENT_PROCESS_TERMINATE
	RuleName: -
	UtcTime: 2021-10-15 20:13:15.555
	ProcessGuid: {00000000-0000-0000-0000-000000000000}
	ProcessId: 16954
	Image: <unknown process>
	User: ubuntu

We now have a user friendly view on the Events that occurred. We can also use sysmonLogView for further filtering like eventid ( with argument ), certain time window or which fields to display. In our case we could filter out our ‘touch’ events with

sudo tail -f /var/log/syslog | sudo /opt/sysmon/sysmonLogView -f Image=/bin/touch

More information on this tool can be found here on Github.

Current limitations (and links to Github issues)

Use JSON as additional output format, which way we can reuse known tools like ‘jq’ to format the events. Additionally integrations are far much easier.
Output location should be configurable to route the logging to ‘/var/log/sysmon’ instead of polutioning the syslog. Many Linux admins aren’t happy with this approach. Apart from this configuration of log handlers like Fluentd, Fluentbit, Filebeat or Logstash becomes much easier.
Generating configuration should be easier, maybe using the ‘-s all’ flag to dump a full example, which is ready for use.
Add container support or better support for running Sysmon as Daemonset inside your AKS cluster. Currently the use of Systemd is a constraint for building a solid container, instead Sysmon should be able to run interactive on the CLI, like with an additional flag.

Conclusion

Overall it seems a good solution, based on eBPF for auditing your Linux systems, especially if your company is using Azure and/or Azure Sentinel as SIEM. For Windows auditing Sysmon is our preferred choice, but for Linux and Kubernetes?

From this perspective things still seem to be lacking the way how we like to integrate things. Yes we can make things work well, but this requires more maintenance and preparation. In these cases Sysdig Falco is more useful, since it supports JSON output and already has good Kubernetes support.

For now both are on my radar, let’s see how both can improve in there direction providing eBPF based auditing.