VBEventTrace v2.1/TBEventTrace v2.2.4

Event Tracing fo Windows (ETW) File Activity Monitor, VB6/twinBASIC x64 port

Update (2023 Feb 17): Removed temporary fix as erase bug has been fixed. Corrected sign issues in some hex literals.

Update (2022 Dec 07): Applied temporary fix for twinBASIC bug that results in the ListView being erased on resize. Also, x64 binary no longer has a twinBASIC banner on startup as I'm now proudly supporting the project with a subscription :)

Event Tracing for Windows (ETW) is a notoriously complex and unfriendly API, but it's extremely powerful. It allows access to messages from the NT Kernel Logger, which provides a profound level of detail about activity on the system. It provides details about many types of activity, but this first project will focus on File Activity. I also plan to follow this up with a monitor for TcpIp and Udp connections.

Given the complexity and unfriendliness that's given it the reputation of the world's worst API, why use it? You can find many projects that monitor file activity, using methods like SHChangeNotify, FindFirstChangeNotification, and monitoring open handles. But the reality is these are all high level methods that don't cover quite a bit of activity. The kernel logger shows activity coming from low level disk and file system drivers. This project started with me wanting to know what was causing idle hard drives to spin up, and none of the higher levels methods offered a clue. Programs like ProcessHacker and FileActivityView use the NT Kernel Logger as well, but I wanted two things: Better control over the process, and doing it in VB6. Why? Well, if you've seen my other projects, you know I'm excessively fond of going way beyond what VB6 was meant for both in terms of low level stuff and modern stuff.

Intro

This project tracks most of the DiskIO events and (FileIo events, providing a great deal of control over what events you watch and filtering them to find what you're looking for. It also looks up name and icon of the process that generated the activity (not always available). With no filtering or only light filtering, a tremendous amount of data is generated. The VB TextBox and ListView simply could not keep up with the rapid input, and all sorts of memory and display issues ensued where text and List Items disappeared. So while the project was already complicated to begin with, the only way to cope with this was to use an API-created Virtual ListView (created via API and using the LVS_OWNERDATA style so it only includes the data currently being displayed). The default mode looks only at DiskIO events, which represent physical disk activity, many FileIO events are only on in-memory caches. It's advisable to select 'Supplement', which uses info from FileIO events to correlate process attribution information, which is often missing from events.

This repository includes 2 versions:

The original VB6 version: VB6 is x86 only, so this version was manually adjusted to operate with an x64 kernel.

The twinBASIC version: can target either, but a 32bit build won't work on x64. twinBASIC is a successor to VB6 that's 100% backwards compatible as a goal, and brings in x64 compilation and new language features. It's 99% backwards compatible in language now, which lets this project run with only changing VB6-specific assembly thunks to regular subclassing. GUI and objects have a bit to go, but this project doesn't use much that's not available... see this thread on VBForums for more info. Version 2.2.2 requires twinBASIC Beta 122 or newer to open and build the source code.

How It Works

Have a read here for an introduction to setting up a Kernel Logger with ETW, and then realize it's even more complicated than that article suggests, because of some VB6 specific issues, and the hell on earth involved in interpreting the data.

Just starting the tracing session has 3 steps. You start with the EVENT_TRACE_PROPERTIES structure. Now, it's daunting enough on it's own. But when you read the article linked, you realize you have to have open bytes appended after the structure for Windows to copy the name into. Then the article doesn't touch on a recurring theme that was the source of a massive headache implementing it... in other languages, some ETW structures get automatically aligned along 8 byte intervals (a Byte is 1 byte, an Integer 2 bytes, a Long 4 bytes... alignment is making each of the largest type appear at, and the total size be, a multiple of its size). Not so in VB-- because of an arcane detail of alignment affecting LARGE_INTEGER: declaring it as two Longs is the standard definition, and indeed seems to mostly match the C/C++ def, except that mentions it's a union with a single 8 byte type. This means it triggers 8-byte alignment even if you don't use the QuadPart. Currency wouldn't help here, because VB masks it being a 2x4 byte UDT under the hood. The only true 8 byte type is Double, but that can't be easily substituted for ULONGLONG. It took quite a bit of crashing and failures to realize this, then properly pad the structures. The twinBASIC version of this project uses its LongLong type to remove the need for manual padding. The code uses it's own structure for the StartTrace function that looks like this:

Public Type EtpKernelTrace
    tProp As EVENT_TRACE_PROPERTIES
    LoggerName(0 To 31) As Byte 'LenB(KERNEL_LOGGER_NAMEW)
    padding2(0 To 3) As Byte
End Type

Needed to include 4 bytes of padding after the structure, then add room for the name, then make sure it's all aligned to 8 byte intervals. In the VB6 version, this is done manually because it's targeting x64 Windows... these structures are passed directly to kernel mode WMI modules without being translated through the WOW64 layer. In the twinBASIC version, it's all handled by the compiler depending on whether you build for x86 or x64 Now we're ready to go, with tStruct being a module-level EtpKernelTrace var:

With tStruct.tProp
    .Wnode.Flags = WNODE_FLAG_TRACED_GUID
    .Wnode.ClientContext = 1&
    .Wnode.tGUID = SelectedGuid
    .Wnode.BufferSize = LenB(tStruct)
    .LogFileMode = EVENT_TRACE_REAL_TIME_MODE 'We're interested in doing real time monitoring, as opposed to processing a .etl file.
    If bUseNewLogMode Then
        .LogFileMode = .LogFileMode Or EVENT_TRACE_SYSTEM_LOGGER_MODE
    End If
    'The enable flags tell the system which classes of events we want to receive data for.
    .EnableFlags = EVENT_TRACE_FLAG_DISK_IO Or EVENT_TRACE_FLAG_DISK_FILE_IO Or EVENT_TRACE_FLAG_FILE_IO_INIT Or _
                    EVENT_TRACE_FLAG_DISK_IO_INIT Or EVENT_TRACE_FLAG_FILE_IO Or EVENT_TRACE_FLAG_NO_SYSCONFIG
    .FlushTimer = 1&
    .LogFileNameOffset = 0&
    .LoggerNameOffset = LenB(tStruct.tProp) + 4 'The logger name gets appended after the structure; but the system looks in 8 byte alignments,
                                                'so because of our padding, we tell it to start after an additional 4 bytes.
End With

'We're now ready to *begin* to start the trace. StartTrace is only 1/3rd of the way there...
hr = StartTraceW(gTraceHandle, StrPtr(SelectedName & vbNullChar), tStruct)

This begins to start a trace session. There's SelectedGuid and SelectedName because there's two options here. In Windows 7 and earlier, the name has to be "NT Kernel Logger", and the Guid has to be SystemTraceControlGuid. If you use that method, there can only be 1 such logger running. You have to stop other apps to run yours, and other apps will stop yours when you start them. On Windows 8 and newer, there can be several such loggers, and you supply a custom name and GUID, and inform it you want a kernel logger with the flag added with bUseNewLogMode. This project supports both methods. The EnableFlags are the event providers you want enabled. This project wants the disk and file io ones, but there's many others. Onto step 2...

Dim tLogfile As EVENT_TRACE_LOGFILEW
ZeroMemory tLogfile, LenB(tLogfile)
tLogfile.LoggerName = StrPtr(SelectedName & vbNullChar)
tLogfile.Mode = PROCESS_TRACE_MODE_REAL_TIME Or PROCESS_TRACE_MODE_EVENT_RECORD 'Prior to Windows Vista, EventRecordCallback wasn't available.
tLogfile.EventCallback = FARPROC(AddressOf EventRecordCallback) 'Further down, you can see the prototype for EventCallback for the older version.
gSessionHandle = OpenTraceW(tLogfile)

We have to tell it again we want to use real time mode, not a .etl log file, and at this point we supply a pointer to a callback that receives events. This project uses a newer type of callback available in Vista+, but has prototypes for the older one. Like a WndProc for subclassing, this has to be in a standard module (.bas); to put it in a class module/form/usercontrol, you'd need the kind of self-subclassing code like you find on the main form (but be careful copying/pasting that, it's been slightly modified and only works with Forms).

The final step is a single call: To ProcessTrace. Only then will you begin receiving events. But of course, this simple call couldn't be simple. ProcessTrace doesn't return until all messages have been processed, which in a real-time trace means indefinitely until you shut it off. So if you call it, execution stops. In that thread. In other languages, spinning off a new thread to call ProcessTrace is easy. In VB, it's painful. The VB6 version project makes use of The trick's VbTrickThreading project to launch