FSBroker

FSBroker is a Go library that acts as an intelligent translation layer on top of fsnotify, converting raw file system events into reliable, high-level actions like Create, Write, Rename, and Remove.

Operating systems emit file system events inconsistently and often obscurely. For example:

• A simple file rename might appear as a CREATE followed by a RENAME on one OS, but REMOVE then CREATE on another.
• Moving a file to the trash might look like a RENAME event on some systems.
• Rapid saves can flood your application with WRITE events.
• System-specific files (.DS_Store, thumbs.db) generate unwanted noise.

fsnotify provides the raw events, but leaves the complex interpretation up to you. FSBroker handles this complexity by:

• Batching: Grouping events that occur close together in time.
• Pattern Recognition: Analyzing sequences of raw events (like CREATE+RENAME) within a time window.
• Deduplication: Filtering out redundant events (e.g., multiple WRITEs).
• Contextual Analysis: Considering platform specifics (Windows vs. macOS vs. Linux) and file metadata.

This allows FSBroker to reliably report the actual user actions happening on the filesystem.

Features

• Accurate Event Interpretation: Translates complex, platform-specific fsnotify event sequences into clear Create, Write, Rename, and Remove actions.
• Intelligent Rename/Move Detection: Correctly identifies file/directory renames and moves, even across different watched directories, providing both old and new paths.
• Reliable Deletion Detection: Differentiates between hard deletes (permanent removal) and soft deletes (moves to trash or to unwatched directories), emitting the appropriate Remove event.
• Write Event Deduplication: Consolidates bursts of WRITE events (e.g., rapid saves) into a single Write event.
• macOS File Clear Handling: Detects file clearing operations on macOS (which don't always emit WRITE) by analyzing CHMOD events and file size.
• Noise Reduction: Automatically ignores irrelevant events (like WRITE on directories in Windows) and common system files (e.g., .DS_Store, thumbs.db). Option to ignore all hidden files.
• Automatic Recursive Watching: Automatically adds newly created subdirectories within watched paths to the watcher.
• Custom Event Filtering: Allows pre-filtering of raw fsnotify events based on path or type before processing.
• Optional Chmod Events: Provides an option (EmitChmod) to receive raw CHMOD events if needed.

Supported Platforms

FSBroker is designed and tested to work on the following operating systems:

• Windows
• macOS
• Linux

While manually tested across these platforms, contributions towards better multi-platform testing are welcome (see Missing Features).

Changelog

• (New v1.0.3) Fix: leaky main loop
• (New v1.0.2) Fix: recusive watch now adds initial subdirectories to watch list
• (New v1.0.1) Formatting, remove README pdf for smaller download size.
• (New v1.0.0) First complete release.
• (New v0.1.8) Update fsnotify to v1.9.0
• (New v0.1.7) Update fsnotify to v1.9.0
• (New v0.1.6) Added the option to ignore hidden files.
• (New v0.1.5) Fix more bugs, added the option to emit chmod events (defaults to false).
• (New v0.1.4) Fix a bug where multiple consecutive file creations would not be detected on Windows.
• (New v0.1.4) Introduce the ability to interpret file moves/renames as deletes in cases where its appropriate.
• (New v0.1.3) Fix a problem where fsnotify would not detect file changes on macOS if the file was cleared.

Installation

To install FS Broker, use go get:

go get github.com/helshabini/fsbroker

Usage

Here is an example of how to use FS Broker:

package main

import (
	"log"
	"time"

	"github.com/helshabini/fsbroker"
)

func main() {
	config := fsbroker.DefaultFSConfig()
	broker, err := fsbroker.NewFSBroker(config)
	if err != nil {
		log.Fatalf("error creating FS Broker: %v", err)
	}
	defer broker.Stop()

	if err := broker.AddRecursiveWatch("watch"); err != nil {
		log.Printf("error adding watch: %v", err)
	}

	broker.Start()

	for {
		select {
		case action := <-broker.Next():
			log.Printf("fs action has occurred: type=%s, path=%s, timestamp=%s, properties=%v", 
				action.Type.String(), action.Subject.Path, action.Timestamp.Format(time.RFC3339), action.Properties)
		case err := <-broker.Error():
			log.Printf("an error has occurred: %v", err)
		}
	}
}

You can also apply your own filters to file system actions:

broker.Filter = func(action *FSAction) bool {
    return action.Type != fsbroker.Remove // Filters out any event that is not Remove
}

or

broker.Filter = func(action *FSAction) bool {
    return action.Subject.Path == "/some/excluded/path" // Filters out any event which is related to this path
}

API

DefaultFSConfig() *FSConfig

Creates a default FS Config instance with these default settings:

• Timeout: 300 * time.Millisecond
• IgnoreSysFiles: true
• IgnoreHiddenFiles: true
• EmitChmod: false

NewFSBroker(config *FSConfig) (*FSBroker, error)

Creates a new FS Broker instance.

• config: FS Config instance.

(*FSBroker) AddRecursiveWatch(path string) error

Adds a recursive watch on the specified path. This will:

• Watch the specified directory and all its subdirectories

• Automatically add watches for newly created subdirectories

• Add all existing files and directories to the watch map

• path: The directory path to watch.

(*FSBroker) AddWatch(path string) error

Adds a watch on a specific file or directory. This will:

• Watch the specified path

• Add all existing files in the directory to the watch map (if path is a directory)

• Not automatically watch new subdirectories (use AddRecursiveWatch for that)

• path: The file or directory path to watch.

(*FSBroker) RemoveWatch(path string)

Removes a watch on a file or directory. This will:

• Remove the watch from fsnotify

• Remove the path and all its subpaths from the watch map

• path: The file or directory path to stop watching.

(*FSBroker) Start()

Starts the FS Broker to begin monitoring file system events. This will:

• Start the event processing loop
• Begin listening for fsnotify events
• Process and emit events through the Next() channel

(*FSBroker) Stop()

Stops the FS Broker. This will:

• Stop the event processing loop
• Close the fsnotify watcher
• Clean up all resources

(*FSBroker) Next() <-chan *FSAction

Returns a channel that receives processed file system events. Each event is an FSAction that represents a high-level file system operation (Create, Write, Rename, or Remove).

(*FSBroker) Error() <-chan error

Returns a channel that receives errors that occur during file system watching or event processing.

(*FSBroker) Filter func(*FSAction) bool

A function that can be set to filter events before they are emitted through the Next() channel. Return true to allow the event to be emitted, false to filter it out.

Debugging

FSBroker uses the standard log/slog package for logging. By default, only messages with INFO level or higher are shown.

To enable detailed debug logging (using slog.LevelDebug), compile or test the package with the fsbroker_debug build tag. This enables verbose logging of internal event processing steps, which can be helpful for troubleshooting.

Examples:

# Run tests with debug logging enabled
go test -v -tags fsbroker_debug ./...

# Build your application with debug logging enabled
go build -tags fsbroker_debug -o myapp main.go

When the tag is not provided, the slog level defaults to INFO, and the slog.Debug calls in the code have minimal performance impact.

Missing features:

Here is a list of features I would like to add in the future, please feel free to submit pull requests:

• Testing on different operating systems

Currently, existing automated tests may give a false negative due to concurrency issues with the way tests are performed. Any contribution to the testing methodology is welcomed.

FAQ

• How does fsbroker work and what does it offer as opposed to fsnotify?

  fsbroker depends on fsnotify as an underlying file system event watcher. However, fsnotify expects you to make sense of the events it emits according to your platform and your use case. Unfortunately, because of how messy file system events are in various operating systems, it is diffucilt to make sense of them.

  It works by accumliating events emitted from fsnotify in an event queue, then process them in batches to try and make sense of what actually happened on the file system.

• Does fsbroker have any overhead on top of fsnotify?

  Yes. To be able to accurately make sense of what happened on the file system, fsbroker needs to keep an in-memory map of your watched directories. We've tried to keep it as small as possible, but it still grows as large as your number of files being watched.

  In addition, fsbroker may perform some additional file checks and syscalls to verify file attributes or existance on disk.

  In my view, these are checks/overheads you would have had to deal with in your code anyway. They are also fairly minimal. So I consider it a very good trade-off.

• How accurate is fsbroker?

  It is not a 100%. But good enough for most use cases, and certainly far better than what fsnotify offers. The accuracy is directly propotional to the timeout interval you set in fsbroker configuration. The more you wait for batching events together, the better your accuracy will be.

• Why is the accuracy of event detection not a 100%?

  There are two main reasons

  • fsbroker depends on accumilating a bunch of fsnotify events togther to try and make sense of what actually happened. Because of the nat