DotNetWorkQueue

A producer / distributed consumer library for .NET applications. Targets .NET 4.8, .NET 8.0, .NET 10.0, and .NET Standard 2.0.

High-level features:

• Queue / de-queue POCOs for distributed processing
• Queue / process LINQ statements (compiled or dynamic, expressed as a string)
• Re-occurring job scheduler

See the Wiki for in-depth documentation.

---

Installation

| Package | Description | NuGet | |---------|-------------|-------| | DotNetWorkQueue | Core library with all abstractions, interfaces, and default implementations | |

Transports

| Package | Description | NuGet | |---------|-------------|-------| | DotNetWorkQueue.Transport.SqlServer | SQL Server transport using Microsoft.Data.SqlClient | | | DotNetWorkQueue.Transport.PostgreSQL | PostgreSQL transport using Npgsql | | | DotNetWorkQueue.Transport.Redis | Redis transport using StackExchange.Redis | | | DotNetWorkQueue.Transport.SQLite | SQLite transport using System.Data.SQLite | | | DotNetWorkQueue.Transport.LiteDb | LiteDB embedded NoSQL transport | | | DotNetWorkQueue.Transport.Memory | In-memory transport for testing and lightweight scenarios | |

Dashboard

| Package | Description | NuGet | |---------|-------------|-------| | DotNetWorkQueue.Dashboard.Api | REST API for monitoring and managing queues across all transports. Queue status, message counts, error tracking, consumer tracking, and admin actions. ASP.NET Core with Swagger/OpenAPI. | | | DotNetWorkQueue.Dashboard.Ui | Blazor Server web UI (MudBlazor) for browsing queues, viewing messages, managing errors, and monitoring consumers. Connects to the Dashboard API. | | | DotNetWorkQueue.Dashboard.Client | Typed API client and consumer registration client. Sends heartbeats with running metric counters (processed, errors, rollbacks, poison messages). No dependency on the core library — HttpClient and System.Text.Json only. | |

Metrics: Built-in metrics are provided via System.Diagnostics.Metrics in the core library. No additional package is needed. To collect and export metrics, configure OpenTelemetry.Metrics in your host application.

---

Differences Between Versions

.NET Standard 2.0, 8.0, and 10.0 are missing the following features compared to the full framework version:

• No support for aborting threads when stopping consumer queues
• No support for dynamic LINQ statements

---

Usage — POCO

| Transport | Producer | Consumer | |-----------|----------|----------| | SQL Server | SQLServerProducer | SQLServerConsumer | | PostgreSQL | PostgreSQLProducer | PostgreSQLConsumer | | Redis | RedisProducer | RedisConsumer | | SQLite | SQLiteProducer | SQLiteConsumer | | LiteDb | LiteDbProducer | LiteDbConsumer |

---

Usage — LINQ Expressions

You can queue LINQ expressions to be executed instead of POCOs. This makes producers and consumers generic — they no longer need to be message-specific. The examples below are not transport-specific and assume any queue creation steps have already been performed.

Note: It is possible for a producer to queue work that a consumer cannot process. In order for a consumer to execute a LINQ statement, all types must be resolvable. For dynamic statements, it is also possible to queue work that doesn't compile due to syntax errors — this won't be discovered until the consumer dequeues the work.

Producer

Note: When passing message or workerNotification as arguments to dynamic LINQ, you must cast them, as the internal compiler treats them as object. This is not necessary when using standard LINQ expressions.

// Cast types when using dynamic LINQ:
(IReceivedMessage<MessageExpression>) message
(IWorkerNotification) workerNotification

SQLiteProducerLinq/Program.cs

When passing value types, you will need to parse them inline. For example, a Guid and an int can be embedded in string literals and parsed using built-in .NET methods:

var id = Guid.NewGuid();
var runTime = 200;
$"(message, workerNotification) => StandardTesting.Run(new Guid(\"{id}\"), int.Parse(\"{runTime}\"))"

This produces a LINQ expression that can be compiled and executed by the consumer, provided it can resolve all referenced types.

Consumer

The consumer is generic and can process any LINQ expression, but it must be able to resolve all types the expression references. You may need to wire up an assembly resolver if your DLLs cannot be located automatically.

• AppDomain.AssemblyResolve (MSDN)
• SQLiteConsumerLinq/Program.cs

Security Considerations

No sandboxing or checking for risky commands is performed. For example, the following statement will cause the consumer host to exit:

"(message, workerNotification) => Environment.Exit(0)"

If configuration files define dynamic LINQ statements, or if you cannot fully trust the producer, consider running the consumer in an application domain sandbox. Without that, the only protection against destructive commands is O/S user permissions:

"(message, workerNotification) => System.IO.Directory.Delete(@\"C:\\Windows\\\", true)"

---

Usage — Job Scheduler

Jobs may be scheduled using Schyntax format. The scheduler and consumers are separate — schedulers only queue work, while standard LINQ consumers handle processing.

Any LINQ statement supported by a LINQ producer can be scheduled via the scheduler.

Multiple schedulers sharing the same schedule can be run for redundancy, but it is important that the clocks on all machines are in sync, or that the same time provider is injected into both schedulers and consumers. See the Wiki for more details.

Note: Using multiple machines with out-of-sync clocks may produce unexpected results. Server-based transports often provide solutions for this. See the Wiki.

See Schyntax for the event scheduling format.

Scheduler

The scheduler and its container must remain in scope for as long as you are scheduling work. Disposing or allowing the scheduler to go out of scope will stop all queuing.

SQliteScheduler/Program.cs

To consume and process scheduled jobs, use a LINQ Consumer:

[SQLiteSchedulerConsumer/Program.cs](https