Etl.Net

Go to full documentation

:rotating_light: I need some help!!! :ambulance:

I need some serious help for the following subject:

• :books: Documentation. The website of ETL.NET needs to be completed. It contains the essential to start and get in touch, but a dramatic amount of serious features is not documented in there.
• :heavy_check_mark: Unit tests. A proper test code coverage is the only real way to be taken seriously when it is about selecting the right open source library. There was a time when some units tests existed only for the reactive engine of the core (Paillave.EtlNet.Core). But it happened I had to decommission it in the even of the V2 release.

:envelope_with_arrow: Anybody who is keen to participate to the project in these regards is very welcome to contact me at admroyer@hotmail.com.

Presentation

Implementation of a mass processing engine to use in a similar way than Linq with every SSIS features and much more. The reactive approach for the implementation of this engine ensures parallelized multi streams, high performances and low memory foot print even with million rows to process.

ETL.NET is fully written in .NET for a multi platform usage and for a straight forward integration in any application.

Extend it takes 5mn... literally.

| Package | nuget version | nuget downloads | |-|-|-| | Paillave.EtlNet.Core | | | | Paillave.EtlNet.Autofac | | | | Paillave.EtlNet.Dropbox | | | | Paillave.EtlNet.EntityFrameworkCore | | | | Paillave.EtlNet.ExcelFile | | | | Paillave.EtlNet.ExecutionToolkit | | | | Paillave.EtlNet.FileSystem | | | | Paillave.EtlNet.FromConfigurationConnectors | | | | Paillave.EtlNet.Ftp | | | | Paillave.EtlNet.Mail | | | | Paillave.EtlNet.GraphApi | | | | Paillave.EtlNet.Sftp | | | | Paillave.EtlNet.SqlServer | | | | Paillave.EtlNet.TextFile | | | | Paillave.EtlNet.Bloomberg | | | | Paillave.EtlNet.XmlFile | | | | Paillave.EtlNet.Zip | | | | Paillave.EtlNet.Pdf | | | | Paillave.EntityFrameworkCoreExtension | | |

Examples

Unzip it, read it, save it, report it

Read all zip files from a folder, unzip csv files that are inside, parse them, exclude duplicates, upsert them into database, and report new or pre existing id corresponding to the email.

dotnet new console -o SimpleTutorial
cd SimpleTutorial
dotnet add package Paillave.EtlNet.Core
dotnet add package Paillave.EtlNet.FileSystem
dotnet add package Paillave.EtlNet.Zip
dotnet add package Paillave.EtlNet.TextFile
dotnet add package Paillave.EtlNet.SqlServer

using System;
using System.Threading.Tasks;
using Paillave.Etl.Core;
using Paillave.Etl.FileSystem;
using Paillave.Etl.Zip;
using Paillave.Etl.TextFile;
using Paillave.Etl.SqlServer;
using Microsoft.Data.SqlClient;
using System.Linq;

namespace SimpleTutorial
{
    class Program
    {
        static async Task Main(string[] args)
        {
            var processRunner = StreamProcessRunner.Create<string>(DefineProcess);
            processRunner.DebugNodeStream += (sender, e) => { /* place a conditional breakpoint here for debug */ };
            using (var cnx = new SqlConnection(args[1]))
            {
                cnx.Open();
                var executionOptions = new ExecutionOptions<string>
                {
                    Resolver = new SimpleDependencyResolver().Register(cnx),
                };
                var res = await processRunner.ExecuteAsync(args[0], executionOptions);
                Console.Write(res.Failed ? "Failed" : "Succeeded");
                if (res.Failed)
                    Console.Write($"{res.ErrorTraceEvent.NodeName}({res.ErrorTraceEvent.NodeTypeName}):{res.ErrorTraceEvent.Content.Message}");
            }
        }
        private static void DefineProcess(ISingleStream<string> contextStream)
        {
            contextStream
                .CrossApplyFolderFiles("list all required files", "*.zip", true)
                .CrossApplyZipFiles("extract files from zip", "*.csv")
                .CrossApplyTextFile("parse file", FlatFileDefinition.Create(i => new Person
                {
                    Email = i.ToColumn("email"),
                    FirstName = i.ToColumn("first name"),
                    LastName = i.ToColumn("last name"),
                    DateOfBirth = i.ToDateColumn("date of birth", "yyyy-MM-dd"),
                    Reputation = i.ToNumberColumn<int?>("reputation", ".")
                }).IsColumnSeparated(','))
                .Distinct("exclude duplicates based on the Email", i => i.Email)
                .SqlServerSave("upsert using Email as key and ignore the Id", o => o
                    .ToTable("dbo.Person")
                    .SeekOn(p => p.Email)
                    .DoNotSave(p => p.Id))
                .Select("define row to report", i => new { i.Email, i.Id })