Numpy.NET

<a href="http://scisharpstack.org"><img src="https://github.com/SciSharp/SciSharp/blob/master/art/scisharp_badge.png" width="200" height="200" align="right" /></a>

Numpy.NET is the most complete .NET binding for NumPy, which is a fundamental library for scientific computing, machine learning and AI in Python. Numpy.NET empowers .NET developers with extensive functionality including multi-dimensional arrays and matrices, linear algebra, FFT and many more via a compatible strong typed API. Several other SciSharp projects like Keras.NET and Torch.NET depend on Numpy.NET.

Example

Check out this example which uses numpy operations to fit a two-layer neural network to random data by manually implementing the forward and backward passes through the network.

Numpy and Intellisense: a developer-friendly combination:

Installation

If you want to use Numpy.NET you have two options:

Numpy.dll

Just reference Numpy.dll via Nuget, set your build configuration to x64 and you are good to go. Thanks to Python.Included it doesn't require a local Python installation or will not clash with existing installations.

Numpy.Bare.dll

In certain use cases you might not want the packaged Python and NumPy packages. In that case you reference Numpy.Bare.dll via Nuget. Depending on the Numpy.Bare nuget version will need Python 3.5, 3.6 or 3.7 and Numpy 1.16 installed for it to work. The first two digits of the Numpy.Bare version indicate which Python version is needed for it to run (i.e. Numpy.Bare v3.6.1.1 needs Python 3.6 installed). If you are getting BadImageFormatException switch between x86 and x64 build settings.

In other cases, you might want to control the install location of the Python installation or even set it up yourself instead of having the Numpy library do it. For those cases Numpy.Bare is also great. Check out the custom installation example if you want to know how.

How does it work?

Numpy.NET uses Python for .NET to call into the Python module numpy. However, this does not mean that it depends on a local Python installation! Numpy.NET.dll uses Python.Included which packages embedded Python 3.7 and automatically deploys it in the user's home directory upon first execution. On subsequent runs, it will find Python already deployed and therefore doesn't install it again. Numpy.NET also packages the NumPy wheel and installs it into the embedded Python installation when not yet installed.

Long story short: as a .NET Developer you don't need to worry about Python at all. You just reference Numpy.NET, use it and it will just work, no matter if you have local Python installations or not.

Multi-threading (Must read!)

Beware: Not following these steps can result in deadlocks or access violation exceptions!

Python/NumPy doesn't have real multi-threading support. There is no advantage in "simultaneously" executing numpy functions on multiple threads because pythonnet requires you to use the Global Interpreter Lock (GIL) to lock access to the Python engine exclusively for only one thread at a time. If you have to call Python from a thread other than the main thread you first must release the main thread's mutex by calling PythonEngine.BeginAllowThreads() or you'll have a deadlock:

var a = np.arange(1000);
var b = np.arange(1000);

// https://github.com/pythonnet/pythonnet/issues/109
PythonEngine.BeginAllowThreads();

Task.Run(()=> {
  // when running on different threads you must lock!
  using (Py.GIL())
  {
    np.matmul(a, b);
  }
}).Wait();

Above example only serves as a reference on how to call numpy from a different thread than the main thread. As said before, having multiple background threads that call into Python doesn't give you multi-core processing because of the requirement to lock the GIL. Not doing so will result in access violation exceptions and/or deadlocks.

Note that you must call a method of np before calling PythonEngine.BeginAllowThreads() in order for the PythonEngine to be initialized. So, for instance, if you want to initialize an inherently multi-threaded .Net Core Web API at startup, do something like this:

np.arange(1);
PythonEngine.BeginAllowThreads();

Also, if you do this, be sure to wrap any calls to Numpy in using (Py.GIL()) { ... } or else you'll get AccessViolationExceptions.

Performance considerations

You might ask how calling into Python affects performance. As always, it depends on your usage. Don't forget that numpy's number crunching algorithms are written in C so the thin pythonnet and Python layers on top won't have a significant impact if you are working with larger amounts of data.

In my experience, calling numpy from C# is about 4 times slower than calling it directly in Python while the execution time of the called operation is of course equal. So if you have an algorithm that needs to call into numpy in a nested loop, Numpy.NET may not be for you due to the call overhead.

All of numpy is centered around the ndarray class which allows you to pass a huge chunk of data into the C routines and let them execute all kinds of operations on the elements efficiently without the need for looping over the data. So if you are manipulating arrays or matrices with thousands or hundreds of thousands of elements, the call overhead will be negligible.

The most performance sensitive aspect is creating an NDarray from a C# array, since the data has to be moved from the CLR into the Python interpreter. pythonnet does not optimize for passing large arrays from C# to Python but we still found a way to do that very efficiently. When creating an array with np.array( ... ) we internally use Marshal.Copy to copy the entire C#-array's memory into the numpy-array's storage. And to efficiently retrieve computation results from numpy there is a method called GetData<T> which will copy the data back to C# in the same way:

// create a 2D-shaped NDarray<int> from an int[]
var m = np.array(new int[] {1, 2, 3, 4});
// calculate the cosine of each element
var result = np.cos(m);
// get the floating point data of the result NDarray back to C#
var data = result.GetData<double>(); // double[] { 0.54030231, -0.41614684, -0.9899925 , -0.65364362 }

Numpy.NET vs NumSharp

The SciSharp team is also developing a pure C# port of NumPy called NumSharp which is quite popular albeit being not quite complete.

There are a couple of other NumPy ports out there featuring subsets of the original library. The only one that matches Numpy.NET in terms of completeness is the IronPython package numpy which is out of date though. The SciSharp team is committed to keeping Numpy.NET up to date with the original library and to feature as much of the original functionality as possible.

Code generation

The vast majority of Numpy.NET's code is generated using CodeMinion by parsing the documentation at docs.scipy.org/doc/numpy/. This allowed us to wrap most of the numpy-API in just two weeks. The rest of the API can be completed in a few more weeks, especially if there is popular demand.

Completion status

The following API categories have been generated (if checked off)

• [x] Array creation routines
• [x] Array manipulation routines
• [x] Binary operations
• [x] String operations
• [x] Datetime Support Functions
• [x] Data type routines
• [x] Optionally Scipy-accelerated routines(numpy.dual)
• [ ] Floating point error handling
• [x] Discrete Fourier Transform(numpy.fft)
• [x] Financial functions
• [ ] Functional programming
• [x] Indexing routines
• [x] Input and output
• [x] Linear algebra(numpy.linalg)
• [x] Logic functions
• [ ] Masked array operations
• [x] Mathematical functions
• [ ] Matrix library(numpy.matlib)
• [ ] Miscellaneous routines
• [x] Padding Arrays
• [ ] Polynomials
• [x] Random sampling(numpy.random)
• [x] Set routines
• [x] Sorting, searching, and counting
• [x] Statistics
• [x] Window functions

Over 500 functions of all 1800 have been generated, most of the missing functions are duplicates on Matrix, Chararray, Record etc.

Auto-generated Unit-Tests

We even generated hundreds of unit tests from all the examples found on the NumPy documentation. Most of them don't compile without fixing them because we did not go so far as to employ a Python-To-C# converter. Instead, the tests are set up with a commented out Python-console log that shows what the expected results are and a commented out block of somewhat C#-ified Python code that doesn't compile without manual editing.

Another reason why this process can not be totally automated is the fact that NumPy obviously changed the way how arrays are printed out on the console after most of the examples where written (they removed extra spaces between the elements). This means that oftentimes the results needs to be reformatted manually for the test to pass.

Getting more unit tests to run is very easy though, and a good portion have already been processed to show that Numpy.NET really works. If you are interested in working on the test suite, please join in and help. You'll learn a lot about NumPy on the way.

Documentation

Since we have taken great care to make Numpy.NET as sim