pynab

Pynab is a rewrite of Newznab, using Python and PostgreSQL. Complexity is way down, consisting of (currently) ~4,000 SLoC, compared to Newznab's ~104,000 lines of php/template. Performance and reliability are significantly improved, as is maintainability and a noted reduction in the sheer terror I experienced upon looking at some of the NN code in an attempt to fix a few annoying bugs.

This project was written almost entirely for my own amusement and use, so it's specifically tailored towards what I was looking for in an indexer - fast, compatible API with few user restrictions and low complexity. I literally just use my indexer to index groups and pass access to my friends, so there's no API limits or the like. If you'd like to use this software and want to add such functionality, please feel free to fork it! I won't have time to work on it beyond my own needs, but this is what open source is for.

Note that because this is purely for API access, the WebUI is very simple. You cannot add users through a web interface, manage releases, etc. Something like 99.9% of the usage of my old Newznab server was API-only, so it's low-priority.

API Compatibility

Sickbeard and branches
NZBDrone
CouchPotato
Headphones (only slightly tested)
NZB360
NZBSearcher

Features

Group indexing
Mostly-accurate release generation (thanks to Newznab's regex collection)
Also mostly-accurate release categorisation (which is easily extensible)
Binary blacklisting (regex thanks to kevinlekiller)
High performance
Developed around pure API usage
Newznab-API compatible (mostly, see below)
TVRage/IMDB/Password post-processing
Release renaming for most obfuscated releases
Pre-DB comparisons to assist in renaming
XMPP PubSub support to push to nzb clients

In development:

Additional PreDB sources
Got an idea? Send it in!

Technical Differences to Newznab

NZBs are imported whole
- Bulk imports of 50gb of nzb.gzs now take hours to process, not weeks
- No more importing in batches of 100 - just point it at a directory of 600,000 NZBs and let it process
- Relies on provided NZBs being complete and mostly good, though
NZBs are stored in the DB
- Commonly-grabbed NZBs are generally cached in RAM
- Big IO savings
- Generally quicker than off the HDD
- You don't run into filesystem problems from having 2.5 million files in a few directories
Very simple query interface
- The vast majority of access to my indexer was API-based (1-5 web hits per month vs 50,000+ api hits)
- It's not a replacement for Newznab if you have a lot of direct user interaction
Simplified authentication
- No more usernames, passwords, or anything, really.
- API key access required for everything - sure, it can be sniffed very easily, but it always could be. Worst that can happen: someone uses the API.
General optimisations
- Several operations have been much-streamlined to prevent wasteful, un-necessary regex processing
- No language wars, but Python is generally quicker than PHP (and will be moreso when PyPy supports 3.3)
- General (significant) database speed improvements

Instructions

Installation and execution is reasonably easy.

Requirements

Python 3.2 or higher
PostgreSQL 9.3+ or MySQL/Maria/Percona 5.5+
A u/WSGI-capable webserver (or use CherryPy)

I've tested the software on Ubuntu Server 12.04-14.10 and Windows 7/8/10, so all should work.

Installation

Ubuntu 12.04 and earlier

Follow the instructions by broknbottle in Issue #15 to install Python 3.3.x, then follow the 13.04 instructions.

Ubuntu 13.04/13.10 and later

If Postgres: install PostgreSQL 9.3/9.4, as per instructions here. If MySQL: install MySQL 5.5+ (preferably 5.6+)

You also need to install Python 3.3/3.4, associated packages and pip3:

> sudo apt-get install python3 python3-setuptools python3-pip libxml2-dev libxslt-dev libyaml-dev

And a few packages required by psycopg2 (you need these even if you're using MySQL):

> sudo apt-get install postgresql-server-dev-9.3

If you're using MySQL, you'll also need to several lines to your /etc/mysql/my.cnf, under [mysqld]:

local_infile = 1
innodb_large_prefix = 1
innodb_file_format = Barracuda
innodb_file_per_table = 1

This allows us to vastly increase the speed of segment writes. Without this, scanning will be impossibly slow!

General *nix

> cd /opt/
> sudo git clone https://github.com/Murodese/pynab.git
> sudo chown -R www-data:www-data pynab
> cd pynab
> sudo cp config_sample.py config.py
> sudo vim config.py [fill in details as appropriate]
> sudo pip3 install -r requirements.txt

If you receive an error message related to an old version of distribute while running pip3, you can install the new version by typing:

sudo easy_install -U distribute

Windows

Running pynab on Windows is possible, but not recommended or well-supported. Lack of screen support means that console output is tricky, so using logfiles is very much recommended.

Clone and configure:

> [browse to desired directory]
> git clone https://github.com/Murodese/pynab.git
> [browse to pynab]
> [copy config_sample.py to config.py]
> [fill in config as appropriate, ensuring to set logfile]

Install pre-reqs. The following packages are available as Windows binaries from here. Select the appropriate package for your version of python (ie. py34 for 3.4, etc):

- lxml
- sqlalchemy
- psycopg2
- markupsafe

Two packages used in pynab require a compiler, such as MinGW. This may also require you to modify some config vars to make pip see the compiler, see here.

Once the compiler has been installed:

> pip install -r requirements.txt

Install/Migrate

New installation? As below:

> sudo python3 install.py [follow instructions]

Migrating from Newznab? Go here: Converting from Newznab

Migrating from pynab-mongo? Go here: Converting from pynab-mongo

Once done:

> sudo chown -R www-data:www-data /opt/pynab
> sudo chown -R www-data:www-data /var/log/pynab [or whatever logging_dir is set to]

The installation script will automatically import necessary data and download the latest regex and blacklists.

Please note that in order to download updated regexes from the Newznab crew, you'll need a NN+ ID. You can get one by following the instructions on their website (generally a donation). You can also import a regex dump or create your own.

Converting from Newznab

WARNING:

This software is unstable as yet, so keep backups of everything - if you're importing NZBs,
make sure you make a copy of them first. The import script will actively delete
things, newznab conversion will just copy - but better to be safe.

Pynab can transfer some data across from Newznab - notably your groups (and settings), categories and TVRage/IMDB data, as well as user details and current API keys. This means that your users should only experience downtime for a short period, and don't have to regenerate their API keys. Hate your users? No problem, they won't even notice the difference and you don't even have to tell them.

To run the conversion, first follow the normal installation instructions. Then:

> python3 scripts/convert_from_newznab.py <mysql hostname> <newznab db>

run --help for additional options.

This will copy over relevant data from your Newznab installation. Because Pynab's method of storing NZBs and metadata is very different to Newznab, we can't do a direct releases table conversion - you need to import the NZBs en-masse. Luckily, this is no longer an incredibly lengthy process - it should only take a few hours to process several hundred thousand NZBs on a reasonable server. Importing 2.5 million releases from my old installation took 11 hours.

To import said NZBs:

> python3 scripts/import.py /path/to/nzbs

For most Newznab installations, it'll look like this:

> python3 scripts/import.py /var/www/newznab/nzbfiles

:warning: Run this script against a copy of the nzb folder, since it automatically deletes NZBS that were successfully imported.

Allow this to finish before starting normal operation.

Converting from pynab-mongo

If you were using pynab-mongo and want to convert your database to the Postgre version, there's a script supplied. It's recommended that you side-load the Postgre branch, rather than cut over directly:

# don't bother running install.py first, as we're copying everything from mongo
# you will, of course, need postgres installed
> cd /opt
> git clone https://github.com/Murodese/pynab.git pynab-postgres
> cd /opt/pynab-postgres
> git checkout development-postgres
> cp config_sample.py config.py
> [edit config.py to add mongo and postgres config]
> python3 scripts/convert_mongo_to_postgre.py

The script handles virtually everything, copying all necessary data. For large installations, this could take some time - there's no quick way to copy that data across. That said, it's not too excessive - for 500k releases, somewhere between 15 minutes to an hour depending on server specs. The migration script is unable to handle existing release file data and will need to re-retrieve it.

Once this is complete, rename the old folder an

Pynab

Install / Use

README