Menotexport

Menotexport (Mendeley-Note-Export) extracts and exports highlights, notes and PDFs from your Mendeley database

IMPORTANT NOTE:

It seems that Mendeley has decided to put an encryption in version 1.19 on the database file from which various information (your highlights, notes, all metadata of the documents) is retrieved by this tool.

Some relavent info on this:

• zotero forum
• Mendeley release note
• a blog sharing a workaround
• issue report

I have little experience handling sqlite data encryption/decryption. So if anyone can offer any suggestion it will be greatly appreciated, including any advises on potential legal issues distributing a tool like this that bypasses their encryption.

Also if you encounter an error when trying to run this tool on your local database file:

# <Menotexport>: Failed to recoganize the given database file.
file is not a database

before we figure out an easy to bypass this, please consider backing up your database file, and trying an older version of Mendeley before 1.19. Sorry for the trouble.

Update 2020-10-09: Here is a method suggested by a user to decrypt the sqlite database file: https://eighty-twenty.org/2018/06/13/mendeley-encrypted-db. See also this issue report.

Update 2021-02-25: Here is some tips for migration to Zenodo, thanks for the inputs from pboley.

More rants

I thought I messed up my sync, but no, it appears to be a much larger scale issue with Mendeley: lots of people are losing their PDFs after syncing. See this twitter and their support page.

I was trying their re-activation workaround, but guess what, they say that sync setting is deprecated in my version (1.17.10) and I need to upgrade to the latest, which encrypts your LOCAL data. WTF Mendeley?!

Lesson learnt: backup your data regularly, and better still, ditch Mendeley.

What does this do?

Menotexport is a simple python solution to help extract and export annotations (highlighted texts, sticky notes and notes) you made in the build-in PDF reader of Mendeley, bulk-export PDFs with annotations, and bulk-export meta-data with annotations to .bib or .ris file.

Mendeley is a desktop and web program for managing and sharing research papers. It offers free desktop clients for Windows, OSX and Linux. But the software is not open source, and their support team has been real slow in responding to customers feature requests, some of which has been proposed by many for YEARS. This tool aims at solving the following:

1. Bulk export annotated PDFs.

Annotations (highlights and notes) made inside Mendeley are saved not directly onto the relevant PDFs, but to a separate database file. Therefore these annotations can not be viewered in other PDF readers other than Mendeley itself.

The native but awkward solution to export a PDF with its annotations is: in Mendeley, open that PDF in the Mendeley PDF reader, go to Files -> Export PDF with annotations. However to export all your collections, this has to be repeated manually for each individual PDF in your library. To make it worse, the annotations exported in this manner are saved as static texts and are not editable.

This tool can bulk export all PDFs with annotations while keeping your Mendeley folder structure, and the annotations are readable and editable by other PDF softwares. PDFs with no annotations are simply copied to the target location, so you have the complete library structure.

2. Extract annotation texts.

To extract texts from the highlights, sticky notes and notes ("General notes" in the right-hand side-bar) in a PDF, other than Copy-n-Paste one by one, some softwares offer an automated solution.

skim on OSX has the functionality to produce a summary of all annotations.

Some versions of Foxit Reader can do that (on windows, not on the Linux version, not sure about Mac).

Pro versions of Adobe Reader may have that too.

Most of the PDF readers in Linux do not have that functionality. (Please let me know if you find one).

This tool could extract the texts from the highlights and notes in the documents in Mendeley to a plain text file, and format the information in a sensible structure using markdown syntax.

3. Export libray to .bib file.

Exporting to .bib is natively supported in Mendeley, by going to Tools -> Options -> Bibtex. There you can specify exporting the whole library to a single file or in a per-folder manner. However, your whole annotations won't be included. This tool helps you pack-up as much information as possible to a .bib file, which might be helpful for people who want to migrate to another management tool without loosing too much efforts put into Mendeley.

Fields that are exported to a .bib entry (as long as they are present in your Mendeley document record):

- citationkey
- authors
- year
- title
- publication
- volume
- issue
- pages
- doi
- abstract
- arxivId
- chapter
- country
- city
- edition
- institution
- isbn
- issn
- month
- day
- publisher
- series
- type
- keywords
- read            # Read or not
- favourite       # Marked as Favourite or starred in Mendeley
- tags            # Tags added to a document
- file            # Location of the attached PDF on local disk
- folder          # Folder name in the Mendeley library

Some other features

1. Export preserves sub-folder structures

Mendeley supports embedded folder structures and is properly addressed by this tool: the exported PDFs, and their corresponding "file" entries in the exported .bib (or .ris) file now reflect the folder structure (empty folders are omitted, embedded folders are processed recursively).

You are allowed to create folders with the same name in Mendeley, so long as they appear in different parent folders. In case you did do so, they will be labelled differently in the GUI version: e.g. "folderA", "folder1/folderA" and "folder2/folderA" are used to distinguish these three "folderA"s.

2. Highlight colors in Mendeley

Mendeley 1.16.1 introduces 7 more highlight colors, these are replicated in the exported PDFs.

3. Extra utility to help improve accuracy in highlight extractions;

Two text extracting utilities (pdfminer and pdftotext) are used to extract highlighted texts from PDFs. The user can choose to install the relevant utility to enable the pdftotext functions (installation details see below) to create better outputs than the default pdfminer results. The cost is an extra dependency to satisfy (see below), and a slight drop in execution speed. However, this new feature is optional: if you don't care about highlight extraction or don't have pdftotext available on the system, it will fall back to the pdfminer-only solution.

4. Zotero-ready output format

Use the "-z" flag (command-line version), or toggle the "For import to Zotero" option (GUI version) to re-format the exported .bib file, making it suitable to import into Zotero. Therefore to migrate over to Zotero, specify "Export PDFs", "Extract highlights", "Extract notes" and "Export to .bib" (by giving a "-pmnb" flag), process a folder or the entire Mendeley library, then point the "import" function of Zotero to the exported .bib file. Document entries with meta-data, notes (highlighted texts + notes), tags and the attached PDFs (if they exist) will be added.

5. Export to .ris format.

Export meta-data and annotations to .ris file. If -z flag is toggled, the output can be properly recognized by Zotero, and a migration to Zotero via the .ris approach can be achieved by a process with -pnmrz options.

6. Custom template formatting for exported annotations.

Annotations (notes+highlights) can be formatted in the way you like.

In the lib folder there is a file annotation_template.py which contains a working example template to format the output of the exported annotations.

To use custom template:

• in command line: add the -t option.
• in GUI: toggle the "use custom template (experimental)" button.

Currently, these variables are available in building a template:

• text : content for a note, or highlighted texts for a highlight.
• page : page number of the note/highlight in the PDF file (not necessarily the same as printed out at the page margin).
• title: title of the document.
• tags : tag list.
• ctime: creation time.
• author : author(s) of the document.
• note_author: author(s) of the annotation.
• citationkey: citation key.
• num : an integer id of the note/highlight, counting restarts in each document.

Put any of them in curly brackets to use them, e.g. {title}. NOTE no spaces in brackets. More instructions can be found in the template file.

For deeper modification of the output formatting, you can hack into this file: /lib/exportannotation.py.

Installation

1. Install via conda.

For command line or GUI usage on Linux (64bit), recommend installing using conda:

conda create -n menotexport python=2.7
source activate menotexport
conda install -c guangzhi menotexport

For the installation of conda (Anaconda or a lighter-weight version: Miniconda), see their official site.

2. Pre-build binary GUI for Windows

For Windows 7 and Windows 10 (64bit) (version 1.4, updated on 08-July-2017), download menotexport-gui-win7-win10.zip from Google Drive: https://drive.google.com/open?id=0B8wpnLHH0j1hTTM5cTE2TXg2b1k, unpack, then launch `