Gulp + Metalsmith = gulpsmith

gulpsmith lets you use Gulp plugins (or vinyl pipelines) with Metalsmith, and use Metalsmith plugins as part of a Gulp or vinyl pipeline. This can be helpful if you:

• Don't want Metalsmith to slurp up an entire directory tree of files,
• Want to upload your Metalsmith build to Amazon S3 or send it someplace via SFTP without first generating files locally and then running a separate uploading process,
• Want to pre- or post-process your Metalsmith build with Gulp plugins, or
• Already run your build process with one tool or the other and don't want to switch, but need both kinds of plugins.

gulpsmith().use(metal_plugin1).use(metal_plugin2)... wraps one or more Metalsmith plugins for use in Gulp, whereas gulpsmith.pipe(stream1).pipe(stream2)... turns a series of Gulp plugins (or vinyl streaming operations) into a plugin that can be passed to Metalsmith's .use() method.

(In addition,gulpsmith.pipe() is Highland-friendly and lets you pass in Highland stream transforms and functions as well as Gulp plugins.)

While a perfect translation between Gulp and Metalsmith is impossible, gulpsmith does its best to be lossless and corruption-free in both directions. When "corruption-free" and "lossless" are in conflict, however, gulpsmith prioritizes being "corruption-free". That is, it chooses to drop conflicting properties during translation, rather than create problems downstream. (See File Conversions and Compatibility, below, for more details.)

Important: starting with gulpsmith 0.6.0, gulp files are created using vinyl 2.1, which means that the list of dropped properties has changed, and the list in this document may not be accurate for vinyl versions > 2.1. You may need to stay on gulpsmith 0.5.5 if your project uses any of the new reserved property names (like history) as metalsmith properties. But if your project uses gulp 4, upgrading to gulpsmith 0.6.0 is required.

Table of Contents

<!-- toc -->

• Using Metalsmith in a Gulp Pipeline
  • Front Matter and File Properties
• Using a Gulp Pipeline as a Metalsmith Plugin
  • Enhanced Features of gulpsmith.pipe()
  • Advanced Pipelines and Error Handling
    • Reusing Pipelines
    • Using Pre-assembled Pipelines
    • Stream Operations Other Than .pipe()
• File Conversions and Compatibilty
  • Reserved Properties
    • Gulp Reserved Property Names
    • Metalsmith Reserved Property Names
  • Gulp Directories and Metalsmith

<!-- toc stop -->

Using Metalsmith in a Gulp Pipeline

To use Metalsmith in a Gulp pipeline, call gulpsmith() with an optional directory name (default is process.cwd()) which will be used to create a Metalsmith instance. The return value is a stream that can be used in a Gulp .pipe() chain, but which also has .use() and .metadata() methods that can be used to configure the Metalsmith instance.

Instead of reading from a source directory and writing to a destination directory, the wrapped Metalsmith instance obtains all its files in-memory from the Gulp pipeline, and will send all its files in-memory to the next stage of the pipeline.

(Because Metalsmith processes files in a group, note that your overall Gulp pipeline's output will pause until all the files from previous stages have been processed by Metalsmith. All of Metalsmith's output files will then be streamed to the next stage of the pipeline, all at once.)

Example:

gulpsmith = require('gulpsmith');

gulp.src("./src/**/*")
.pipe(some_gulp_plugin(some_options))
.pipe(
    gulpsmith()     // defaults to process.cwd() if no dir supplied

    // You can initialize the metalsmith instance with metadata
    .metadata({site_name: "My Site"})

    // and .use() as many Metalsmith plugins as you like 
    .use(markdown())
    .use(permalinks('posts/:title'))
)
.pipe(another_gulp_plugin(more_options))
.pipe(gulp.dest("./build")

Front Matter and File Properties

Unlike Metalsmith, Gulp doesn't read YAML front matter by default. So if you want the front matter to be available in Metalsmith, you will need to use the gulp-front-matter plugin, and insert something like this to promote the .frontMatter properties before piping to gulpsmith():

gulp_front_matter = require('gulp-front-matter');
assign = require('lodash.assign');

gulp.src("./src/**/*")

.pipe(gulp_front_matter()).on("data", function(file) {
    assign(file, file.frontMatter); 
    delete file.frontMatter;
})

.pipe(gulpsmith()
    .use(...)
    .use(...)
)

This will extract the front matter and promote it to properties on the file, where Metalsmith expects to find it. (Alternately, you could use gulp-append-data and the data property instead, to load data from adjacent .json files in place of YAML front matter!)

Of course, there are other Gulp plugins that add useful properties to files, and those properties will of course be available to your Metalsmith plugins as well.

(For example, if you pass some files through the gulp-jshint plugin before they go to Metalsmith, the Metalsmith plugins will see a jshint property on the files, with sub-properties for success, errorCount, etc. If you use gulp-sourcemaps, your files will have a sourceMap property, and so on.)

Using a Gulp Pipeline as a Metalsmith Plugin

To use Gulp plugins or other streams as a Metalsmith plugin, simply begin the pipeline with gulpsmith.pipe():

gulpsmith = require('gulpsmith')

Metalsmith(__dirname)
.use(drafts())
.use(markdown())
.use(gulpsmith
    .pipe(some_gulp_plugin(some_options))
    .pipe(another_gulp_plugin(more_options))
    .pipe(as_many_as(you_like))
)
.use(more_metalsmith_plugins())
.build()

From the point of view of the Gulp plugins, the file objects will have a cwd property equal to the Metalsmith base directory, and a base property equal to the Metalsmith source directory. They will have a dummy stat property containing only the Metalsmith file's mode, along with any other data properties that were attached to the file by Metalsmith or its plugins (e.g. from the files' YAML front matter).

In this usage pattern, there is no gulp.src() or gulp.dest(), because Metalsmith handles the reading and writing of files. If the Gulp pipeline drops or renames any of the input files, they will be dropped or renamed in the Metalsmith pipeline as well.

(If you want to, though, you can include a gulp.dest(), or any other Gulp output plugin in your pipeline. Just make sure that you also do something to drop the written files from the resulting stream (e.g. using gulp-filter), unless you want Metalsmith to also output the files itself. Doing both can be useful if you use a Gulp plugin to upload files, but you also want Metalsmith to output a local copy.)

Enhanced Features of gulpsmith.pipe()

Under the hood, gulpsmith.pipe() is a thin wrapper around Highland's _.pipeline() function. This means that:

• You can pass multiple plugins in (e.g. gulpsmith.pipe(plugin1, plugin2,...)
• You can pass in Highland transforms as plugins (e.g. using gulpsmith.pipe(_.where({published:true})) to pass through only posts with a true .published property.)
• You can pass in functions that accept a Highland stream and return a modified version of it, e.g.:

gulpsmith.pipe( 
    function(stream) { 
        return stream.map(something).filter(otherthing); 
    }
)

In addition, Highland's error forwarding makes sure that errors in anything passed to gulpsmith.pipe() are passed on to Metalsmith. (More on this in the next section.)

Advanced Pipelines and Error Handling

If the pipeline you're using in Metalsmith is built strictly via a series of of gulpsmith.pipe().pipe()... calls, and you don't save a Metalsmith instance to repeatedly call .build() or .run() on, you probably don't need to read the rest of this section.

If you need to do something more complex, however, you need to be aware of three things:

1. Unlike most Metalsmith plugins, Gulp plugins/pipelines are stateful and cannot be used for more than one build run.

2. If you pass a precomposed pipeline of plugins to gulpsmith.pipe(), it may not report errors properly, thereby hanging or crashing your build if an error occurs.

3. Unlike the normal stream .pipe() method, gulpsmith.pipe() does not return the piped-to stream: it returns a Metalsmith plugin that just happens to also have a .pipe() method for further chaining!

The following three sub-sections will tell you what you need to know to apply or work around these issues.

Reusing Pipelines

If you want to reuse the same Metalsmith instance over and over with the same Gulp pipeline embedded as a plugin, you must recreate the pipeline on each run. (Sorry, that's just how Node streams work!)

It's easy to do that though, if you need to. Just write a short in-line plugin that re-creates the pipeline each time, like this:

Metalsmith(__dirname)
.use(drafts())
.use(markdown())
.use(function() {   // inline Metalsmith plugin...
    return gulpsmith
        .pipe(some_gulp_plugin(some_options))
        .pipe(another_gulp_plugin(more_options))
        .pipe(as_many_as(you_like))
    .apply(