Snakemake is a workflow tool

• Being developed by Johannes Köster
• Started 2011 at TU Dortmund in our bioinformatics group
• Now part of his PhD thesis
• Used by a lot of groups now

Snakemake is inspired by make

• but with Python-derived, readable syntax
• and a focus on bioinformatics pipelines

A workflow is a set of rules

• A rule describes how to transform one file into another:

  Rule 1: file.bam → file.vcf

  Rule 2: file.fastq → file.bam

• Filenames determine which rule to apply
• Execution order is figured out automatically

• Rules are not executed if output exists and is newer than input

Rules are written in a “Snakefile”

When run, snakemake expects a Snakefile in the current directory. An example:

rule:
    input: 'slides.md'
    output: 'slides.html'
    shell:
        'pandoc -t revealjs -o {output} {input}'

Running snakemake slides.html generates slides.html from slides.md.

Rules and parameters can have names

rule pandoc:
    input:
        md='slides.md',
        template='default.revealjs'
    output:
        html='slides.html'
    shell:
        'pandoc --template={input.template} -o {output.html} {input.md}'

Multiple output files are also possible (unlike make)!

Shell commands must be quoted

• Use """ for multi-line strings
• Use {...} to access parameters
• Double { and } if part of the command

rule:
    input: 'file.txt'
    output: 'file.sum'
    shell:
        """
        awk '{{ s += $3 }} END {{ print s }}' {input} > {output}
        echo "finished"
        """

Wildcards generalize rules

rule targets:
    input: 'slides.html', 'talk.html'

rule pandoc:
    input:
        md='{base}.md',
        template='default.revealjs'
    output:
        html='{base}.html'
    shell:
        'pandoc --template={input.template} -o {output.html} {input.md}'

• First rule is executed if nothing given on command line.
• Wildcards such as {base} in input and output file names are allowed.

Running snakemake generates both slides.html and talk.html.

Snakefiles can contain Python code

Using pandas and matplotlib to plot a coverage histogram:

import pandas as pd
import matplotlib.pyplot as plt

rule plot_coverage_histogram:
    output: pdf='{base}.covhist.pdf'
    input: txt='{base}.covhist.txt'
    run:
        d = pd.read_csv(input.txt, sep='\t', index_col=0, dtype=int)
        fig = plt.figure()
        ax = fig.gca()
        ax.plot(d.index, d.frequency, 'bo')
        fig.savefig(output.pdf)

• Note how input.txt and output.pdf are used
• Snakefiles are Python programs with some extra syntax for rules!

Visualize the dependencies

snakemake --dag (DAG: directed acyclic graph)

Bonus section

Use if for conditional rules

if MERGE_PROGRAM == 'flash':
    rule flash_merge:
        """Use FLASH to merge paired-end reads"""
        output: 'merged.fastq.gz'
        input: 'reads.1.fastq', 'reads.2.fastq'
        threads: 8
        shell:
            """
            flash -t {threads} -c -M {MAX_OVERLAP} {input} | gzip > {output}
            """
elif MERGE_PROGRAM == 'pear':
    rule pear_merge:
        """Use pear to merge paired-end reads"""
        ...
else:
    sys.exit("MERGE_PROGRAM not recognized")

Simplify debugging

To use the unofficial bash strict mode, put this at the beginning of your Snakefile:

shell.prefix("set -euo pipefail;")

If a command fails, the entire rule will fail – this is what you want.

Snakemake has a browser interface

Start it with snakemake --gui