---
title: "ChIP-Seq Workflow Template" 
author: "Student Name: ttest"
date: "Last update: `r format(Sys.time(), '%d %B, %Y')`" 
output:
  BiocStyle::html_document:
    toc_float: true
    code_folding: show
package: systemPipeR
vignette: |
  %\VignetteEncoding{UTF-8}
  %\VignetteIndexEntry{WF: ChIP-Seq Workflow Template}
  %\VignetteEngine{knitr::rmarkdown}
fontsize: 14pt
bibliography: bibtex.bib
weight: 9
type: docs
---

<!--
- Compile from command-line
Rscript -e "rmarkdown::render('spchipseq.Rmd', c('BiocStyle::html_document'), clean=F); knitr::knit('spchipseq.Rmd', tangle=TRUE)"
-->

```{css, echo=FALSE}
pre code {
white-space: pre !important;
overflow-x: scroll !important;
word-break: keep-all !important;
word-wrap: initial !important;
}
```

```{r style, echo = FALSE, results = 'asis'}
BiocStyle::markdown()
options(width=60, max.print=1000)
knitr::opts_chunk$set(
    eval=as.logical(Sys.getenv("KNITR_EVAL", "TRUE")),
    cache=as.logical(Sys.getenv("KNITR_CACHE", "TRUE")), 
    tidy.opts=list(width.cutoff=60), tidy=TRUE)
```

```{r setup, echo=FALSE, message=FALSE, wwarning=FALSE, eval=FALSE}
suppressPackageStartupMessages({
    library(systemPipeR)
})
```

<div style="text-align: right"> 
Source code downloads: &nbsp; &nbsp;
[ [.Rmd](https://raw.githubusercontent.com/tgirke/GEN242//main/content/en/tutorials/spchipseq/spchipseq.Rmd) ] &nbsp; &nbsp; 
[ [.R](https://raw.githubusercontent.com/tgirke/GEN242/main/content/en/tutorials/spchipseq/spchipseq.R) ] 
</div>


# Introduction

The following analyzes the ChIP-Seq data from Kaufman et al. [-@Kaufmann2010-me] using 
for peak calling MACS2 where the uninduced sample serves as input (reference). Prior to running
this analysis the corresponding FASTQ files need to be downloaded following the instructions
[here](https://girke.bioinformatics.ucr.edu/GEN242/assignments/projects/project_data/). 

For learning purposes one can use the much smaller toy data set and ChIP-Seq `Rmd`
workflow instance provided by the `systemPipeRdata` package. This toy workflow instance 
can be conveniently obtained by running `genWorkenvir(workflow='chipseq')` (see
below). The FASTQ data used for this toy instance are the same as for the
RNA-Seq workflow, but the analysis code in the `Rmd` file is almost identical
to the ChIP-Seq workflow below. 

For the analysis of the Kaufman et al. [-@Kaufmann2010-me] data set (see download [here](https://girke.bioinformatics.ucr.edu/GEN242/assignments/projects/project_data/))
users want to use the `Rmd` instance linked from the top right corner of this page. 
Additional detail about this is provided [here](https://girke.bioinformatics.ucr.edu/GEN242/assignments/projects/project_data/#workflow-rmd-file).

## Experimental design

Typically, users want to specify here all information relevant for the
analysis of their NGS study. This includes detailed descriptions of
FASTQ files, experimental design, reference genome, gene annotations,
etc.

## Workflow environment

<font color="red">NOTE: this section</font> describes how to set up the proper 
environment (directory structure) for running `systemPipeR` workflows. After 
mastering this task the workflow run instructions <font color="red">can be deleted</font> 
since they are not expected to be included in a final HTML/PDF report of a workflow.

1. If a remote system or cluster is used, then users need to log in to the
   remote system first. The following applies to an HPC cluster (_e.g._ HPCC
   cluster). 

   A terminal application needs to be used to log in to a user's cluster account. Next, one
   can open an interactive session on a computer node with `srun --x11`. More details about
   argument settings for `srun` are available in this [HPCC
   manual](http://hpcc.ucr.edu/manuals_linux-cluster_jobs.html#partitions) or
   the HPCC section of this website
   [here](https://girke.bioinformatics.ucr.edu/GEN242/tutorials/linux/linux/#job-submission-with-sbatch).
   Next, load the R version required for running the workflow with `module load`. Sometimes it may be necessary to
   first unload an active software version before loading another version, _e.g._ `module unload R`.

## Experimental design

Typically, users want to specify here all information relevant for the
analysis of their NGS study. This includes detailed descriptions of
FASTQ files, experimental design, reference genome, gene annotations,
etc.

## Workflow environment

<font color="red">NOTE: this section</font> describes how to set up the proper 
environment (directory structure) for running `systemPipeR` workflows. After 
mastering this task the workflow run instructions <font color="red">can be deleted</font> 
since they are not expected to be included in a final HTML/PDF report of a workflow.

1. If a remote system or cluster is used, then users need to log in to the
   remote system first. The following applies to an HPC cluster (_e.g._ HPCC
   cluster). 

   A terminal application needs to be used to log in to a user's cluster account. Next, one
   can open an interactive session on a computer node with `srun --x11`. More details about
   argument settings for `srun` are available in this [HPCC
   manual](http://hpcc.ucr.edu/manuals_linux-cluster_jobs.html#partitions) or
   the HPCC section of this website
   [here](https://girke.bioinformatics.ucr.edu/GEN242/tutorials/linux/linux/#job-submission-with-sbatch).
   Next, load the R version required for running the workflow with `module load`. Sometimes it may be necessary to
   first unload an active software version before loading another version, _e.g._ `module unload R`.

```{bash, eval=FALSE}
srun --x11 --partition=gen242 --mem=20gb --cpus-per-task 8 --ntasks 1 --time 20:00:00 --pty bash -l
module unload R; module load R/4.1.2
```

2. Load a workflow template with the `genWorkenvir` function. This can be done from the command-line or from within R. 
   However, only one of the two options needs to be used.

From command-line

```{bash, eval=FALSE}
$ Rscript -e "systemPipeRdata::genWorkenvir(workflow='chipseq')"
$ cd chipseq
```

From R

```{r gen_workflow_envir, eval=FALSE}
library(systemPipeRdata)
genWorkenvir(workflow="chipseq")
setwd("chipseq")
```

3. Optional: if the user wishes to use another `Rmd` file than the template 
instance provided by the `genWorkenvir` function, then it can be copied or downloaded 
into the root directory of the workflow environment (_e.g._ with `cp` or `wget`).

4. Now one can open from the root directory of the workflow the corresponding 
R Markdown script (_e.g._ systemPipeChIPseq.Rmd) using an R IDE, such as _nvim-r_, 
_ESS_ or RStudio. Subsequently, the workflow can be run as outlined below. For 
learning purposes it is recommended to run workflows for the first time interactively. 
Once all workflow steps are understood and possibly modified to custom needs, 
one can run the workflow from start to finish with a single command using `runWF()`.

## Load packages

The `systemPipeR` package needs to be loaded to perform the analysis 
steps shown in this report [@H_Backman2016-bt]. The package allows users
to run the entire analysis workflow interactively or with a single command 
while also generating the corresponding analysis report. For details
see `systemPipeR's` main [vignette](http://www.bioconductor.org/packages/devel/bioc/vignettes/systemPipeR/inst/doc/systemPipeR.html).

```{r load_systempiper, eval=TRUE, message=FALSE, warning=FALSE}
library(systemPipeR)
```

To apply workflows to custom data, the user needs to modify the _`targets`_ file and if
necessary update the corresponding parameter (_`.cwl`_ and _`.yml`_) files. 
A collection of pre-generated _`.cwl`_ and _`.yml`_ files are provided in the _`param/cwl`_ subdirectory 
of each workflow template. They are also viewable in the GitHub repository of _`systemPipeRdata`_ ([see
here](https://github.com/tgirke/systemPipeRdata/tree/master/inst/extdata/param/cwl)).
For more information of the structure of the *targets* file, please consult the documentation 
[here](http://www.bioconductor.org/packages/release/bioc/vignettes/systemPipeR/inst/doc/systemPipeR.html#25_structure_of_targets_file). More details about the new parameter files from `systemPipeR` can be found [here](http://www.bioconductor.org/packages/release/bioc/vignettes/systemPipeR/inst/doc/systemPipeR.html#26_structure_of_the_new_param_files_and_construct_sysargs2_container). 

## Import custom functions

Custom functions for the challenge projects can be imported with the source
command from a local R script (here [challengeProject_Fct.R](https://raw.githubusercontent.com/tgirke/GEN242//main/content/en/tutorials/spchipseq/challengeProject_Fct.R)). Skip this step if such a script is not available. Alternatively, 
these functions can be loaded from a custom R package. 

```{r load_custom_fct, eval=FALSE, message=FALSE}
source("challengeProject_Fct.R")
```

## Experiment definition provided by `targets` file

The `targets` file defines all FASTQ files and sample comparisons of the analysis workflow. 
If needed the tab separated (TSV) version of this file can be downloaded from [here](https://github.com/tgirke/GEN242/tree/main/content/en/assignments/Projects/targets_files)
and the corresponding Google Sheet is [here](https://docs.google.com/spreadsheets/d/1w9V3JDOsXR8qW_qNqoXO0E0UR_Oev4k7-o2y6NMDTt8/edit#gid=472150521).

```{r load_targets_file, eval=TRUE}
targetspath <- "targets_chipseq.txt"
targets <- read.delim(targetspath, comment.char = "#")
knitr::kable(targets)
```

# Workflow steps

This tutorial will demonstrate how to either build and run the workflow automatically, 
or in an interactive mode by appending each step with the `appendStep` method. In both
cases the `SYSargsList` object will be populated with the instructions for running each
workflow step, while supporting both command-line steps as well as line-wise R commands 
defined in the corresponding code chunks of this or any `Rmd` file that has been properly
formatted.

To create a workflow within _`systemPipeR`_, we can start by defining an empty
`SYSargsList` container. When restarting an existing workflow one
can set `resume=TRUE` under the `SPRproject()` function call. 

```{r create_workflow, message=FALSE, eval=FALSE}
library(systemPipeR)
sal <- SPRproject()
sal
sal <- importWF(sal, file_path = "systemPipeChIPseq.Rmd")
sal
resources <- list(conffile=".batchtools.conf.R",
                  template="batchtools.slurm.tmpl", 
                  Njobs=8, 
                  walltime=180, ## minutes
                  ntasks=1,
                  ncpus=4, 
                  memory=4096, ## Mb
                  partition = "gen242"
                  )
# sal <- SPRproject(resume=TRUE)
sal <- addResources(sal, step = c("preprocessing", "bowtie2_alignment"), resources = resources)
getRversion() # tested on 4.2.0
system("hostname")
sal <- runWF(sal)
sal <- renderReport(sal) # URL: https://cluster.hpcc.ucr.edu/~ttest/chipseq/SPR_Report.html
rmarkdown::render("systemPipeChIPseq.Rmd", clean=TRUE, output_format="BiocStyle::html_document")
```

Next, the `importWF` function will load the entire workflow into the `SYSargsList` object
(here `sal`). Subsequently, the `runWF()` function will run the workflow from start to finish. 
If needed, specific workflow steps can be executed by assigning their corresponding
position numbers within the workflow to the `steps` argument (see `?runWF`). After completion of the workflow
one can render a scientific analysis report in HTML format with the `renderReport()` 
function that uses R Markdown internally.

```{r importWF, message=FALSE, eval=FALSE}
sal <- importWF(sal, file_path = "systemPipeChIPseq.Rmd") ## Import all the Workflow steps
sal
sal <- runWF(sal) # Runs workflow
sal <- renderReport(sal)  # Renders report
```

## Required packages and resources

The `systemPipeR` package needs to be loaded [@H_Backman2016-bt].

```{r load_SPR, message=FALSE, eval=FALSE, spr=TRUE}
appendStep(sal) <- LineWise(code = {
                library(systemPipeR)
                }, step_name = "load_SPR")
```

## Read preprocessing

### Read quality filtering and trimming

The function `preprocessReads` allows to apply predefined or custom
read preprocessing functions to all FASTQ files referenced in a
`SYSargsList` container, such as quality filtering or adapter trimming
routines. The paths to the resulting output FASTQ files are stored in the
`outfiles` slot of the `SYSargsList` object. The following example performs adapter trimming with
the `trimLRPatterns` function from the `Biostrings` package.
After the trimming step a new targets file is generated (here
`targets_trim.txt`) containing the paths to the trimmed FASTQ files.
The new targets file can be used for the next workflow step with an updated
`SYSargs2` instance, _e.g._ running the NGS alignments using the
trimmed FASTQ files.

Here, we are appending this step to the `SYSargsList` object created previously. 
All the parameters are defined on the `preprocessReads/preprocessReads-pe.yml` and 
`preprocessReads/preprocessReads-pe.cwl` files.

```{r preprocessing, message=FALSE, eval=FALSE, spr=TRUE}
appendStep(sal) <- SYSargsList(
    step_name = "preprocessing",
    targets = "targets_chipseq.txt", dir = TRUE,
    wf_file = "preprocessReads/preprocessReads-se.cwl",
    input_file = "preprocessReads/preprocessReads-se.yml",
    dir_path = "param/cwl",
    inputvars = c(
        FileName = "_FASTQ_PATH1_",
        SampleName = "_SampleName_"
    ),
    dependency = c("load_SPR"))
```

After, we can check the `trimLRPatterns` function in input parameter:

```{r editing_preprocessing, message=FALSE, eval=FALSE}
yamlinput(sal, "preprocessing")$Fct
```

After the preprocessing step, the `outfiles` files can be used to generate the new 
targets files containing the paths to the trimmed FASTQ files. The new targets 
information can be used for the next workflow step instance, _e.g._ running the 
NGS alignments with the trimmed FASTQ files. The `appendStep` function is
automatically handling this connectivity between steps. Please check the `Alignments` 
step for more details.

### FASTQ quality report

The following `seeFastq` and `seeFastqPlot` functions generate and plot a series of useful 
quality statistics for a set of FASTQ files including per cycle quality box
plots, base proportions, base-level quality trends, relative k-mer
diversity, length and occurrence distribution of reads, number of reads
above quality cutoffs and mean quality distribution. The results are
written to a PDF file named `fastqReport.png`.

```{r fastq_report, eval=FALSE, message=FALSE, spr=TRUE}
appendStep(sal) <- LineWise(
    code = {
        fastq <- getColumn(sal, step = "preprocessing", "targetsWF", column = 1)
        fqlist <- seeFastq(fastq = fastq, batchsize = 10000, klength = 8)
        png("./results/fastqReport.png", height = 1700, width = 4*length(fqlist)*98)
        seeFastqPlot(fqlist)
        dev.off()
    }, step_name = "fastq_report", 
    dependency = "preprocessing")
``` 

![](./results/fastqReport.png)
<div align="center">Figure 1: FASTQ quality report for 7 samples.</div></br>

## Alignments

### Read mapping with `Bowtie2` 

The NGS reads of this project will be aligned with `Bowtie2` against the
reference genome sequence [@Langmead2012-bs]. The parameter settings of the
Bowtie2 index are defined in the `bowtie2-index.cwl` and `bowtie2-index.yml` files. 

Building the index:

```{r bowtie2_index, eval=FALSE, spr=TRUE}
appendStep(sal) <- SYSargsList(
    step_name = "bowtie2_index",
    dir = FALSE, targets = NULL,
    wf_file = "bowtie2/bowtie2-index.cwl",
    input_file = "bowtie2/bowtie2-index.yml",
    dir_path = "param/cwl",
    inputvars = NULL,
    dependency = c("preprocessing")
)
```

The parameter settings of the aligner are defined in the `workflow_bowtie2-se.cwl` 
and `workflow_bowtie2-se.yml` files. The following shows how to construct the 
corresponding *SYSargsList* object.

In ChIP-Seq experiments it is usually more appropriate to eliminate reads mapping 
to multiple locations. To achieve this, users want to remove the argument setting 
`-k 50 non-deterministic` in the configuration files.

```{r bowtie2_alignment, eval=FALSE, spr=TRUE}
appendStep(sal) <- SYSargsList(
    step_name = "bowtie2_alignment",
    dir = TRUE,
    targets = "targets_chipseq.txt",
    wf_file = "workflow-bowtie2/workflow_bowtie2-se.cwl",
    input_file = "workflow-bowtie2/workflow_bowtie2-se.yml",
    dir_path = "param/cwl",
    inputvars = c(
        FileName = "_FASTQ_PATH1_",
        SampleName = "_SampleName_"
    ),
    dependency = c("bowtie2_index")
)
```

To double-check the command line for each sample, please use the following:

```{r bowtie2_alignment_check, eval=FALSE}
cmdlist(sal, step="bowtie2_alignment", targets=1)
```

### Read and alignment stats

The following provides an overview of the number of reads in each sample
and how many of them aligned to the reference.

```{r align_stats, eval=FALSE, spr=TRUE}
appendStep(sal) <- LineWise(
    code = {
        fqpaths <- getColumn(sal, step = "bowtie2_alignment", "targetsWF", column = "FileName")
        bampaths <- getColumn(sal, step = "bowtie2_alignment", "outfiles", column = "samtools_sort_bam")
        read_statsDF <- alignStats(args = bampaths, fqpaths = fqpaths, pairEnd = TRUE)
        write.table(read_statsDF, "results/alignStats.xls", row.names=FALSE, quote=FALSE, sep="\t")
        }, 
    step_name = "align_stats", 
    dependency = "bowtie2_alignment")
```

## Create symbolic links for viewing BAM files in IGV

The `symLink2bam` function creates symbolic links to view the BAM alignment
files in a genome browser such as IGV without moving these large files to a
local system. The corresponding URLs are written to a file with a path
specified under `urlfile`, here `IGVurl.txt`. Please replace the directory
and the user name. The following parameter settings will create a
subdirectory under `~/.html` called `somedir` of the user account. The user
name under `urlbase`, here `<username>`, needs to be changed to the corresponding
user name of the person running this function.

```{r bam_IGV, eval=FALSE, spr=TRUE}
appendStep(sal) <- LineWise(
    code = {
        bampaths <- getColumn(sal, step = "bowtie2_alignment", "outfiles", 
                              column = "samtools_sort_bam")
        bampaths <- setNames(normalizePath(bampaths), names(bampaths))
        symLink2bam(
            sysargs = bampaths, htmldir = c("~/.html/", "somedir/"),
            urlbase="https://cluster.hpcc.ucr.edu/~ttest/",
            urlfile = "./results/IGVurl.txt")
    },
    step_name = "bam_IGV",
    dependency = "bowtie2_alignment",
    run_step = "optional"
)
```

## Peak calling with MACS2

## Merge BAM files of replicates prior to peak calling

Merging BAM files of technical and/or biological replicates can improve
the sensitivity of the peak calling by increasing the depth of read
coverage. The `mergeBamByFactor` function merges BAM files based on grouping information
specified by a `factor`, here the `Factor` column of the imported targets file. 
It also returns an updated `targets` object containing the paths to the
merged BAM files as well as to any unmerged files without replicates.
The updated `targets` object can be used to update the `SYSargsList` object.

This step can be skipped if merging of BAM files is not desired.

```{r merge_bams, eval=FALSE, spr=TRUE}
appendStep(sal) <- LineWise(
    code = {
        bampaths <- getColumn(sal, step = "bowtie2_alignment", "outfiles", column = "samtools_sort_bam")
        merge_bams <- mergeBamByFactor(args=bampaths, targetsDF = targetsWF(sal)[["bowtie2_alignment"]], overwrite=TRUE)
        updateColumn(sal, step = "merge_bams", position = "targetsWF") <- merge_bams
        writeTargets(sal, step = "merge_bams", file = "targets_merge_bams.txt", overwrite = TRUE)
    },
    step_name = "merge_bams",
    dependency = "bowtie2_alignment"
)
```

### Peak calling with input/reference sample

MACS2 can perform peak calling on ChIP-Seq data with and without input
samples [@Zhang2008-pc]. 

The following performs peak calling with input sample. The input sample 
can be most conveniently specified in the `SampleReference` column of the 
initial `targets` file. The `writeTargetsRef` function uses this 
information to create a `targets` file intermediate for running MACS2 
with the corresponding input sample(s).

```{r writeTargetsRef, eval=FALSE, spr=TRUE}
appendStep(sal) <- LineWise(
    code = {
        writeTargetsRef(infile = "targets_merge_bams.txt", 
                outfile = "targets_bam_ref.txt", silent = FALSE, overwrite = TRUE)
    },
    step_name = "writeTargetsRef",
    dependency = "merge_bams"
)
```

```{r call_peaks_macs_withref, eval=FALSE, spr=TRUE}
appendStep(sal) <- SYSargsList(
    step_name = "call_peaks_macs_withref",
    targets = "targets_bam_ref.txt",
    wf_file = "MACS2/macs2-input.cwl",
    input_file = "MACS2/macs2-input.yml",
    dir_path = "param/cwl",
    inputvars = c(
        FileName1 = "_FASTQ_PATH1_",
        FileName2 = "_FASTQ_PATH2_",
        SampleReference = "_SampleName_"
    ),
    id = "SampleReference", 
    dependency = c("writeTargetsRef")
)
```

The peak calling results from MACS2 are written for each sample to
separate files in the `results/call_peaks_macs_withref` directory. They are 
named after the corresponding files with extensions used by MACS2.

## Annotate peaks with genomic context

### Annotation with `ChIPseeker` package

The following annotates the identified peaks with genomic context information
using the `ChIPseeker` package [@Yu2015-xu].

```{r annotation_ChIPseeker, eval=FALSE, spr=TRUE}
appendStep(sal) <- LineWise(
    code = {
        library(ChIPseeker); library(GenomicFeatures)
        peaks_files <- getColumn(sal, step = "call_peaks_macs_withref", "outfiles", column = "peaks_xls")
        txdb <- suppressWarnings(makeTxDbFromGFF(file="data/tair10.gff", format="gff", dataSource="TAIR", 
                                organism="Arabidopsis thaliana"))
        for(i in seq(along=peaks_files)) {
            peakAnno <- annotatePeak(peaks_files[i], TxDb=txdb, verbose=FALSE)
            df <- as.data.frame(peakAnno)
            outpaths <- paste0("./results/", names(peaks_files), "_ChIPseeker_annotated.xls")
            names(outpaths) <- names(peaks_files)
            write.table(df, outpaths[i], quote=FALSE, row.names=FALSE, sep="\t")
        }
        updateColumn(sal, step = "annotation_ChIPseeker", position = "outfiles") <- data.frame(outpaths)
    },
    step_name = "annotation_ChIPseeker",
    dependency = "call_peaks_macs_withref"
)

```

The peak annotation results are written to the `results` directory. 

Summary plots provided by the `ChIPseeker` package. Here applied only to one sample
for demonstration purposes.

```{r ChIPseeker_plots, eval=FALSE, spr=TRUE}
appendStep(sal) <- LineWise(
    code = {
        peaks_files <- getColumn(sal, step = "call_peaks_macs_withref", "outfiles", column = "peaks_xls")
        peak <- readPeakFile(peaks_files[1])
        png("results/peakscoverage.png", width=680)
        covplot(peak, weightCol="X.log10.pvalue.")
        dev.off()
        png("results/peaksHeatmap.png", width=680)
        peakHeatmap(peaks_files[1], TxDb=txdb, upstream=1000, downstream=1000, 
                    color="red")
        dev.off()
        png("results/peaksProfile.png", width=680)
        plotAvgProf2(peaks_files[1], TxDb=txdb, upstream=1000, downstream=1000, 
                     xlab="Genomic Region (5'->3')", ylab = "Read Count Frequency", 
                     conf=0.05)
        dev.off()
    },
    step_name = "ChIPseeker_plots",
    dependency = "annotation_ChIPseeker"
)
```

## Count reads overlapping peaks

The `countRangeset` function is a convenience wrapper to perform read counting
iteratively over serveral range sets, here peak range sets. Internally,
the read counting is performed with the `summarizeOverlaps` function from the 
`GenomicAlignments` package. The resulting count tables are directly saved to 
files, one for each peak set.

```{r count_peak_ranges, eval=FALSE, spr=TRUE}
appendStep(sal) <- LineWise(
    code = {
        library(GenomicRanges)
        bam_files <- getColumn(sal, step = "bowtie2_alignment", "outfiles", column = "samtools_sort_bam")
        args <- getColumn(sal, step = "call_peaks_macs_withref", "outfiles", column = "peaks_xls")
        outfiles <- paste0("./results/", names(args), "_countDF.xls")
        bfl <- BamFileList(bam_files, yieldSize=50000, index=character())
        countDFnames <- countRangeset(bfl, args, outfiles, mode="Union", ignore.strand=TRUE)
        updateColumn(sal, step = "count_peak_ranges", position = "outfiles") <- data.frame(countDFnames)
    },
    step_name = "count_peak_ranges",
    dependency = "call_peaks_macs_withref",
)
```

Shows count table generated in previous step (`results/AP1_1_countDF.xls`).
To avoid slowdowns of the load time of this page, ony 200 rows of the source 
table are imported into the below `datatable` view .

```{r show_counts_table, eval=FALSE}
countDF <- read.delim("results/AP1_1_countDF.xls")[1:200,]
colnames(countDF)[1] <- "PeakIDs"
library(DT)
datatable(countDF)
```

## Differential binding analysis

The `runDiff` function performs differential binding analysis in batch mode for
several count tables using `edgeR` or `DESeq2` [@Robinson2010-uk; @Love2014-sh].
Internally, it calls the functions `run_edgeR` and `run_DESeq2`. It also returns 
the filtering results and plots from the downstream `filterDEGs` function using 
the fold change and FDR cutoffs provided under the `dbrfilter` argument.

```{r diff_bind_analysis, eval=FALSE, spr=TRUE}
appendStep(sal) <- LineWise(
    code = {
        countDF_files <- getColumn(sal, step = "count_peak_ranges", "outfiles")
        outfiles <- paste0("./results/", names(countDF_files), "_peaks_edgeR.xls")
        names(outfiles) <- names(countDF_files)
        cmp <- readComp(file =stepsWF(sal)[["bowtie2_alignment"]],
                        format="matrix") 
        dbrlist <- runDiff(args=countDF_files, outfiles = outfiles, diffFct=run_edgeR, 
                           targets=targetsWF(sal)[["bowtie2_alignment"]], cmp=cmp[[1]], 
                           independent=TRUE, dbrfilter=c(Fold=2, FDR=1))
    },
    step_name = "diff_bind_analysis",
    dependency = "count_peak_ranges",
)
```

## GO term enrichment analysis

### Obtain gene-to-GO mappings

The following shows how to obtain gene-to-GO mappings from `biomaRt` (here for *A.
thaliana*) and how to organize them for the downstream GO term
enrichment analysis. Alternatively, the gene-to-GO mappings can be
obtained for many organisms from Bioconductor’s `*.db` genome annotation
packages or GO annotation files provided by various genome databases.
For each annotation this relatively slow preprocessing step needs to be
performed only once. Subsequently, the preprocessed data can be loaded
with the `load` function as shown in the next subsection.


### GO term enrichment analysis

The following performs GO term enrichment analysis for each annotated peak
set. Note: the following assumes that the GO annotation data exists under
`data/GO/catdb.RData`. 

```{r get_go_annot, eval=FALSE, spr=TRUE}
appendStep(sal) <- LineWise(
    code = {
        library("biomaRt")
        # listMarts() # To choose BioMart database
        # listMarts(host="plants.ensembl.org")
        m <- useMart("plants_mart", host="https://plants.ensembl.org")
        m <- useMart("plants_mart", dataset="athaliana_eg_gene", host="https://plants.ensembl.org")
        go <- getBM(attributes=c("go_id", "tair_locus", "namespace_1003"), mart=m)
        go <- go[go[,3]!="",]; go[,3] <- as.character(go[,3])
        go[go[,3]=="molecular_function", 3] <- "F"; go[go[,3]=="biological_process", 3] <- "P"; go[go[,3]=="cellular_component", 3] <- "C"
        go[1:4,]
        if(!dir.exists("./data/GO")) dir.create("./data/GO")
        write.table(go, "data/GO/GOannotationsBiomart_mod.txt", quote=FALSE, row.names=FALSE, col.names=FALSE, sep="\t")
        catdb <- makeCATdb(myfile="data/GO/GOannotationsBiomart_mod.txt", lib=NULL, org="", colno=c(1,2,3), idconv=NULL)
        save(catdb, file="data/GO/catdb.RData")
    }, 
    step_name = "get_go_annot",
    dependency = "annotation_ChIPseeker",
)
```

### GO term enrichment analysis

The following performs GO term enrichment analysis for each annotated peak
set. Note: the following assumes that the GO annotation data exists under
`data/GO/catdb.RData`. 

```{r go_enrich, eval=FALSE, spr=TRUE}
appendStep(sal) <- LineWise(
    code = {
        annofiles <- getColumn(sal, step = "annotation_ChIPseeker", "outfiles")
        gene_ids <- sapply(annofiles, 
                           function(x) unique(as.character
                                              (read.delim(x)[,"geneId"])), simplify=FALSE)
        load("data/GO/catdb.RData")
        BatchResult <- GOCluster_Report(catdb=catdb, setlist=gene_ids, method="all", 
                                        id_type="gene", CLSZ=2, cutoff=0.9, 
                                        gocats=c("MF", "BP", "CC"), recordSpecGO=NULL)
        write.table(BatchResult, "results/GOBatchAll.xls", quote=FALSE, row.names=FALSE, sep="\t")
    },
    step_name = "go_enrich",
    dependency = "annotation_ChIPseeker",
)
```

Shows GO term enrichment results from previous step. The last gene identifier column (10) 
of this table has been excluded in this viewing instance to minimize the complexity of the
result.
To avoid slowdowns of the load time of this page, only 50 rows of the source 
table are imported into the below `datatable` view .

```{r show_GO_table, eval=FALSE}
BatchResult <- read.delim("results/GOBatchAll.xls")[1:50,]
library(DT)
datatable(BatchResult[,-10], options = list(scrollX = TRUE, autoWidth = TRUE))
```

## Motif analysis

### Parse DNA sequences of peak regions from genome

Enrichment analysis of known DNA binding motifs or _de novo_ discovery
of novel motifs requires the DNA sequences of the identified peak
regions. To parse the corresponding sequences from the reference genome,
the `getSeq` function from the `Biostrings` package can be used. The 
following example parses the sequences for each peak set and saves the 
results to separate FASTA files, one for each peak set. In addition, the 
sequences in the FASTA files are ranked (sorted) by increasing p-values 
as expected by some motif discovery tools, such as `BCRANK`.

```{r parse_peak_sequences, eval=FALSE, spr=TRUE}
appendStep(sal) <- LineWise(
    code = {
        library(Biostrings); library(seqLogo); library(BCRANK)
        rangefiles <- getColumn(sal, step = "call_peaks_macs_withref", "outfiles")
        for(i in seq(along=rangefiles)) {
            df <- read.delim(rangefiles[i], comment="#")
            peaks <- as(df, "GRanges")
            names(peaks) <- paste0(as.character(seqnames(peaks)), "_", start(peaks),
                                   "-", end(peaks))
            peaks <- peaks[order(values(peaks)$X.log10.pvalue., decreasing=TRUE)]
            pseq <- getSeq(FaFile("./data/tair10.fasta"), peaks)
            names(pseq) <- names(peaks)
            writeXStringSet(pseq, paste0(rangefiles[i], ".fasta"))
            }
        },
    step_name = "parse_peak_sequences",
    dependency = "call_peaks_macs_withref"
)
```

### Motif discovery with `BCRANK`

The Bioconductor package `BCRANK` is one of the many tools available for 
_de novo_ discovery of DNA binding motifs in peak regions of ChIP-Seq
experiments. The given example applies this method on the first peak
sample set and plots the sequence logo of the highest ranking motif.

```{r bcrank_enrich, eval=FALSE, spr=TRUE}
appendStep(sal) <- LineWise(
    code = {
        library(Biostrings); library(seqLogo); library(BCRANK)
        rangefiles <- getColumn(sal, step = "call_peaks_macs_withref", "outfiles")
        set.seed(0)
        BCRANKout <- bcrank(paste0(rangefiles[1], ".fasta"), restarts=25,
                            use.P1=TRUE, use.P2=TRUE)
        toptable(BCRANKout)
        topMotif <- toptable(BCRANKout, 1)
        weightMatrix <- pwm(topMotif, normalize = FALSE)
        weightMatrixNormalized <- pwm(topMotif, normalize = TRUE)
        png("results/seqlogo.png", width=680)
        seqLogo(weightMatrixNormalized)
        dev.off()
        },
    step_name = "bcrank_enrich",
    dependency = "call_peaks_macs_withref"
)
```

![](./results/seqlogo.png)
<div align="center">Figure 2: One of the motifs identified by `BCRANK`</div></br>

## Version Information

```{r sessionInfo, eval=FALSE, spr=TRUE}
appendStep(sal) <- LineWise(
    code = {
        sessionInfo()
        },
    step_name = "sessionInfo",
    dependency = "bcrank_enrich"
)
```

```{r sessionInfo2, eval=TRUE}
        sessionInfo()
```

# Running workflow

## Interactive job submissions in a single machine

For running the workflow, `runWF` function will execute all the steps store in 
the workflow container. The execution will be on a single machine without 
submitting to a queuing system of a computer cluster. 

```{r runWF, eval=FALSE}
sal <- runWF(sal)
```

## Parallelization on clusters

Alternatively, the computation can be greatly accelerated by processing many files 
in parallel using several compute nodes of a cluster, where a scheduling/queuing
system is used for load balancing. 

The `resources` list object provides the number of independent parallel cluster 
processes defined under the `Njobs` element in the list. The following example 
will run 18 processes in parallel using each 4 CPU cores. 
If the resources available on a cluster allow running all 18 processes at the 
same time, then the shown sample submission will utilize in a total of 72 CPU cores.

Note, `runWF` can be used with most queueing systems as it is based on utilities 
from the `batchtools` package, which supports the use of template files (_`*.tmpl`_)
for defining the run parameters of different schedulers. To run the following 
code, one needs to have both a `conffile` (see _`.batchtools.conf.R`_ samples [here](https://mllg.github.io/batchtools/)) 
and a `template` file (see _`*.tmpl`_ samples [here](https://github.com/mllg/batchtools/tree/master/inst/templates)) 
for the queueing available on a system. The following example uses the sample 
`conffile` and `template` files for the Slurm scheduler provided by this package. 

The resources can be appended when the step is generated, or it is possible to 
add these resources later, as the following example using the `addResources` 
function:

```{r runWF_cluster, eval=FALSE}
resources <- list(conffile=".batchtools.conf.R",
                  template="batchtools.slurm.tmpl", 
                  Njobs=18, 
                  walltime=120, ## minutes
                  ntasks=1,
                  ncpus=4, 
                  memory=1024, ## Mb
                  partition = "short"
                  )
sal <- addResources(sal, c("bowtie2_alignment"), resources = resources)
sal <- runWF(sal)
```

## Visualize workflow

_`systemPipeR`_ workflows instances can be visualized with the `plotWF` function.

```{r plotWF, eval=TRUE}
plotWF(sal)
```

## Checking workflow status

To check the summary of the workflow, we can use:

```{r statusWF, eval=FALSE}
sal
statusWF(sal)
```

## Accessing logs report

_`systemPipeR`_ compiles all the workflow execution logs in one central location,
making it easier to check any standard output (`stdout`) or standard error
(`stderr`) for any command-line tools used on the workflow or the R code stdout.

```{r logsWF, eval=FALSE}
sal <- renderLogs(sal)
```


# Funding

This project was supported by funds from the National Institutes of
Health (NIH) and the National Science Foundation (NSF).

# References