#+STARTUP: showall
#+TITLE: Hledger Flow: Step-By-Step
#+AUTHOR:
#+REVEAL_TRANS: default
#+REVEAL_THEME: beige
#+OPTIONS: num:nil
#+PROPERTY: header-args:sh :prologue exec 2>&1 :epilogue echo :

* Part 1

  This is the first part in a a series of step-by-step instructions.

  They are intended to be read in sequence. Head over to the [[file:README.org][docs README]] to see all parts.

* About This Document

This document is a [[https://www.offerzen.com/blog/literate-programming-empower-your-writing-with-emacs-org-mode][literate]] [[https://orgmode.org/worg/org-contrib/babel/intro.html][program]].
You can read it like a normal article, either [[https://github.com/apauley/hledger-flow/blob/master/docs/part1.org][on the web]] or [[https://pauley.org.za/hledger-flow/][as a slide show]].

But you can also [[https://github.com/apauley/hledger-flow][clone the repository]] and open [[https://raw.githubusercontent.com/apauley/hledger-flow/master/docs/part1.org][this org-mode file]] in emacs.
Then you can execute each code snippet by pressing =C-c C-c= with your cursor on the relevant code block.

* The Story of an African Chef

Meet Gawie de Groot. Gawie is a successful [[https://en.wikipedia.org/wiki/Gonimbrasia_belina#As_food][mopane worm]] chef, based in the northernmost section of the [[https://en.wikipedia.org/wiki/Kruger_National_Park][Kruger National Park]].

[[./img/mopane-worm-meal.jpg]]


#+BEGIN_SRC org :results none :exports none
Image downloaded from https://commons.wikimedia.org/wiki/File:Mopane-worm-meal.jpg
Author: Ling Symon
#+END_SRC

#+REVEAL: split

In times gone by he used to work in a big city as an IT professional.
But after a few years he decided to choose a quieter life in the bushveld.

He now works at a renowned bush restaurant aptly named the "=Grillerige Groen Goggatjie=".

#+REVEAL: split

Demand for his traditional African cuisine has sky-rocketed, with visitors from all over Zimbabwe, Mozambique and South Africa
coming to enjoy his dishes.

Business has been good the last few years, but Gawie wants to make sure that he'll also be OK when business is down.

It is time for Gawie to take charge of his finances.

* The First CSV Statement

Gawie followed the [[https://github.com/apauley/hledger-flow#build-instructions][build instructions]] and ended up with his very own =hledger-flow= executable.

#+REVEAL: split

Let's just run this =hledger-flow= command and see what happens...

#+NAME: hm-noargs
#+BEGIN_SRC sh :results output :exports both
hledger-flow
#+END_SRC

#+RESULTS: hm-noargs
#+begin_example
An hledger workflow focusing on automated statement import and classification:
https://github.com/apauley/hledger-flow#readme

Usage: hledger-flow ([-v|--verbose] [-H|--hledger-path HLEDGER-PATH] 
                      [--show-options] [--batch-size SIZE] [--sequential] 
                      (import | report) |
                      (-V|--version))

Available options:
  -h,--help                Show this help text
  -v,--verbose             Print more verbose output
  -H,--hledger-path HLEDGER-PATH
                           The full path to an hledger executable
  --show-options           Print the options this program will run with
  --batch-size SIZE        Parallel processing of files are done in batches of
                           the specified size. Default: 200. Ignored during
                           sequential processing.
  --sequential             Disable parallel processing
  -V,--version             Display version information

Available commands:
  import                   Uses hledger with your own rules and/or scripts to convert electronic statements into categorised journal files
  report                   Generate Reports

#+end_example

Well OK, at least it prints some helpful output.

#+REVEAL: split

Gawie wants to import his CSV statements.

Luckily the =import= subcommand has a bit of help text as well:

#+NAME: hm-import-help
#+BEGIN_SRC sh :results org :exports both
hledger-flow import --help
#+END_SRC

#+RESULTS: hm-import-help
#+begin_src org
Usage: hledger-flow import [DIR] [--start-year YEAR] [--new-files-only]
  Uses hledger with your own rules and/or scripts to convert electronic statements into categorised journal files

Available options:
  DIR                      The directory to import. Use the base directory for a
                           full import or a sub-directory for a partial import.
                           Defaults to the current directory. This behaviour is
                           changing: see --enable-future-rundir
  --start-year YEAR        Import only from the specified year and onwards,
                           ignoring previous years. By default all available
                           years are imported. Valid values include a 4-digit
                           year or 'current' for the current year
  --new-files-only         Don't regenerate transaction files if they are
                           already present. This applies to hledger journal
                           files as well as files produced by the preprocess and
                           construct scripts.
  -h,--help                Show this help text

#+end_src

#+REVEAL: split

Gawie starts by creating a new directory specifically for his hledger journals:

#+NAME: rm-fin-dir
#+BEGIN_SRC sh :results none :exports results
rm -rf my-finances
#+END_SRC

#+NAME: new-fin-dir
#+BEGIN_SRC sh :results none :exports both
mkdir my-finances
#+END_SRC

Now he can point the import to his new finances directory:
#+NAME: import1
#+BEGIN_SRC sh :results org :exports both
hledger-flow import ./my-finances
#+END_SRC

#+REVEAL: split

Hmmm, an error:
#+RESULTS: import1
#+begin_src org
hledger-flow: Unable to find an import directory at "./my-finances/" (or in any of its parent directories).

Have a look at the documentation for more information:
https://github.com/apauley/hledger-flow#getting-started

#+end_src

Gawie carefully interprets the error message using the skills he obtained during his years as an IT professional.

He concludes that =hledger-flow= expects to find its input files in specifically named directories.

#+REVEAL: split

Looking at the [[https://github.com/apauley/hledger-flow#input-files][documentation]] he sees there should be several account and bank-specific directories
under the =import= directory.

#+REVEAL: split

Gawie's salary is deposited into his cheque account at =Bogart Bank=, so this seems like a good account to start with:

#+NAME: first-input-file
#+BEGIN_SRC sh :results none :exports both
mkdir -p my-finances/import/gawie/bogart/cheque/1-in/2016/
cp Downloads/Bogart/123456789_2016-03-30.csv \
  my-finances/import/gawie/bogart/cheque/1-in/2016/
#+END_SRC

#+REVEAL: split

Let's see what our tree structure looks like now:
#+NAME: tree-after-1st-file
#+BEGIN_SRC sh :results org :exports both
tree my-finances/
#+END_SRC

#+RESULTS: tree-after-1st-file
#+begin_src org
my-finances/
`-- import
    `-- gawie
        `-- bogart
            `-- cheque
                `-- 1-in
                    `-- 2016
                        `-- 123456789_2016-03-30.csv

6 directories, 1 file

#+end_src

#+REVEAL: split

It is time to add what we have to source control.

#+NAME: git-init
#+BEGIN_SRC sh :results none :exports both
cd my-finances/
git init .
git add .
git commit -m 'Initial commit'
cd ..
#+END_SRC

#+REVEAL: split

Let's try the import again:
#+NAME: import2
#+BEGIN_SRC sh :results org :exports both
hledger-flow import ./my-finances
#+END_SRC

#+RESULTS: import2
#+begin_src org
I couldn't find an hledger rules file while trying to import
import/gawie/bogart/cheque/1-in/2016/123456789_2016-03-30.csv

I will happily use the first rules file I can find from any one of these 2 files:
import/gawie/bogart/cheque/bogart-cheque.rules
import/bogart.rules

Here is a bit of documentation about rules files that you may find helpful:
https://github.com/apauley/hledger-flow#rules-files

#+end_src

#+REVEAL: split

Another cryptic error.

This one is caused by a missing [[https://github.com/apauley/hledger-flow#the-rules-file][rules file]].

#+REVEAL: split

After looking through the [[http://hledger.org/csv.html][hledger documentation on CSV rules files]],
Gawie concludes that the dates in Bogart Bank's CSV statement is incompatible with basic logic, reason and decency.

Luckily he isn't the only one suffering at the hands of bureaucratic incompetence: someone else has already written [[https://github.com/apauley/fnb-csv-demoronizer][a script]] to
fix stupid dates like those used by Bogart Bank.

#+REVEAL: split

This looks like a job for a [[https://github.com/apauley/hledger-flow#the-preprocess-script][preprocess script]].

#+REVEAL: split

Gawie adds the CSV transformation script as a submodule to his repository:

#+NAME: git-submodule-demoronizer
#+BEGIN_SRC sh :results none :exports both
cd my-finances/
git submodule add https://github.com/apauley/fnb-csv-demoronizer.git
git commit -m 'Added submodule: fnb-csv-demoronizer'
cd ..
#+END_SRC

#+REVEAL: split

=hledger-flow= looks for a file named [[https://github.com/apauley/hledger-flow#the-preprocess-script][preprocess]] in the account directory.

#+REVEAL: split

Gawie just creates a symbolic link named =preprocess=.
This works because the downloaded script takes an input file and an output file as the first two positional arguments,
very much as the =preprocess= script would expect.
And luckily it ignores the other parameters that =hledger-flow= sends through.

#+REVEAL: split

#+NAME: symlink-demoronizer
#+BEGIN_SRC sh :results none :exports both
cd my-finances/import/gawie/bogart/cheque
ln -s ../../../../fnb-csv-demoronizer/fnb-csv-demoronizer preprocess
#+END_SRC

Now when we try the import again, it still displays an error due to our missing rules file:

#+REVEAL: split

#+NAME: import3
#+BEGIN_SRC sh :results org :exports both
hledger-flow import ./my-finances
#+END_SRC

#+RESULTS: import3
#+begin_src org
I couldn't find an hledger rules file while trying to import
import/gawie/bogart/cheque/2-preprocessed/2016/123456789_2016-03-30.csv

I will happily use the first rules file I can find from any one of these 2 files:
import/gawie/bogart/cheque/bogart-cheque.rules
import/bogart.rules

Here is a bit of documentation about rules files that you may find helpful:
https://github.com/apauley/hledger-flow#rules-files

#+end_src

This time we can see that our statement was preprocessed despite the rules file error:

#+NAME: head-preprocess
#+BEGIN_SRC sh :results org :exports both
head -n 2 my-finances/import/gawie/bogart/cheque/2-preprocessed/2016/123456789_2016-03-30.csv
#+END_SRC

#+RESULTS: head-preprocess
#+begin_src org
"5","'Nommer'","'Datum'","'Beskrywing1'","'Beskrywing2'","'Beskrywing3'","'Bedrag'","'Saldo'","'Opgeloopte Koste'"
"5","1","2016-03-01","#Monthly Bank Fee","","","-500.00","40000.00",""

#+end_src

#+REVEAL: split

Time for another git checkpoint.

#+NAME: git-checkpoint-preprocess
#+BEGIN_SRC sh :results none :exports both
cd my-finances/
git add .
git commit -m 'The preprocessed CSV now has dates we can work with!'
cd ..
#+END_SRC

#+REVEAL: split

Now that we have sane dates in a CSV file, let's try to create a [[http://hledger.org/manual.html#csv-rules][rules file]]:
#+NAME: bogart-cheque-rules-file
#+BEGIN_SRC hledger :tangle my-finances/import/gawie/bogart/cheque/bogart-cheque.rules
skip 1

fields _, _, date, desc1, desc2, desc3, amount, balance, _

currency R
status *

account1 Assets:Current:Gawie:Bogart:Cheque
description %desc1/%desc2/%desc3
#+END_SRC

Gawie saves this file as =my-finances/import/gawie/bogart/cheque/bogart-cheque.rules=.

#+REVEAL: split

#+NAME: tangle-rules
#+BEGIN_SRC emacs-lisp :results none :exports results
; Narrator: this just tells emacs to write out the rules file. Carry on.
; FIXME: This should just tangle the one relevant block, not all tangle blocks
(org-babel-tangle-file (buffer-file-name))
#+END_SRC

Time for another git checkpoint.

#+NAME: git-checkpoint-rules
#+BEGIN_SRC sh :results none :exports both
cd my-finances/
git add .
git commit -m 'A CSV rules file'
cd ..
#+END_SRC

#+REVEAL: split

This time the import is successful, and we see a number of newly generated files:
#+NAME: import4
#+BEGIN_SRC sh :results org :exports both
hledger-flow import ./my-finances
tree my-finances
#+END_SRC

#+REVEAL: split

#+RESULTS: import4
#+begin_src org
Wrote include files for 1 journals in 0.003716s
Imported 1/1 journals in 0.101596s
my-finances
|-- all-years.journal
|-- fnb-csv-demoronizer
|   |-- README.org
|   `-- fnb-csv-demoronizer
`-- import
    |-- 2016-include.journal
    |-- all-years.journal
    `-- gawie
        |-- 2016-include.journal
        |-- all-years.journal
        `-- bogart
            |-- 2016-include.journal
            |-- all-years.journal
            `-- cheque
                |-- 1-in
                |   `-- 2016
                |       `-- 123456789_2016-03-30.csv
                |-- 2-preprocessed
                |   `-- 2016
                |       `-- 123456789_2016-03-30.csv
                |-- 2016-include.journal
                |-- 3-journal
                |   `-- 2016
                |       `-- 123456789_2016-03-30.journal
                |-- all-years.journal
                |-- bogart-cheque.rules
                `-- preprocess -> ../../../../fnb-csv-demoronizer/fnb-csv-demoronizer

11 directories, 16 files

#+end_src

#+REVEAL: split

Bogart Bank's CSV file has been transformed into an =hledger= journal file.

This is the first transaction in the file:
#+NAME: head-1st-journal
#+BEGIN_SRC sh :results org :exports both
head -n 3 my-finances/import/gawie/bogart/cheque/3-journal/2016/123456789_2016-03-30.journal
#+END_SRC

#+RESULTS: head-1st-journal
#+begin_src org
2016-03-01 * #Monthly Bank Fee//
    Assets:Current:Gawie:Bogart:Cheque        R-500.00 = R40000.00
    expenses:unknown                           R500.00

#+end_src

#+REVEAL: split

A final checkpoint and we're done with part 1.

#+NAME: git-checkpoint-1st-journal
#+BEGIN_SRC sh :results none :exports both
cd my-finances/
git add .
git commit -m 'My first imported journal'
cd ..
#+END_SRC

#+REVEAL: split

The story continues with [[file:part2.org][part 2]].