======================== beancount: CHANGES ======================== Note: This file contains a list changes in the 'v2' branch. The master branch will rely on the Git repository history. 2022-02-15 - Released 2.3.5 with only a few minor bugfixes. Mainly released for 3.10. 2021-03-20 - Fixed #639: Released 2.3.4. 2021-01-09 - Fixed bug in booking that assumed number is set on cost. I did this while running booking manually on entries freshly created on an importer (never ahd done that before, will be customary in v3). 2021-01-08 - Fixed recently introduced bug with the export script that reordered the columns. - Fixed bug with printer leaving trailing whitespace. Was annoying. 2020-12-20 - Merged half of branch pr128_seltzered_beanpriceupdate on this side of Beancount. The other side will be patched to http://github.com/beancount/beanprice. This adds some trimming functions to beancount.ops.lifetimes. 2020-12-14 - Fixed #588: Catch more narrow exceptions from plugins and render full stack when caught. 2020-11-25 - Fixed #584 - a bug which was occurring because a pathological example using a very large precision was used in inferring the precision of a currency and a subsequent attempt to quantize a large number exceeded the default precision. 2020-11-22 - Fixed bug in parser.parse_file() which when passed a filename in would leave the file unclosed. - Fixed latent bug with key/value parsing (incorrect grammar rule, was working anyway somehow). - Internal changes to lexer and parser to minimize diffs with ongoing work in 'master' / C++ scanner/parser. 2020-11-17 - Made it legal to insert a balance entry after account closure. It is a common case that transactions happen on the last day of an account and the balance directive operates at the beginning of the day. It was arguable whether directives should be allowed after closure; notes and documents were allowed because statements and phone calls may still occur after an account is closed, but I suppose you can make the argument that a statement balance can also occur at a later date, so voila, make it legal. You still cannot post transactions to a closed account, which is the thing that really matters. 2020-11-15 - Render the tolerance in Balance directives (merged from Aaron Lindsay). 2020-11-01 - In the process of removing dependencies to the beancount.ops.holdings module, significantly sped up the net-worth-over-time script by using simple aggregation to an inventory (which is how it should be done). - Allow balance_by_account() method to compress positions that aren't booked, as an option. - Added split() method to Inventory, like segregate_units but with no constraints. - Fixed #255: Added a new beancount.core.prices.project() method to project prices to another cost currency. 2020-10-26 - Fixed Github Actions workflows and Wheel generation. - Released 2.3.3 to trigger new binaries. 2020-10-25 - Removed the part of setup_test checks running the binaries with --help, as they fail starting at 3.8. Someone with more time would create a test using pytest-virtualenv. 2020-10-17 - Fixed #566: Duplicate postings on entries should count toward distinct hashes. 2020-10-10 - Moved Emacs support to its own repository (http://github.com/beancount/beancount-mode). 2020-09-12 - Added metadata on implicitly generated Price directives (from implicit_prices plugin). 2020-09-12 - Released 2.3.2. 2020-09-07 - Fixed #401: Fixed initialization bugs on min() and max() functions for the SQL aggregator. - Fixed #364: Fixed important core bug in how the CostSpec were converted to Cost for cost specs with total and per-unit cost numbers, like this {106.935 # 6.98 USD}. For short positions, the cost was computed as negative. In a sense this was a naive bug, but a very important one. This occurred only for short positions. 2020-08-19 - Added an option to the printer to allow an arbitraty Python value to be output. This is useful when creating values programmatically with types not supported by the syntax. 2020-07-25 - Release 2.3.1 as dependency for beanprice. - Moved beancount.prices.find_prices to beancount.ops.find_prices in preparation of moving price fetching code to a new 'beanprice' repository. 2020-07-17 - Fixed bug #508 in beancount.plugins.unique_prices. 2020-07-06 - Updated export script to honor lowercase "ignore" instead of "IGNORE" and set default tax type to "taxable" instead of "TAXABLE" so that changes in PR 472 don't pick up the metadata as currencies. 2020-07-05 - Removed bean-report and all associated codes. Use bean-query instead (or Fava). - Removed bean-web and bean-bake (for v3). Use Fava instead. - Moved beancount.web.scrape to beancount.utils.scrape in preparation of removing beancount.web. - Removed experiments/portfolio for its dependency on beancount.reports code (this script was obsolete anyway). - Moved get_assets_holdings() function out of reports code to preserve it for use by an experimental script (in preparation for removing beancount.reports). - Moved beancount.scripts.tutorial to beancount.reports.tutorial in preparation of removing beancount.reports and beancount.web. - Moved beancount.reports.context to beancount.parser.context in preparation of removing beancount.reports and beancount.web. - Moved beancount.reports.table to beancount.utils.table in preparation of removing beancount.reports and beancount.web. 2020-07-04 - Fixed #494, invalid account used for clamping the period view (Equity:Conversions:Previous instead of Equity:Conversions:Current). 2020-06-27 - Merged PR #484, some parser improvements by @dnicolodi. - Merged PR #488 from branch beancount/pr135_ppmarciniak_fixnetworth by @pmarciniak. - Merged branch pr131_tohojo_csvtweaks (BitBucket PR 131) by @tohojo, fixed encoding bug in reading the file for the date and allow customizing the number parser. 2020-06-26 - Merged #480 - parser: Read input data from Python file objects by dnicolodi. 2020-06-10 - Released 2.3.0. See #459. 2020-06-03 - Added a 'hooks' argument to beancount.ingest.scripts_utils.ingest() replacing 'detect_duplicates_func'. 'find_duplicate_entries()' is still inserted if no other filter functions are specified (I'd like to remove this eventually; not changing behavior here for now.) - Added an example for how to run the importers explictly via a Python script calling the ingest() function. - Fixed bug with explicit invocation of importers via ingest() function when calling the 'file' sub command. - Some installations of pdfminer install 'pdf2txt.py' as 'pdf2txt'. Adjusted ingestion example scripts. 2020-06-02 - Modified recently added --no-cache option to also delete the cache if it exists. Honor the same environment variables. The motivation behind this change is that while --no-cache is useful for debugging plugins, it can be disconcerting that if you remove it on the next run it picks up the old existing cache. Deleting it on usage of --no-cache is reasonable behavior and avoids the introduction of a --clear-cache option; this should be sufficient. 2020-05-25 - Fixed #449: Added options to bean-check: --no-cache and --cache-filename to override the cache location. These are added to bean-check only and are useful during development. The 'BEANCOUNT_DISABLE_LOAD_CACHE' and 'BEANCOUNT_LOAD_CACHE_FILENAME' environment variables are still honored by the loader and work across all other programs. 2020-05-23 - Fixed #184: Added explicit test for Inventory.average(). - Fixed #404: Iex price source not working any more. - Fixed #281: Query "id" column does not identify transactions uniquely. Made the default hash_entry() function include metadata. Exclude metadata and derived fields explicitly in the context of comparison. - Fixed #182: Eliding an amount with total price raises TypeError. - Added options to filter accounts in "onecommodity" plugin: skip accounts with explicitly declared commodities, metadata of "onecommodity: FALSE" and added regular expression to specify the list of accounts to match (default: all). 2020-05-22 - Migration of code repository to Github. 2020-05-02 - Minor performance improvment by avoiding calling __iadd__ in a few places. - Fixed bug due to the removal of __version__ in apiclient: https://github.com/googleapis/google-api-python-client/commit/1d8ec6874e1c6081893de7cd7cbc86d1f6580320 - Changes all imports from 'apiclient' to 'googleapiclient', which is now the recommended way. - Fixed some warnings. - Disabled expensive assertions in Inventory class and implemented a special case for adding to an empty inventory to avoid expensive aggregation. This can occur in aggregation loops and is unnecessarily expensive when adding a single, very large inventory. - Made balance check aggregation avoid an initial empty inventory object, and changed the balance check code to call the realization utility function compute_balance(). 2020-04-11 - Fixed #425: Tentative bug fix for crash on decref over null traceback. The Python API for this function may return some null values. Not taking any chances. This should fix this particular bug. 2020-03-28 - Allow the "divert_expenses" plugin to override diversions on postings that aren't for expenses accounts. I needed this to account to a deposit to my kid's daycare, which is an asset account in his diverted ledger. 2020-03-22 - Added docstrings on some import code (while installing a new importer; my stuff is sometimes poorly documented I'm afraid). 2020-02-23 - Added new features to the CSV importer: * A new 'invert_sign' option to invert the sign of the amount. * A new field, 'CATEGORY', which goes unrendered. * Replace newlines from all narrations by semi-colons. These were driven by recent changes to the Amex CSV format. 2019-12-29 - Added a CSV field for a unique reference number. If present, this is set as a link on the transaction. 2019-12-25 - Applied patch from Daniele Nicolodi (https://bitbucket.org/blais/beancount/pull-requests/139/parser-work-around-python38-bug-causing-a) to work around typing constraints around signed size types introduced in Python 3.8. See also https://bugs.python.org/issue38913. - Fixed bug with running python3 as a subprocess during testing. Fixes an error that occurs with the tutorial file generation. 2019-11-29 - Added 'encoding' option to CSV importer. Amex encodes as 'latin1', not 'utf-8' and this turned out to be necessary. 2019-11-10 - Added an option to upload-to-sheets script to apply a minimum to the number of rows resized to. The problem was I had another sheet longer than the auto-updated data sheet, and when new rows were being inserted, rows in the referencing sheet were being incremented automatically to refer to the now displaced previous rows in the updated data sheet (even if they used to refer to rows after the last row in the data sheet). Avoid resizing in the number of rows up or down sheet should avoid most of these problems (except when at the boundary, but then the minimum number of rows could just be increased). This is quick fix that should work in the medium term. 2019-10-20 - More changes to unittest while porting to Bazel build (for C++ support). 2019-10-11 - Added __main__ block to all tests because ongoing port to Bazel build requires it. 2019-10-10 - Moved test data for b.utils.file_type for Bazel support. It otherwise assumes file_type.py is a package. 2019-08-21 - Released 2.2.3. - Fixed bug in split_expenses plugin. 2019-06-01 - Upgraded Bison to 3.4 (built from source). Silenced warnings. 2019-05-01 - Fixed #389: Patch for b.p.forecast improvements. - Added rolled back change to default value of flag used when using the 'txn' syntax. See issue #295 for a discussion. 2019-03-16 - Had a very quick shot at implementing trade tracking, added the list of matches positions from the booking methods. This doesn't work yet, I need to make changes to b.c.Inventory in order to make this work. Partial changelist. 2019-03-03 - Fixed #378: Rendering metadata with null values wasn't properly implemented. In the process, I explicit parsing of None values by allowing "NULL". - Fixed #309: Catch PermissionError to work around https://bugs.python.org/issue25481. 2019-02-27 - Implemented a quick prototype of the current trading accounts method of Peter Selinger in 'beancount.plugins.currency_accounts'. A few warts remain: * FIXME: We need to handle these important cases (they're not frivolous, this is a prototype), probably by inserting some exceptions with collaborating code in the booking (e.g. insert some metadata that disables price conversions on those postings). * FIXME(2): Ouch! Some of the residual seeps through here, where there are more than a single currency block. This needs fixing too. You can easily mitigate some of this to some extent, by excluding transactions which don't have any price conversion in them. 2019-02-26 - Fixed #376: Coerce implicit boolean to true boolean. 2019-02-10 - Fixed #374: Option --all treatment was broken in #210. Also catch and print errors. - Fixed rare failure of shutil.rmtree() in encryption cleanup test. 2019-02-03 - Fixed #372: Removed formatted string literal to keep compat with Python 3.5. 2019-01-29 - Fixed #362: Predicate matching a parent account was matching accounts with word breaks but not at the boundary of an account name. This was causing a rare (but important) failure in how Pad balances were calculated. 2019-01-28 - Released 2.2.1 for bug fixes. 2019-01-25 - Improve output of tags and links for bean-report ledger and hledger. 2019-01-22 - Fixed problem with invalid format strings with 0-width field when rendering MISSING. 2019-01-05 - Merged pull request #85 (https://bitbucket.org/blais/beancount/pull-requests/85) from hardikar, adding function grepn() to the shell. - Merged pull request #86 (https://bitbucket.org/blais/beancount/pull-requests/86) from hardikar, adding function subst() to the shell. - Merged pull request #89 (https://bitbucket.org/blais/beancount/pull-requests/89) from hardikar, adding functions date_diff(), date_add() to the shell. - Implemented best compression for zipfile in bean-bake as per Justus Pendleton's suggestion. For context, see this: https://groups.google.com/d/msgid/beancount/de75671d-532a-4deb-bd0f-fd9377e63753%40googlegroups.com?utm_medium=email&utm_source=footer 2018-12-18 - Fixed #214: Add directory of current filename to sys.path with new option "insert_pythonpath". When this option is encountered in a file, we prepend the path of the file in which it is encountered at the front of the PYTHONPATH for the rest of the duration of the problem (we could consider resetting this after parsing and processing the plugins; not doing it now). - Added detection of hitting the root if the search for the repository root failed. - Fixed #341: find_repository_root() fails on source tarball. Renamed PKGINFO to PKG-INFO to match what pip sdist produces. - Change version number from 2.2.0-devel to 2.2.0-dev which apparently is recognized by pip (thanks Jakob Schnitzer). https://bitbucket.org/blais/beancount/pull-requests/91/default/diff#comment-85707662 2018-12-12 - Suppressed almost all the Python 3.8 warnings. 2018-11-16 - Added more logging in loader (done while debugging something else). 2018-10-26 - Fixed #302: Fixed second bug with font-locking for account names (starting with digit). - Fixed bug ingest that would cause relative filenames to fail. See this thread for details: https://groups.google.com/d/msgid/beancount/709141d5-88a4-444a-acef-94bf43a9b388%40googlegroups.com?utm_medium=email&utm_source=footer - Fixed bug in ingest whereby the custom duplicates detection function wasn't being called. 2018-10-25 - Fixed unit test failure on BitBucket pipelines using a temp directory context manager. - Applied PR by Richard W to install the mixins package (install was broken). - Fixed #302: beancount.el does not fontify account names with single letter components (based on bug report by arianaut@). 2018-10-13 - Fixed #158: Missing MIME type for QFX files. 2018-10-04 - Merged new_ingest_mixins: Added new implementations of mixins for ingestion: * beancount.ingest.importers.mixins.identifier which implements identify(). * beancount.ingest.importers.mixins.filing which implements file_account() and file_name(). * beancount.ingest.importers.mixins.config which implement a simple schema with a configuration. These new mixins are intended to be more independent and more easily composable than the older ones. The beancount.ingest.importers.regexp has been removed. If you need it in interim, I made a copy under experiments/ingest/regexp.py. * Version: 2.1.3 2018-10-04 - Fixed #323: Escape strings in printer. - Minor bug fix for beancount.prices.price on null result. 2018-09-19 - Removed TRUNCATE(), after realizing ROOT() can already do the same thing. 2018-09-18 - Added TRUNCATE() function to SQL shell for Vikas Rawal. 2018-09-03 - Added a parameter to the cmptest assertions to disable real booking and allow a weak local form of booking conversion that allows interpolation. (I'm using this in another project to assert some incomplete transactions.) 2018-08-30 - Made assertions on entries off of the TestCase class, as free functions (so that they can be used in another project). 2018-08-26 - Fixed tests for pipeline running pytest (somehow the curses functions fail, probably due to better environment isolation from pytest). - Implementation of beancount.ingest.regression_pytest, a pytest replacement for beancount.ingest.regression, which was for nose. This is simpler and better and more straightforward (no conditionals; all regression files must have all accompanying expect files and all must pass). See the module docstring for new instructions on how to use this. This should not break the older tests, as I haven't changed the older support. It (beancount.ingest.regression) will be removed eventually. (This completes porting Beancount to use pytest instead of nosetests.) 2018-08-25 - Today we're switching to pytest as the official test runner. All the tests pass, and it even runs faster. It's about time! - Removed all references and workaround to nose. 2018-08-23 - Oops... removing forgotten trace in bean-web. - Fixed #325: Implemented support for including encrypted files (for Milind Kamble and using some of this changes). 2018-08-17 - Added 'issuer' column to the export script. I'm going to use this to produce a deaggregated list of exposure to individual stocks from a list of ETFs. 2018-08-25 - Upgrade requirement to bison-3.0.5 for development. 2018-08-12 - After many years of using something like it to track my investment portfolio using a Google Sheets doc, I cleaned up, rewrote and move the asset list exporting script to beancount.projects.export and deleted the old version under experiments/upload. 2018-08-05 - Fixed #322: Adjustments to StopIteration treatment from within generators were breaking bean-example generation code. - Implemented a new plugin which asserts that on accounts with 'NONE' booking method reducing postings' cost basis is within the vicinity of the average cost basis in the account. This allows you to approximate the 'AVERAGE' booking method avoid too much cost basis leakage until it's implemented properly. 2018-08-02 - Created a new 'commodity_attr' plugin to asset the presence and validity (in a set, like an enum) of metadata attributes on Commodity directives. I export my holdings joined to a table of per-Commodity attributes and I've made a mistake this week that ended up showing up as an overestimated portfolio value on my portfolio tracking Google Sheets doc. 2018-07-31 - Fixed bug in new ingest() function found as I refactored my importers. - Fixed bug with the pipeline making sure to set the CWD of the subprocess when we run an import test. - Fixed bug with previous commit containing stray 'string' reference. (My environment has autoimports by default.) - Implemented a beancount.ingest.scripts_utils.ingest() function that will act as a main program, so you can write your own importer scripts from top to bottom without having to rely on bean-identiy, bean-extract or bean-file. The purpose of this change was to allow the user to insert an optional argument 'detect_duplicates_func' in order to implement their own duplicates detection. See discussion under https://bitbucket.org/blais/beancount/pull-requests/75/. In order to install your duplicates detector, just call ingest at the end of your importer script, like this: my_importers = [ ... ] ingest(my_importers, my_duplicates_func) I could have chosen to add a "hook" option and such to Beancount, but I preferred moving the whole import framework one level closer to code. I also like that it's not just an imported module; it's really just your script, with three subcommands. You could add more custom import-level stuff in there. The ingest() function is a main program, and commands like bean-extract myimporters.py ~/Downloads --reverse -e myledger.beancount are invoked similarly by calling the Python script directly, like this: ./myimporters.py -d ~/Downloads extract --reverse -e myledger.beancount These two should be equivalent. The older way of doing things--that is, calling bean-identify, bean-extract and bean-file--are still supported regardless of whether you've inserted a call to ingest() in your importers configuration or not. This is made to work via a rather labyrinthine trampoline of functions calls (in particular, getting the arguments setup right was surprisingly tricky). I'm still debating whether this should be "the new way", or just for those wishing to build their own duplicate detection heuristic. Feedback (on the list) would be interesting. 2018-07-29 - Updated to flex-2.6.4. 2018-07-28 - Added a utility to import modules by dotted string name. 2018-07-19 * Fixed #319: Incorrect version number from 2.1.2-devel to 2.1.3-devel. 2018-07-28 - Added automated unit test for calling identify() from the importers test suite, expecting True, always. 2018-05-31 - Created a new plugin: "fill_account" to automatically insert a posting with a default account name where there's a single posting. Will use this while traveling as a lot of the expenses come from cash; this should significantly shorten such files. - Changed template for version number from {date|short} to {date(date, "%s%)} as per discussion on #304 (thanks to Jason Chu for looking up the detail for this). 2018-05-29 - Refactored the call to find duplicates in imported files so that it can be called only once with all the results from all the importers. This should eventually allow a user-provided function that can merge entries between importers. Existing behavior should be unaffected (though there may be a small delay until everything gets printed out; all the importers are invoked before any output is written). See #298. 2018-05-28 - Fixed bug with tempdir being a symlink on Mac, which was making one test fail. - Fixed unit tests which failed due to timestamps getting rendered with an offset on a Mac while on a train in Japan. - Render tags and links on Document directive. - Minor bug fix: don't fail on rendering a MISSING instnace. - In 'divert_expenses' plugin, allow limiting diversions to one posting only. - Fixed unit test for git checkouts. This should fix the Travis-CI build. * Version: 2.1.2 2018-05-12 - Fixed setup.py to catch exceptions in the absence of git as well. * Version: 2.1.1 2018-05-12 - Fixed installation with missing version string. * Version: 2.1.0 2018-05-12 - Added back lexer test from Adrian Medrano Calvo's UTF8 changes to head. 2018-05-08 - Fixed #287: In the price fetcher, store the datetime instances as naive-in-UTC in order to avoid a serialization problem with older Python versions. 2018-05-06 - Changes to support git versioning if hg versioning is not available. 2018-05-05 - Fixed #289: Add option to find beancount version. All scripts now support --version. - Amended PR64 to remove the need for __getitem__(). Clarified docstring. - Merged PR64 from Jakob Schnitzer: Inventory: use dict instead of list as a base. 2018-05-04 - Fixed #290: Validation of account name components was incorrectly swapped. 2018-05-03 - Amended PR74: Adding a new unit test, fail if tags / link appear after the first posting. Performance amendment to tags/links and clarification to the freezing of tags & links and their default values. - Merged PR74 / Fixed #99: Allow tags and links to be spread over several lines. - Amended PR14: added simple unit testing for regexp, changes in parsing UTF-8 account type to match changes in Beancount head. - Merged PR14 by Adrian Medrano Calvo: Support account names to consist of Unicode letters and numbers. 2018-05-02 - Removed a few accidental __author__ metadatas from files, normalized to __copyright__ as all other files. 2018-05-01 - Added patch for clean build for lexer. 2.6.1 results in a build warning. - Merged PR73 (Ethan Glasser-Camp) - Cleanups to lexer and more restrictive syntax for comments within a transaction's indented block. - Fixed failing test re. new grammar. - Merged PR29: Update grammar to allow comments in the middle of transactions (Ethan Glasser-Camp). - Merged PR71: Use enhanced extract method which passes existing entries (Patrick Ruckstuhl). 2018-04-29 - Added polymorphic POSSIGN(_, account) function to the SQL shell in order to support correcting for per-account sign. - Implemented polymorphic NEG() function in the SQL shell, which works on instances of Decimal, Amount, Position, Inventory. - Changed the SQL shell's operators to dispatch to Python's default operator selection instead of coercing to Decimal instances. 2018-04-28 - Added QUARTER() function to the SQL shell. (See https://groups.google.com/d/msg/beancount/fTlSnQRPayo/1yN8zd76AAAJ.) - Updated version of flex on Ubuntu (2.6.1) generates warnings. Fixed a few. But Beancount now fails on account names with a dash in them; added a test for those. (The lexer is not the issue; I suspect a bison update from 4/17.) 2018-04-18 - Fixed #157: Building a price map with zero prices should not fail, especially when done accidentally. - Fixed #264: Converted zip compression in bake from using an external zip tool to using the 'zipfile' module packaged with Python. I'm doing this to fix the CI build as well. 2018-04-17 - Fixed #265: Refined the timezone specification of the price sources. In doing this, I documented it in the beancount.prices.source module and now pass in a timezone-aware datetime instance instead of a date, and the source is now required to also require a timezone-aware datetime instance. I fixed all the tests, and added a utility to simulate running them from different timezones. - Implemented multi-file support for bean-doctor context command. Use it like this: bean-doctor context : The previous syntax is still supported; like this: bean-doctor context 2018-04-05 - Since I added a dependency on 'requests', forgot to add to checkdeps. Done. 2018-04-04 - Handle error on an invalid implementation of a price source (e.g., in this case an obsoleted price source string of 'google' imports something else and referenceing .Source fails). - Bubble up error from Yahoo price source. - Fixed #267: Implemented a price source from the IEX exchange. 2018-04-03 - Implemented Quandl importer on its REST API. Another implementation, based on the Quandl Python client API was proposed by Hugo Ideler (see PR34), but I didn't want to add a dependency on the client library (nor on Pandas), and after reading the description on Quandl API's website, I figured the library was simple enough the output could be parsed directly. I wrote a new implementation from scratch, parsing just the time-series datasets. - Merged PR68 from Martin Michlmayr, added Travis Yaml support file. - Fixed Emacs functionality to uncomment regions (change submitted by Stefan Monnier on the list). 2018-04-02 - Fixed #193: Removed the broken Google Finance price source importer. Google has removed this service in the favor of a special Search page and this has been broken for the recent little while. - Fixed #203: Updated the Yahoo price source implementation to use the undocumented v7 and v8 APIs; moved away from the "ichart" and "download" APIs. The Yahoo importer is back in function. This introduced a new dependency on the awesome 'requests' library, which really, was quite inevitable). 2018-04-01 - Merged PR65: Use parse_version to compare versions; then replicated parse_version() locally to avoid the dependency on setuptools. - Merged PR15 by Johannes Harms: Added the following features to the forecasting plugin: - Yearly repetition of recurring transactions (instead of just monthly) - Users may optionally specify an end date - Users may optionally specify the number of repetitions "I used this code to calculate scenarios for a possible real estate purchase and I contribute a cleaned-up version as an example file." - Merged PR54 & Fixed #232: Align quoted commodities correctly in ledger price statements (from Martin Michlmayr). - Merged PR57: query_execute: Allow ordering by columns which may be None (from Jakob Schnitzer). Made some adjustments to generalize on all datatypes. - Merged PR62: Also allow 0/0 in SAFEDIV() (from Jakob Schnitzer), with small adjustments to catch only the specific invalid operations on Decimal. - Merged PR64 from yagebu@ (Jakob Schnitzer), which makes the Inventory class rely on the fact that Position is immutable in order to avoid unnecessary copies. This yields an 18% speedup with no difference. - Removed unused and invalid mutating method Position.set_units(). Amazing obsolete stuff like this can still be found. 2018-03-31 - Fixed #10: "print from close_date" causes problems on account open. This was stemming from a bug in the implementation of the FROM clause execution. I changed the "entry" context functions to also accept a RowContext object, as the simple functions may be called in either entry or posting context. (The SQL shell remains the part of Beancount which is severely undertested and wading into that code once again has convinced me I need to rewrite the shell from scratch, it'll be simpler and better. The distinction between entry and posting context will go away in the next version.) - Fixed #120: Documents directive with a trailing slash fails. 2018-03-28 - Fixed failing test in amount_test probably due to some recent merges. 2018-03-27 - Completed PR #59 which adds support for globbing patterns in includes, inserting a chdir() around glob.glob() to handle expansion of relative filenames. (Credits to Martin Michlmayr for submitting the patch in the first place.) * Version: 2.0.0 2018-03-27 - Released 2.0.0. Today we're starting versioning Beancount. Many people have been waiting for this in order to manage their dependencies (e.g. Fava users). We're loosely adopting semantic versioning (https://semver.org/) where bug fixes will be posted as 0.0.x releases, and new features that don't break existing functionality (nor change it too much) will be posted as 0.x.0 releases. Note: There's nothing special about this particular point in time, other than I haven't added much features in the last six months and have mainly been fixing bugs and accepting bug reports in order to make sure things are stable. There are plenty remaining bug reports and it's not perfect. Addressing some of these will involve making changes that could break existing functionality. For this reason, I prefer to release now and make those changes as part of point releases later on. 2018-03-26 - Fixed #245: Synchronized the definition of a valid account regular expression between the lexer and the later enhanced version (in Python) which checks the configured prefixes. A valid subaccount name may start with either a capital letter or a number. 2018-03-23 - Removed deprecated methods from beancount.core (position, inventory, interpolation). 2018-03-22 - Fixed #249: Error messages involving postings should start counting at 1 instead of 0. (I agree.) - Added aliases for COMMODITY_META() vs. CURRENCY_META() and CURRENCY() and COMMODITY(). These terms are used interchangeably. In the next version of the shell, only one of them will be available. - Removed the ability to explicitly set a different price currency than a cost currency. I'm stunned my test for this is gone, I must have removed this by accident? No idea. It should not be allowed. 2018-03-19 - Fixed #218, added a SAFEDIV function to the shell language. - Fixed #239, HELP command should tolerate semicolons and whitespace. (Also, don't print shell commands in test (remove docstrings), we want the name of the tests to list, not the input.) 2018-03-13 - Removed SIMPLE booking method and all traces of it, including the 'booking_algorithm' option. Only the later (I hesitate to say "newer" because it's been around for years) FULL booking method is supported onwards. - Fixed #35: Implemented implicit group-by syntax. That is, if you omit the GROUP BY clause, it is automatically inferred to be the non-aggregate columns. This make writing the type of simple queries we do much more convenient and allows you to be a little sloppy. - Fixed bug with unit test util for docutil, which was stripping the first empty line of the docstring (and failing). 2018-03-02 - Made the 'readline' library conditional, for Windows support. See PR46. 2018-02-23 - Fixed #219: Restored sorting of the entries in the extractor. File order will have to be maintained by extracting the line no, which is used as a secondary key. 2018-02-19 - Added prototype for converting Google Docs doc from docx and rst conversions, using the docx blocks to insert indented versions in the rst file. - Removed deprecated beancount.ops.prices module. 2018-02-17 - Added an option to the CSV importer for the string separating the narration (default: "; "). - Merged a change submitted by @droogmic to add options for the dateutils parser, in order to disambiguate between day-first (DD-MM-YYYY) vs. month-first (MM-DD-YYYY) date representations. - Created a new plugin "check_closing" that automatically inserts a zero balance check after postings which are known - and flagged, using metadata - to be closing trades. - Fixed bug in "sellgains" plugin whereby a price of zero units failed an assertion. Added a test. 2018-02-12 - Added a side-effects local variable saver utility for writing importers. - Fixed #212: Clarified error message for error in covering non-aggregate targets. - The Importer.extract() method now accepts a new parameters with the prior entries (or None, if not specified) as 'existing_entries'. It's free to use that as it pleases in order to flag entries as duplicates. The entries returned by Importer.extract() will be checked for __duplicate__ metadata and automatically inserted to that set if it is present. This allows the importer to return some duplicate entries for context - which will be rendered as such in the output, e.g. commented out - without having to necessarily throw them away. (The previous signature is still supported in order to avoid breaking everyone's importers.) 2018-01-31 - Modified 'divert_expenses' plugin to store the original account name as metadata of diverted postings, so that they can be picked up by the extract_tagged script and brought back into the extract file. 2018-01-29 - Created a new 'divert_expenses' plugin to redirect Expenses postings from transactions tagged with a particular tag to another account. See this thread for details: https://docs.google.com/drawings/d/18fTrrGlmz0jFbfcGGHTffbdRwbmST8r9_3O26Dd1Xww/edit?usp=sharing - Added account_types.is_account_type() function, which should be used above account_types.get_account_type(). 2018-01-23 - Removed the insertion of 'balance' metadata from the importer. We use it for inserting Balance entries, not attach as metadata. - Disabled Balance insertion CSV test; 03d9a5c3b2784a6f6a05c12eaeb7776f0478a96d broke the insertion of Balance entries without inserting 'balance' metadata. We'll probably have to revert the sorting changes and that. For now, fix the test. - Fixed #210: Changed the bean-price option --undeclared to accept a string value, the name of the Python source module to use for fetching undeclared commodities. - Added an error on using explicit merge cost syntax (this one: {..., *}). It's not supported yet, like AVERAGE cost booking (I probably should not have enabled the syntax until it was done. I was hopeful and moving fast at the time). 2018-01-16 - CSV importer: Added an option not to add balance metadata to output. - CSV importer: Also, fixed a bug I introduced in the recent PR merge with rendering the name of the importer. - SQL importer: Made "location" field support transactions with metadata with missing filename and line no. Some plugins might be doing that. 2018-01-08 - Fixed #209: Don't require 'ido. User will have to make sure that beancount-use-ido is false. 2018-01-07 - Merged https://bitbucket.org/blais/beancount/pull-requests/28/allow-one-character-top-level-account, allowing one character top-level account names in the syntax. This doesn't mess with the flag token generateion because the account name is required to have at least two components. 2017-12-09 - Fixed rendering of price directives to use the display context precision. 2017-11-24 - Silenced annoying warning about file_cache version incompatibility in upload-to-sheets. 2017-10-28 - Fixed bug with identification based on the first line of a file. - Fixed bug with 'context' on a directive that's not a transaction. 2017-09-30 - Fixed a bug in reduction booking that would fail when the units are omitted. 2017-09-17 - Fixed bug in Balance check, detect when an account does not exist and handle it properly. - Added bean-difference script, used to reconcile transactions in multiple files after a long trip. 2017-09-06 - Added ABS() implementation to SQL shell for Position and Inventory types. See https://groups.google.com/d/msg/beancount/0i4w98LmS6E/JJGOKSB8CQAJ 2017-08-08 - Accept account name components as numbers only. The grammar safely supports this because of the leading semi-colon, all the tests pass, and everything runs on my own very large input files. - Fixed a failing test with MIME mis-categorization which occurred under Ubuntu. 2017-07-25 - Fixed #181: Support underscores in Python module names for price sources (thanks to Stephan Mueller for reporting). 2017-07-23 - Fixed #179: Scientific notation should never appear in rendered output. 2017-07-02 - Fixed minor bug with rendering postings with missing numbers. 2017-07-06 - More changes on #173: use a stringify two-level macro in order to place the parser source hash as a string in the source file. Hopefully this should also work under MSDEV. 2017-07-05 - Fixed #173: Added a potential fix for building under MSDEV. Thanks to Alan Ray for pointing out a method to build with MSDEV under Windows. 2017-06-25 - Fixed a minor bug in the tolerance inference code that wouldn't accept a posting with a missing number only, e.g. Assets:Checking USD @ 1.32 CAD. - Added a "--no-colon" option to both bean-web and bean-bake in order to support Windows filesystems which do not allow colons in filenames. 2017-06-22 - Added --numberify option to bean-query, which for CSV output will remove the currencies of any Amount column. This is great for then uploading to a spreadsheet. 2017-05-28 - Merged patch from Felix Eckhofer that adds an option to bean-format to make it use a fixed width. I've been wanting this myself for a while actually, I think a fixed value makes more sense than what I'd done previously and should eventually be made the default. 2017-05-24 - Implemented beancount.plugins.auto and beancount.plugins.pedantic. I've been wanting to do this for a while. When writing quick-and-dirty little tests, inserting 'plugin "beancount.plugins.auto"' is quick and does the job, and if you want absolutely all the verifications and to make Beancount as pedantic as possible, you should be able to adopt 'plugin "beancount.plugins.pedantic"'. In order to do that I had to make the loader support plugin definitions whose __plugins__ attribute is a function reference, not just a string. It supports both now. - A number like this now fails to parse: 1,245,449,72.00 USD. I added a test in the lexer to tighten the parser. This has been brought up before and will potentially avoid input errors. 2017-05-20 - Clarified error message for price format specified on the command-line. 2017-05-08 - Made the open/close map return by the getters not a defaultdict by default, so that accidentally looking up elements won't mutate it. 2017-04-30 - Moved src/python/beancount/... to beancount/... The purpose is to structure this project in the most common way people expect it (most Python OSS projects have this structure, with the package name directory at the root). I've had this other structure for a while mainly for historical reasons. I wanted to make this clean before baking a final 2.0 release and switching to numbered releases. - Moved src/elisp/beancount.el to editors/emacs/beancount.el and made necessary adjustments. - Merged incomplete changes from booking branch to "default." The work on supporting self-reduction (see http://furius.ca/benacount/doc/self-reductions) won't be carried during 2.0, it will be subject of a subsequent release. - This concludes the merging of all the branches, which I've done in order to move the source to a more regular project directory (under /beancount). 2017-04-29 - Fixed #27: IN membership operator did not have a proper precedence definition. 2017-03-19 - Fixed minor bug with CSV importer that would occur when the file has no valid columns. 2017-01-26 - (internal) Moved the implementation of booking methods to its own beancount.parser.booking_methods module. - (internal) Split up the booking methods to functions for exhaustive testing and allowing others to implement new methods more easily if desired. 2017-01-22 - Added new price source fetcher from OANDA's API. It supports a large number of currency pairs with tickers like this "XXX_YYY", such as "EUR_USD". Use it like the other price sources, e.g., like this: bean-price -e 'USD:oanda/EUR_USD' - Began working on the conversion of Google Docs to reStructuredText for inclusion in Dominik Aumayr's docs. 2017-01-21 - Fixed #155: Use entry_points instead of scripts in setup.py. - Added an extra check for Balance directives, verifying that asserted currencies are accepted currencies for that account (if the account's corresponding currencies have been declared explicitly). In other words, this will now fail instead of being silently ignored: 2017-01-21 open Assets:Checking CAD 2017-01-21 balance Assets:Checking 0.00 USD - Fixed bug in sheets_upload.py / upload-to-sheets whereby all the fields of a sheet being updated just in order to resize it was causing the sheet to be reindexed at the front of the list of sheets. - Merged 'arithmetic_ops' branch with no effect, in order to move all files and release. This will have to be revisited later. 2017-01-19 - Fixed bug with generation of column names in sheets uploader script. 2017-01-16 - Removed tools/gauth (used to be beancount.docs.gauth). All users of this are slightly different, and they're utility scripts. Unfactoring. - Removed beancount.docs from Beancount library. These are internal tools used to manage documentation, should not be included in the installed codebase. - Some cleanup of internal tools. - Added some basic unit tetsts for beancount.tools.sheets_upload. - Moved bin/upload-to-sheets to src/python/beancount/tools/sheets_upload.py and replaced it with an associated stub for it. - Moved bin/treeify to src/python/beancount/tools/treeify.py. The script is still usable in a standalone form (no new dependencies have been added). To have all the source code in one place is more convenient for maintenance. - Created a beancount.tools package to contain all standalone tools which don't depend on Beancount modules or packages. - Removed branch "returns2" and moved associated projects script beancount.projects.returns to experiments/returns. This code isn't robust enough to work for everyone, and so therefore I'm moving it back to experiments until I have time to produce something that will work very robustly, and on all my own accounts as well. 2017-01-15 (from branch "conversions") - Created and test beancount.core.convert Posting and Position conversion functions: get_units(), get_cost(), get_weight(), get_value() (for market value). - Implemented Inventory.reduce() and associated test, which works in conjunction with those. - Implemented more straightforward versions of convert_position() and convert_amount() in module beancount.core.convert, to convert positions and amounts to other currencies. - Deprecated Inventory.units(); use Inventory.reduce(convert.get_units) - Deprecated Inventory.cost(); use Inventory.reduce(convert.get_cost) - Deprecated Inventory.get_units() to Inventory.get_currency_units() in order to avoid future name confusion. - Deprecated Position.at_cost(); use convert.get_cost(position) instead. - Deprecated Position.get_cost(); use convert.get_cost(position) instead. - Deprecated b.c.interpolation.compute_cost_basis(); use Inventory.(pos for pos in positions if pos.cost is not None).reduce(convert.get_cost) instead. - Deprecated get_posting_weight(); use convert.get_weight() instead. - Removed b.ops.holdings.get_pholding_market_value(), which was unused. If you need a market-value function, use convert.get_value() instead. - Converted all the old code to these new converted functions. - Moved module beancount.ops.prices to beancount.core.prices, and added a deprecation warning. - Deprecated shell function PRICE(). Renamed it to GETPRICE() in order to avoid confusion. - Removed beancount.core.prices.convert_amount(). Use beancount.core.convert.convert_amount() instead. Note that a failure doesn't return None anymore, but an unmodified amount. Also notice the reordering of parameters to be consistent with the other functions in that module. - Removed beancount.core.prices.get_position_market_value(); use convert.get_value() instead. - Removed beancount.core.prices.get_inventory_market_value(); use Inventory.reduce(convert.get_value, price_map) instead. - Removed private functions beancount.query.query_env.convert_amount(), beancount.query.query_env.convert_position(), and beancount.query.query_env.value_position(), beancount.query.query_env.value_inventory(). These used to issue warnings if the conversion could not be carried out; now they simply keeps the units of the position instead. 2017-01-15 - Fixed all tests for Python-3.6. Beancount has not officially been ported to Python-3.6. 2017-01-14 - Fixed bug with generation of example that would produce an incorrect initial balance for banking accounts. The example test should have been broken. (No idea why this wasn't deteted earlier, probably by chance.) 2017-01-08 - Completed "Intro to Double-Entry Accounting" doc. http://furius.ca/beancount/doc/intro - Added a --reverse/--descending flag to bean-extract to print the extracted entries in reverse order. 2016-12-17 - Added a --title option to upload-to-sheets in order to conveniently set the document's title. - Removed obsolete variable LEGACY_DEFAULT_TOLERANCES which had been forgotten. 2016-12-15 - Made beancount.ingest.extract print out a stack trace when an unexpected exception is raised. - Fixed bugs in beancount.query.query_render where expand wasn't being passed through the rendering functions. - Deprecated bin/upload-csv-to-google-sheet; I moved the last version to beancount/experiments/gapis in case you've come to be dependent on its particularities. It doesn't need any of the Beancount libraries, so you can just copy it somewhere and keep using it if so desired. - Replaced "bin/upload-csv-to-google-sheet" to "bin/upload-to-sheets". The newer script is a complete rewrite which does not require the antiquated gdata library, and which uses the latest and greatest version of the Google Sheets API (v4). You don't need gdata anymore. Moreover, this one runs in Python3 instead of depending on Python2. The script has slightly improved invocation semantics, and I'll using it regularly to share CSV files with others, most often CSV files produced by bean-query. 2016-12-11 - Changed b.p.split_expenses outputs to be more useful. 2016-12-10 - Made Amount, Posting, Cost and CostSpec all immutable classes that inherit from typing.NamedTuple. - Removed mutable method Posting.add(). 2016-12-05 - Merged changes made for calculating wash sales a long time ago. I had implemnted a temporary kludge to carry dates on all augmenting postings, left it in a branch so I could use it there for taxes every year. Now that the booking branch is merged, this is supported by default, so I could undo that kludge. Some changes remained: * The Inventory.add_amount() method used to return the modified position. Now it returns the position before being modified, which is more useful. * The list-wash-sales.py script has been updated. It now works on 'default'. * The 'carry_date_and_book_cost' branch has been merged away. 2016-12-04 - Made .tags and .links attributes of b.data.Transaction and b.data.Document directives ensured to always be a set when they're empty. If there are no tags (or links), those attributes are set to b.data.EMPTY_SET, whose value is a frozenset(). This should make the processing of those attributes a bit simpler. Note that if you create transactions from plugins and their tags and/or links attributes are set None, they will now fail. The right thing to do is to set their value to an empty set, or reuse the immutable beancount.core.data.EMPTY_SET object. If you need to circumvent this temporarily, set the following option in your file, which will allow None again: option "allow_deprecated_none_for_tags_and_links" "TRUE" See {1989ba9e0e79} for details. - Updated all copyright and license notices on all files. - Added a script to automatically verify that all source files include a copyright notice. - Removed ancestral Beancount documentation from eons ago. These notes and conversationsa are now wildly irrelevant to the current state of Beancount, and all the documentation from Google Docs is vastly more recent and obsoletes all these old snippets and notes. - Moved all development tools under tools/, from etc/ and experiments/code-health/. 2016-12-03 - Fixed fava bug #429, avoid empty conversion entries if the cost of the conversion balance is zero. See: https://github.com/beancount/fava/issues/429. 2016-11-20 - Fixed bugs in bean-format; in order to do this properly I made bean-format reuse the globally defined regular expressions for currency and account in the rest of Beancount's code. This makes bean-format depend on the Beancount installation. Bah, it's reasonable. This issue is related to issue #146. - Added ANY_META() function to the SQL shell's environment; ANY_META() allows you to query a posting's metadata, falling back on the parent entry's metadata. This idea was suggested by Stefano Zacchiroli in https://groups.google.com/d/msg/beancount/jh-n4rvbQAI/Hz5Mmd60BAAJ. I also added the first unit tests for environment functions--I'll have to write all of those at some point. - Made unsupported cost specifications in the SIMPLE booking algorithm issue an appropriate error. For example, instead of just ignoring the cost spec in this: 2014-10-15 * "buy widgets" Assets:Inventory 10 WIDGET {} ;; Not supported. Assets:Cash -80 GBP It will issue an error. See https://groups.google.com/d/msg/beancount/9NBcT-SXZMQ/sy7Z3dIdBQAJ for context. 2016-11-12 - Made the Emacs prefix key configurable via a new beancount-mode-map-prefix variable. See discussion here: https://groups.google.com/d/msg/beancount/_7TscZ9phNE/K8PkcRMpBAAJ 2016-11-06 - Fixed #139: Full booking parsing had been more liberal than the old SIMPLE booking algorithm can interpret. In the old code, explicitly disallow some interpolation cases which cannot be handled. - Fixed #129: Ignore custom directives for Ledger and HLedger conversion. 2016-11-05 - Fixed #142: bean-check should return non-zero exit code upon failure. 2016-10-30 - Merge the current version of the 'booking' branch and made the "FULL" booking algorithm the default one. * The "experiment_booking_algorithm" option has been renamed to "booking_algorithm". Its default value is now "FULL"; everyone will be using the new booking code. If for some reason you have a transitional need to revert to the older booking method use this option. option "booking_algorithm" "SIMPLE" * The new booking algorithm is _MUCH_ more powerful than the one old. I'll describe it soon in a new document, but briefly: - It supports many more interpolation capabilities; try omitting amounts or currencies which you think it ought be able to figure out automatically and you might find it does. - All positions carry a date. This means that if you have transactions that would merge because they weren't specified with a date, like this you may face a problem: 2016-01-15 * Assets:Investments 70 UUG {10.00 USD} ... 2016-02-16 * Assets:Investments 25 UUG {10.00 USD} ... 2016-03-30 * Assets:Investments -80 UUG {10.00 USD} ... Those used to resolve automatically because both augmenting lots would merge into a single one without a date. Now that all positions have an acquisition date, the -80 reducing posting is ambiguous, because it matches against these: Assets:Investments 70 UUG {10.00 USD, 2016-01-15} Assets:Investments 25 UUG {10.00 USD, 2016-02-16} Either you'll have to disambiguate, or let Beancount do so. One way to do this is to add more specific matching data, and split the posting. An even easier way to do this is to make the default booking method of this account to "FIFO" or "LIFO", which will automatically select the oldest (newest) postings without issuing a warning. If you find yourself in a bind, please email the mailing-list with a specific and as small as possible data input that reproduces the problem. - Removed parsing of the pipe (|) symbol as a separator for strings on transaction description lines. This was vestigial from the days long long ago when Beancount was attempting to be compatible with Ledger syntax; this is wholly unnecessary how. You will have to convert transactions like this: 2015-08-25 * "Tunnel" | "Coffees" Expenses:Food:Coffee 15 CAD Assets:Cash Into transactions like this: 2015-08-25 * "Tunnel" "Coffees" Expenses:Food:Coffee 15 CAD Assets:Cash This can be done using a sed script, or as a macro in your favorite text editor. If you need a temporary respite, you can use the following option, which will replace all the error messages by a single warning: option "allow_pipe_separator" "TRUE" Support for this option will be removed eventually. - Moved the {{...}}} total cost syntax from a legacy support with only "/" separator and a date, no label, to full support. Previously, only these syntaxes were supported for total cost: Assets:Account 100 MSFT {{2000.00 USD}} Assets:Account 100 MSFT {{2000.00 USD / 2016-10-30}} Changes: * The "/" separator support has been removed, you must now use a comma, like for the {...} full booking syntax. * The same syntax as for the regular {...} compound cost is supported; however, specifying a compound amount will raise an error, and the per-unit cost is interpreted as a total cost. For example, this: Assets:Account 100 MSFT {{2000.00 USD}} is interpreted as: Assets:Account 100 MSFT {0 # 2000.00 USD} which is equivalent to: Assets:Account 100 MSFT {20.00 USD} And the following will raise an error, it is not legal syntax: Assets:Account 100 MSFT {{0 # 2000.00 USD}} Finally, specifying a date and a label can be done, just like for {...}: Assets:Account 100 MSFT {{2000.00 USD, 2011-10-30, "ref32"}} The intention here is to deprecate the older limited syntax while keeping support for total cost input using {{...}}. Based on previous questions, I suspect that many users are still using {{...}} instead of {{# ...}}, so I won't remove it just yet. - Removed very old compatibility code in beancount.core.amount that lived since I cleaned up the definition of the D() constructor and moved it to beancount.core.number. - Removed built-in aliases for auto_accounts and implicit_prices modules, from beancount.ops to beancount.plugins. - Removed deprecated option "default_tolerance" (which had been renamed to "inferred_tolerance_default"). - Removed deprecated option "tolerance". - Remove deprecated use of 'plugin' as an option. The directive ought to be the only way. - Removed deprecated option "experiment_explicit_tolerances". - Renamed "experiment_booking_algorithm" to "booking_algorithm", and changed the default value to "FULL". The old "SIMPLE" algorithm will be deprecated eventually. - Removed old code that would warn if the user set account_rounding to a full account name; it just needs be a subaccount. - Commit to Python 3.5. 3.5.0 released more than a year ago. You need to update or stick with the 2.0b13 release. - Remove old code to dispatch between an external Enum class and the built-in one that's included in 3.5 (commit to 3.5). - Added type annotations to beancount.core.data. This will grow more over time. (I also need to add rules to use those.) 2016-10-23 - Modified parsing of transaction strings and links and tags. I did this to factor out the parsing of tags and links in order to add it to other transactions and minimize grammar code. There is user-visible impact to this change: all of a transaction's strings must appear before tags or links. You cannot place tags or links before or in-between strings anymore. This is a reasonable restriction and I doubt anybody did anyway. - Added support for tags and links to the Document directive. This works just as for Transactions, e.g. 2016-10-23 document Assets:Checking "/statements/oct.pdf" #banking ^9fed846348c2 The current tags on the tag stack are also included, like for transactions. 2016-10-10 - Removed "experiment_explicit_tolerances" flag; made this supported by default permanently. 2016-10-08 - Documented a subtle problem reported about balances against inventory of mixed at-cost and no-cost, added more context to the error and a unit test for balance checks with mixed at-cost and no-cost positions. - Fixed a bug in the full booking matching code whereby an existing position not held at cost would trip up the matching code. (Credits to Son Nguyen for finding and reporting.) - During ingest, when multiple extractors match a particular file, select the first matching account instead of raising an error. - Fixed accidentally colliding key-binding for invoking bean-price from Emacs, to C-c p. 2016-09-25 - Fixed typo in grammar that was parsing a transaction flag as '#' into '*'. - Added "cost_date" and "cost_label" columns in the SQL shell. 2016-09-04 - Implemented new "run" shell command that allows you to run a named query. For example, if an input file has a Query directive like this: 2016-09-01 query "home" "select account, sum(position) where account ~ 'Home' group by 1" And you input this command: run home The entries will be closed at tue query date (2016-09-01) and the query automatically run. If you want to always include all the transactions, set a date far into the future. This works from the command-line and from the shell, equally. You can also run all the queries in an input file, like this: run * To view the list of available queries, just type the "run" command without an argument. 2016-08-21 - Fixed a SQL shell bug with extracting UNITS() from a Position column. - Finally properly fixed the timezone of the Google Finance price fetcher. (It was done incorrectly and I found my unit tests failing whenever I would work on Beancount from another timezone.) 2016-08-17 - Added beancount.core.amount.abs() function for absolute values of Amount. - Added rendering of links to beancount.plugins.split_expenses plugin, and made it quiet by default. Added --output-stdout option to make it do that again. - Created a new experimental script that attempts to match postings between two ledgers. A pair of (filename, account-regexp) is provided for each, and then the amounts and links are compared and paired up, and remaining unmatched postings are printed out. This is very useful to reconcile trip expenses between a personal ledger and the ledger for a trip or external project where some of the expenses are paid from the personal ledger as contributions to the trip or project. - Made the upload-csv-to-google-sheet script use the basename (without extension) of the given filenames as sheet names instead of "Sheet X" numbers. (Note: This entire thing be rewritten using the new Google Sheets API which won't require gdata and which will be more stable.) - Added a quick implementation of b.q.query_render.render_csv() and refactored b.q.query_render.render_text() to use common code. Both functions now have a new 'expand' option which will create multiple rows for cells which return a list of strings. The new default behavior is to concatenate the strings with commas. The shell (bean-query) has a new 'expand' shell variable to match. Also, the -f/--format option of it is now supported, and in interactive mode it sets the default value for 'expand'. - Since output to CSV is now implemented, I enabled the --output-csv option of b.p.split_expenses as well. 2016-08-14 - Fixed bug in beancount.utils.file_type for Python3, whereby magic now returns bytes, not a string 2016-08-04 - Made account name components allowed to have two characters (beyond the first component). For instance, Assets:Investments:F is now a valid account name. 2016-07-11 - Added parsing a transaction date column to metadata in the CSV importer. 2016-07-01 - bean-format now accepts input from stdin. 2016-06-30 - Make it possible for a BALANCES statement to have a WHERE clause. BALANCES is now equivalent to SELECT account, sum(position) FROM ... WHERE ... GROUP BY account Inching one step closer to removing that WHERE clause eventually. 2016-06-18 - Added support for columns AMOUNT_DEBIT and AMOUNT_CREDIT in the csv importer. - Merged ongoing progress from 'booking' branch; if you don't use any of the in-progress new booking methods, this should not have any effect. However, the 'booking_method has changed names and you may have to adjust its values if you've been tinkering with it: * 'booking_method' used to take on value "SIMPLE" or "FULL" and dispatch betwee the current (old) booking implementation and the new one which will support many other booking methods. This option is now named 'experiment_booking_algorithm' (a new option) and takes on the same values. The default value is "SIMPLE", which keeps the behavior unchanged. Use "FULL" if you want to test out the new booking algorithm: the new algorithms should work except for the AVERAGE cost method. WARNING: This is still under flux and considered unreleased. " 'booking_method': This is a different option now, and takes the value of any one of the booking methods visible here: https://bitbucket.org/blais/beancount/src/7b9e00e6e19136f680befc1d4322f47cdc294cf6/src/python/beancount/core/data.py?at=booking&fileviewer=file-view-default#data.py-64 2016-06-13 - Ported beancount.docs.download_docs to the v3 drive API and the use of a service account instead of a full access to a user account. Service accounts are easier to configure and work with. - Ported beancount.docs.upload_options to the v3 drive API. 2016-06-05 - Added a new column to the posting context: "other_accounts" is a set of account strings for the other postings of the transaction. - Added a new JOINSTR() generic function to the shell, to reduce the repeated column above to a single comma-separated string. 2016-05-28 - Made various changes to the shell in order to make it possible to convert to the market value: * CONVERT() has been converted to convert to the market value, when applied to a position held at cost. However, it still requires a target currency and it may fail if you apply it to a position for which the cost currency differs from that of the target currency, i.e., it will not automatically chase down the transitive conversions toward the desired target currency. * A new function VALUE() has been added, which is similar to CONVERT() but which when applied to a position held at cost converts to the cost currency at the current price. It's similar to CONVERT() but you don't have to provide the target currency. And similarly to CONVERT(), you may provide a particular date, and if at that date there is no available conversion rate in the prices database, the conversion will fail analogously and spit out an error message. * PRICE() has been renamed to GETPRICE() to avoid some confusion. PRICE() does not convert, it merely gets the price from the price database. GETPRICE() is thus a more appropriate name. 2016-05-22 - Created the beancount.plugins.mark_unverified plugin as a response to this thread: https://groups.google.com/d/msg/beancount/hQoOSJE7qNo/-E__v-XhEgAJ This plugin makes it possible to compute the balances as of the last Balance directive of each account by adding metadata to postings which appear after that account's last balance. You'd filter out unverified postings with an SQL query on meta-data like this: SELECT account, sum(position) WHERE NOT meta('unverified') GROUP BY 1 ORDER BY 1 Note that the resulting balanaces almost certainly don't sum up to zero, because of differing dates of the Balance directives. Also, note that accounts without a single Balance directive will not have their postings marked at all. 2016-05-08 - In the output of bean-price, sort prices exclusively by currency, regardless of date. This is because on weekends you'll tend to get slightly varying dates, and manual/occasional price updates are usually put in "chunks" in a file section. It's easier to visually find the price by scanning by currency order. 2016-05-05 - Added a shell function to filter a set of strings (for tags or links) with a regular expression and return the first one. This way you can extract some of the tags or links with a pattern as columns. 2016-05-01 - Fixed issue #124, a leak in the grammar parser which would leak the entire set of entries on every instance of parse. - Fixed some warnings about unclosed files which occur only on default python branch. 2016-04-29 - Don't just log an extraction error to logging.error() in the regression tests; let the exception through. This makes building and debugging importers much easier. - Created an experimental script to extract date/location information required for a naturalization application. 2016-04-24 - Rewrote the section on "Core Data Structures" of the Design Doc, to reflect the newer internals of Beancount, those which have been merged from the ongoing 'booking' branch. See http://furius.ca/beancount/doc/design-doc. 2016-04-23 - Added beancount.ingest.importers.fileonly, a simplistic importer which can be instantiated to just file away some downloads without extracting anything from them. 2016-04-18 - In bean-doctor context, all the accounts should appear in the post-transaction balances, even if their balance is empty. This had been fixed for the pre-transaction balances but I stumbled upon a case of post-transaction balances not showing an empty account. 2016-04-17 - Added support for the BEANCOUNT_LOAD_CACHE_FILENAME environment variable, which can be set in order to overridde the location of the pickle cache. This variable may contain the string "{filename}", and if it is present the basename of the beancount file will be spliced in. This can be useful if, for example, you are reading your Beancount input file from a read-only filesystem and still want to use the cache by specifying an alternative filename. - Fixed a bug when reading from a read-only filesystem, the loader cache would fail to be removed and raise an exception. - Fixed an important bug in beancount.ingest whereby instances of b.i.cache.FileMemo weren't being shared as widely as they could have been. FileMemo instances now have to be created via b.i.cache.get_file() and FileMemo is now a hidden class _FileMemo. See this discussion for more details: https://groups.google.com/d/msg/beancount/b-ZqYqag7AU/N-AaGSZOAwAJ 2016-04-10 - Create a new 'beancount.plugins.fix_payees' plugin that allows the user to provide a set of rules to rewrite and clean up the names of the payees from the Beancount file. While it would be ideal to do this at import time, in practice there are a lot of downloaded files' "memo" fields which get set as the payee automatically. These "memos" aren't very clean and readable strings: they sometimes contain numbers, partial city names, and other junk. In order to produce clean reports where one might want to aggregate by payee, it's useful to rewrite the payees to make them nice and clean. You can use this plugin to do that. - Made the Custom directive accept ACCOUNT tokens. You can now do something like this: 2014-06-20 custom "budget" Assets:Account2 "balance < 200.00 USD" Prior to this, the account name had to be represented as a string. (Users of the burgeoning Custom directive not like this very much.) Note that account names and regular strings are both output from the parser as 'str' objects, so in order to make this possible, I had to change the representation of the values to a pair of (value, dtype). Account names have a 'dtype' of beancount.core.account.TYPE. (Eventually I will probably want to make account strings represented with their own custom datatypes, but that will require a fair amount of work on the codebase; this substitute is fine for now, as there are no other places where such a disambiguation is necessary.) 2016-04-04 - Added a test to make sure that building a source dist includes the header files and C implementation files. - Sort Document directives after Transactions when they occur on the same day. The reason for this is that statements typically include transactions on that final day, and we'd like for them to occur at the right place in the journals. See originating thread: https://groups.google.com/d/msg/beancount/3ItOYS9DJx4/VyTJB3ssBQAJ 2016-04-03 - Fixed a bug in "bean-doctor context" which wouldn't report the full set of accounts when an account's before-balance would be empty. - Fixed a bug in "bean-doctor linked" whereby the balances wouldn't get printed out if there was no link. Print the balances before and after; in fact, maybe this command should be renamed to "balances" and have an option to support links. 2016-03-31 - Added TODAY() function to extract today's date. Not sure how useful it'll be. 2016-03-27 - Added a PRICE() function to extract entries from the price database. You can now query for "SELECT PRICE('USD', 'CAD', date)", for example. - Added a boolean renderer for the SQL shell. - Added support for arithmetic operations to the bean-query SQL shell. 2016-03-26 - Added an optional date argument to convert, so you can provide the requested date in a SQL query, as in "CONVERT(sum(position), "CAD", 2015-06-01)". - Improved the rendering of the context, removed the weird comments it used to render with, render an account name if the inventory is empty. - Ignore .DS_Store files from the file finder in ingest. 2016-03-22 - Support multiple accounts in same ofx file. See https://bitbucket.org/blais/beancount/pull-requests/10/. I applied this one-liner: issue a balance for each statement, use the maximum date for the file_date() method, and added unit tests for this. 2016-03-21 - Fixed bug in beancount.ingest.file whereby two files being filed to the same destination filename would clobber each other; with this fix, instead, an error will be generated. - Create an example of an importer just filing a PDF file using the external tool PDFminer2. If the tool isn't installed, nothing should "break" (but the importer won't really do its job). The tests should be skipped automatically, and the importing will just ignore the PDF file to be filed. 2016-03-20 - Implemented beancount.ingest.importers.regexp and beancount.ingest.importers.config mixin classes to help support functionality which used to be available in LedgerHub. - Removed the temporary beancount.ingest.importers.compat backwards-compatibility helper for LedgerHub. Not needed. - Completed the documentation for the new import framework "beancount.ingest". The document is here: http://furius.ca/beancount/doc/ingest 2016-03-19 - Centralized the <3.4 and >=3.4 support for Enum class into beancount.utils.misc_utils.Enum. - Added an option to the OFX importer to let the instantiator decides whether a balance directive should not be inserted (NONE), should be inserted at the declared date of a tag (DECLARED), or right after the last entry imported from the file (LAST). 2016-03-18 - Merged 'web_months' branch modified from changes originally submitted by Dave Stephens, which implements a monthly view. You enter the monthly view from the annual view. 2016-03-14 - Added META() and ENTRY_META() functions to the SQL shell to answer a question on the mailing-list (from richieuk63 at gmail dot com). - Fixed a bug with the OFX importer whereby the Balance directive should be generated to a day following the date provided in the LEDGERBAL directive. - Added forgotten dependency on "beautifulsoup4" in setup.py. 2016-03-13 - Merged more changes from the 'ingest' branch which furthers the work of integrating the LedgerHub code into Beancount (almost done now). * Added mimetype() and head() methods to the file cache object passed to the extractors, as a convenience instead of having clients call the cached convert() method. * Added paranoid exception handlers to all the methods called on the importer classes. * Added tests for the examples provided in "examples/ingest". * Refactor some of the code of beancount.ingest.file. * Fixed bugs with ensuring the filenames are absolute names. * Cleaned up the output of the commands. * Made the header and section strings configurable from the import configuration. * Cleaned up the OFX importer from LedgerHub, added unit tests for it, and added it to the beancount.ingest.importers directory. In addition, added it to the example configuration for ingest under beancount/examples/ingest. * Added a dependency on the beautifulsoup4 package (bs4) in order to parse XML files. - Added a beancount.core.amount.add() function to add two amounts. - Made the directive printer be more lenient to support incomplete postings. - Moved the "src/clojure" and "src/golang" directories to the experiments/ directory; they really are experiments. Should have done this a long time ago. 2016-03-12 - Converted the amount.amount_*() functions to amount.*(). It's just simpler and nicer. - Made the beancount.report.balance_reports text reports rendered using the precision from the DisplayContext, and realization.dump_balances() require a formatter argument. More probaby needs to be done on issue #108 to close it, but that's a good start. - Made the beancount.plugins.auto_accounts plugin insert entries that will sort stably. - Sync up the print report code with the print command code, fixing a bug with precision not being rendered correctly by the print report. - Fixed #104: bean-format would fail on numbers with commas in them. - Fixed #106: The "render_commas" option wasn't being honored. I have no idea how things came to be this way, but I fixed it. When rendering numbers, inserting commas for thousands (e.g., "1,000" vs. "1000") is now controlled by the "render_commas" option, and the default is not to render them (FALSE). Note: This should affect the web interface, the reports, and the PRINT command, but so far NOT the output of the SQL query reports. The reason for this is historical: the query rendering code does not make use of the DisplayContext; it needs to be converted to do that. See https://bitbucket.org/blais/beancount/issues/105/ https://bitbucket.org/blais/beancount/issues/106/ https://bitbucket.org/blais/beancount/issues/107/ and https://groups.google.com/d/msg/beancount/frfN1zc6TEc/d5OjuDnREgAJ for a discussion. 2016-03-06 - Fixed issue #83: Renamed 'default_tolerance' option to 'inferred_tolerance_default'. - Added exception handlers for all importer callbacks. Badly written importers should never completely throw off the tools. 2016-03-05 - Improved handling of corrupted or out-of-date pickle cache file. - Fixed a test that would only fail in a timzone != EST. - Made bean-file use the configuration filename's directory as the default output dir instead of requiring an option. The behavior of providing a reasonable default makes most sense. 2016-02-28 - Moved beancount.scripts.report to beancount.report.report in order to be consistent with the way other libraries are invoked. I had to move beancount.report.report out of the way (to beancount.report.base) to make space for it. - Moved beancount.scripts.query to beancount.query.shell in order to be consistent with the way other libraries are invoked. - Merged fix by yagebu for bug found in b.ops.summarize() that would not yield a list of ordered transactions. - Introduced a new "Custom" directive type. You can enter directives like this: 2016-02-28 custom "budget" .... where "..." can be a mixed list of * string * date * boolean * number * amount (number + currency) The first element is a string, which is intended to be used as the runtime type of the directive. This can be used to prototype new features in plugins or otherwise. Right now, no consistency is enforced on the datatypes declared for directives with the same type name. Also, there is currently no way for a plugin to declare what a valid set of tokens may be (but I may do this later on to simplify error reporting for plugin implementors). 2016-02-27 - Fixed rendering bug in bean-web whereby empty cells would receive a 0.00 amount. - Improved rendering of the account name in the name of the compat importer. 2016-02-24 - Changed the bean-doctor "linked" command to fetch the set of links from the entire set of entries. - Added ABS() function to the shell, in order to dig around for transactions with a particular amount. 2016-02-22 - Fixed issue #98: There was a bug when fetching historical data from a Yahoo price source was failing, it wasn't handled correctly. - Merged LedgerHub back into beancount under a new "beancount.ingest" package. The new protocol for importers can be read at: /home/blais/p/repos/beancount1/src/python/beancount/ingest/importer.py There are three new scripts: bean-identify bean-extract bean-file These correspond one-to-one to their ledgerhub variants. There is a bit more work to be done to complete this - see the top of the TODO file in the "ingest" branch if you're interested - and I'll write documentation on this once all the work is completed. LedgerHub is now officially dead. 2016-02-20 - Create an beancount.utils.misc_utils.is_sorted() utility function to be used in the new importer code. - Match a price source over the entire source string. 2016-02-19 - Now accepting '=' in currency symbol in order to accommodate currencies in Yahoo, which look like this "CNYUSD=X". 2016-02-15 - Handle an incomplete pickle file by ignoring it and regenerating it silently. 2016-02-14 - Baked a 2.0b5 and 2.0b6 releases in a further attempt to get the included files to be included by setuptools and for pip3 to be able to make a local build. setuptools is horrible. - Baked a 2.0b4 release for a more recent PyPI availability. - Fixed issue #72 - Thanks to Gary Peck for figuring out what was wrong with setuptools - and now you should be able to install via pip3 again, which will pull all the dependencies automatically. If setuptools isn't installed, this should degrade peacefully to distutil. 2016-02-06 - IMPORTANT API CHANGES FOLLOW: I made some important changes to the following classes: beancount.core.position.Position beancount.core.inventory.Inventory beancount.core.data.Posting "Position" objects have been restructured to reflect the structure implied by the syntax. Instead of a .number (Decimal) and .lot (Lot) objects, a Position now has a .units (Amount) and .cost (Cost) objects. The "Lot" class has been removed; it has been replaced by a "Cost" class with .number (Decimal), .currency (string), .date (datetime.date), .label (string) attributes. The "Inventory" class is a list of "Position" objects, and as such it has received corresponding adjustments and minor changes. "Posting" objects do not contain a .position attribute anymore. Instead, the data structure has been flattened to contain .units and .cost objects directly. As a result, a Posting acts a bit like a subclass of a Position rather than using composition to contain one. During the parsing phase, a "Posting" object's .cost attribute does not point to a "Cost" instance, but rather to a "CostSpec" instance, which reflects the unresolved and partially incomplete lot specification from the input syntax that needs to get resolved to a particular lot during the new booking process. Here's a pictorial representation of changes. Before we had: Assets:Checking -40 QQQ {66.32 USD, 2015-12-14} @ 70.32 USD `---account---' `number' `--------------lot--------------' `--price--' `currency'`---cost--' `--date--' posting (Posting) .account (str) .position (Position) .number (Decimal) .lot (Lot) .currency (str) .cost (Amount) .number (Decimal) .currency (str) .lot_date (datetime.date) .price (Amount) .number (Decimal) .currency (str) After the changes we have something that looks more like the input: Assets:Checking -40 QQQ {66.32 USD, 2015-12-14, "0d6f0"} @ 70.32 USD `---account---' `-units-' `------------cost-----------------' `--price--' number currency date label posting (Posting) .account (str) .units (Amount) .number (Decimal) .currency (str) .cost (Cost, CostSpec, or None) .number (Decimal) .currency (str) .date (datetime.date, or None) .label (str, or None) .price (Amount, or None) .number (Decimal) .currency (str) And field-by-field conversions: posting.account -> posting.account posting.position.number -> posting.units.number posting.position.lot.currency -> posting.units.currency posting.position.lot.cost.number -> posting.cost.number posting.position.lot.cost.currency -> posting.cost.currency posting.position.lot.lot_date -> posting.cost.date (none) -> posting.cost.label posting.price.number -> posting.price.number posting.price.currency -> posting.price.currency Notes: * These changes are mostly driven by the need for the parser to output incomplete cost specifications. The booking of lots against existing inventory is isolated in a beancount.parser.booking and before booking we need to be able to represent missing values from the input syntax, since the booking process is able to interpolate and fill in these missing values. It was required to make important changes to the data structures for this reason. The parser outputs 'posting.cost' with an instance of a new "CostSpec" class, not "Cost". After booking, all the postings's costs are replaced by instances of "Cost" and "CostSpec" is gone. * This representation is much closer to the syntax. It's also flatter and simpler. The original representation's design was driven by an old idea that the lots were to be used as the key in Inventory mappings, but as it turns out, operations on lots are just as well implemented with an association list and performance isn't a problem. Moreover, lots with a cost are never merged using such a mapping, only reduced against, so this was an unnecessary distraction. * If you were creating Position objects in plugins or other code, please look at the new interface and make necessary changes to your code. You can create new Position objects from a pair of Amount and Cost objects. - beancount.core.position.lot_currency_pair() has been converted to beancount.core.position.Position.currency_pair(). - Renamed beancount.core.position.Position.cost() to beancount.core.position.Position.at_cost() due to the conflict resulting from the Position class changes. - There is a new option "booking_method" which should be set to the value "SIMPLE" to maintain the current booking algorithm (it's the default if left unset), but which can be set to "FULL" to enable the yet-unreleased, new experimental booking method. Do this at your own risk; this is not ready for public use, not effectively working yet. 2016-02-05 - Fixed bug with query parser: a single dot (.) would be matched as a number. It should require at least some digits. - Fixed bug whereby a missing cost number would be rendered as "0"; if needs be a null value. 2016-01-31 - Added python-magic as an optional depdendency, in order to support imports code. Updated http://furius.ca/beancount/doc/install - Renamed beancount.scripts.checkdeps to beancount.scripts.deps (annoyed with name too similar to beancount.scripts.check and I trip over it all the time). - Moved ledgerhub.core.compress to beancount.ops.compress and its associated test. - Moved ledgerhub.identify.filetype to beancount.utils.file_type and its associated test and test files. - Small improvement to beancount.utils.test_utils.skipIfRaises() to accept multiple exceptions. 2016-01-25 - Moved many of the utility code from ledgerhub.utils to beancount.utils. This is beginning the process of migrating and rewriting LedgerHub and integrating it into the Beancount source code. - Moved the 'remove_crdb' utility script from LedgerHub into beancount/experiments/imports. 2016-01-18 - Committed the Query directive to non-experimental status, for beancount-web's usage. 2016-01-16 - Created the beancount.plugins.coherent_cost plugin which checks that one's currencies are either always used with cost, or never at all. This is by far the most typical situation whereby a currency is always held at cost, or never (and perhaps converted at price or not). - Created a function to filter just the transactions, which is such an incredibly common operation I should have done this a while ago. I should convert all code to use it. 2016-01-16 - Added a feature to the "beancount.plugins.ira_contribs" plugin to allow the further restriction that the trigger posting must include a particular flag in order for the insertion to occur. This is to be used in account that have sales, whereby just looking at the sign of the posting isn't enough to decide that an insertion must be made. 2016-01-10 - Addressed issue #96, supporting an explicitly installed C decimal version when the Python3 installation does not include one. From beancount.core.number: Note: Python 3.3 is supposed to guarantee a fast "C" decimal implementation, as it comes with its source code. However, in practice, many distributions, including the popular Ubuntu distro, simply don't compile it. Other distros (e.g., Arch Linux) insist on breaking up Python into multiple packages in order to manage the dependency to the mpdec library, and make that installation optional. Moreover, there seems to have been changes in the dispatching of pure-Python and fast C decimal implementations in Python-3.5's source code itself. The result is that despite Python-3.3 or above being installed, a fast C decimal implementation may or may not be present. Thus, we must detect it here. I believe this subtle problem should go away with Python-3.5 and above. What do we do? We import the default 'decimal' module, and if it's the pure Python implementation, we attempt to import an explicitly installed 'cdecimal', which you can install with pip3 install m3-cdecimal Also added a check for this in checkdeps. Hopefully this resolves the issue once and for all. 2016-01-09 - Fixed a bug (issue #94) in the load cache that wouldn't handle files disappearing after having been moved. 2016-01-03 - Moved beancount.web.scrape.scrape() to beancount.web.web.scrape_webapp() to isolate beancount.web.scrape from dependencies, so that other projects may reuse it. 2015-12-25 - Closed multiple obsolete branches: parse_test_utils, display_context, performance, datatypes. - Added some utilities for parsing strings for writing tests that use the local variables present in the caller (a bit of magic). - Improved date parsing from command-line options for some scripts; imported from branch 'text_statements'. 2015-12-24 - Implemented metadata stack and "pushmeta" and "popmeta" directives mirroring "pushtag" and "poptag". These new directives work similarly, but setting metadata fields instead of tags themselves. The parser maintains a stack of values for each metadata field. Values explicitly set on the Transactions themselves are not overridden by the meta stack. 2015-12-21 - Created a new project script that uses specific metadata on Open directives to generate a document of a user's assets, institution names, addresses and contact info, and account types and approximate value. Note: I haven't made the example file include necessary fields to demo or document this yet. - The loader now supports loading encrypted files transparently. If a filename ends with .gpg or .asc (and contains armored content), the file is run through a gpg subprocess to decrypt it. No temporary files are created. Note that caching for those files is disabled. If you're writing a script, you can also call beancount.loaded.load_encrypted_file() directly to avoid the check. - Implemented the 'beancount.plugins.merge_meta' plugin that copies metadata from an external file into a regular file. It is essentially a data join on the key of accounts for Open and Close directives, and a join on the key of currency on Commodity directives. - Added an option to Inventory.to_string() to avoid rendering parentheses. - Added variable and username expansion support for the loader filename. - Removed --batch mode for the encryption support; it needs to be potentially interactive. 2015-12-17 - In prices, removed undocumented support for automatically extracting Commodities with a "quote" metadata field. This had been used in times past and had been brought in as legacy, but now neither the export code nor the prices code use it. Vaccummed away. 2015-12-13 - Fixed a minor bug with the loader cache whereby errors weren't getting emitted on a cache hit (Issue #92). 2015-12-12 - Made bean-web's --incognito mode replace integer numbers, not just fractional ones. (See issue #86). - Output a warning when cdecimal isn't installed. This may occur on some distribution which break up Python into multiple packages in order to allow the user to avoid depending on a mpdec library install. Arch Linux suffers from this problem. - Made some improvements to the loader caching algorithm in order not to rely on the percision of timestamps from the filesystem. Filesystems implement one-second precision which is not sufficient in some case. I now cache both the time and the size of files in order to check for a cache hit, and this paves the way for hashing the contents of files as well eventually. - Began implementing aggregation of some of the attributes of options map across include files. Both "operating_currency" and "commodities" will be aggregated correctly. (This was a request for beancount-web.) - Fixed flaky test that failed only in homecooked install of Python 3.5.1. - Fixed a minor bug that would not print out costs with zero amounts. 2015-12-11 - In the context page dedicated to rendering the details of an entry, render an "Open" link on the upper right that can be clicked on in order to invoke a desktop handler that will open a text editor to the containing file and at the right line number. 2015-12-10 - Renamed beancount.reports.context.render_entry_context() to render_file_context() and implemented render_entry_context() to accept an entry directly instead of a (filename, line-no), so that bug reporters can insert some print statements. I also removed the dcontext instance required to be passed, since all the callers were fetching it from options_map. This should make it easier for those sending bug reports to insert quick calls to render the context around a particular entry or file location. 2015-12-09 - Fixed installation snafu: forgot to add the beancount.prices and bean-price install to the setup.py file. 2015-12-07 - Fixed a bug in the load cache, whereby the load cache would not update itself on an included file. In order to do this, first I made the loader always return the full correct set of unique filenames processed in a load call, and then I used this to check the latest modified time. - I also used this to fix a related bug in the bean-web watcher: If an included file was being modified, bean-web would not reload itself automatically. redstreet0 sent a pull request to highlight the problem (thanks!): https://bitbucket.org/blais/beancount/pull-requests/6/feature-include-files-refresh/diff 2015-12-03 - Added ROOT(account, num) account manipulation function to the shell. 2015-12-02 - Fixed issue #84: The sorting of entries from multiple files was incorrectly placed, due to recent changes merged from the booking branch. - Fixed a bug deleting the load cache (was using an invalid filename). 2015-11-29 - Made the pickle load cache enabled by default. This means you can now remove the BEANCOUNT_LOAD_CACHE environment variable. The load cache will be written out automatically, but only if it takes more than the time threshold (1 sec) to load a Beancount file. If it takes less than this time, no cache is written. (This saves the creation of these pesky crumbs for files it doesn't matter for.) If you would like to always disable the load cache, set the environment variable BEANCOUNT_DISABLE_LOAD_CACHE. I don't recommend it--this works well and for large files makes loading much, much faster. 2015-11-28 - Implemented a script used to fetch prices online and output them in Beancount input format. This script is able not only to take an explicit list of prices to get, but also able to read a Beancount input file and figure out the list of commodities held at a particular point in time and fetch only those prices. A "price" metadata field on a Commodity directive is used to specify how and where to get prices from. It comes with support for Google Finance and Yahoo! Finance price sources and the syntax supports custom-written Python modules so you can write your own price fetching extensions, that integrate with the price fetching tool. The new code is located under "src/python/beancount/prices" and this new tool is documented in http://furius.ca/beancount/doc/prices. (Note: This is brand new code, probably imperfect, but I tried to keep this simple yet complex enough to be practical. I've been using a variant of it for a while, I'm going to merge this now and use it myself, and I hope it works for others.) 2015-11-26 - I redesigned the portfolio export process in order to (a) simplify it greatly (the process was a bit confusing, between the use of three metadat fields: "ticker", "export" and "quote"), and (b) to separate it entirely from the upcoming price fetching script metadata. The new outcome is very similiar, however, except it uses only a SINGLE metadata field: "export", and the documentation has been rewritten to reflect this: https://docs.google.com/document/d/1mNyE_ONuyEkF_I2l6V_AoAU5HJgI654AOBhHsnNPPqw/edit# If you have an existing file, here's what you need to do to it: * Remove all the 'quote' metadata. You won't need it. * If you have 'ticker' metadata: convert those to 'export'. * If you have 'export' metadata: keep it. * If you had an 'export' metadata with a "MONEY" spec, move that into the 'export' field after the symbol to export to. The example has been ported, so you can refer to that as well. I think the new process is much simpler to understand and the documentation shorter and easier to read. It should be a trivial exercise to convert one's input file to the newer format. 2015-11-22 - Removed beancount.core.data.AttrDict and replace it by a regular dict. This removes some annoyances when pickling and copying metadata. Metadata is now represented by a plain old dict. (Fixes issue #79.) 2015-11-09 - Implemented a simple pickle-date on beancount.loader.load_file(), which will reload the parsed & processed contents of a top-level file if a cache file is present and more recent than the file itself, or update the contents of the cache after loading if not. (This works a bit like .pyc byte-compilation in Python or .elc in Emacs LISP.) This makes the loading of large files multiple times ~15x faster, which is a big deal since Beancount has become a little bit slow (I need to spend time optimizing at some point). This feature is a bit experimental, so for now it only gets turned on if the BEANCOUNT_LOAD_CACHE environment variable is set. 2015-10-04 - Implemented --output= option for bean-query tool. 2015-10-02 - Implemented an experimental new Query directive whose purpose is to define pre-canned SQL queries in the Beancount file itself and to then build tools to automate the generation of reports of those queries. For the moment being, I'm just playing with this to convince myself that this can be powerful and generic enough to provide more generally. I've found myself running specific queries on speific files, e.g., during trips and I'd like to automate those to some extend, perhaps replace the script code that lives in beancount.plugins.split_expenses with this. Anyhow, this is not supported but if you understand this and really want to turn it on, set this option in your input file: option "experimental_query_directive" "TRUE" The directives aren't used anywhere yet, and you'll have to read source code to figure out what they look like. - Implemented a "reload" command in the shell. Typing "reload" will refresh the data from the latest file contents. 2015-09-29 - Tightened the screws on the Amount() constructor a bit and made changes ini order to allow it to accept a sentinel value for MISSING values from the parser. Changes to beancount.core.amount.Amount are: 1. A string for the number is not accepted anymore, that is, Amount(str, str) is not supported anymore. Amount() now unconditionally requires a Decimal instance, so you must use Amount(Decimal, str). In any case, this constrcutros was likely abused only by me in the tests. I'd rather have a tighter, less permissive constructor. 2. The convenience constructor beancount.core.amount.from_string() is now officially aliased as beancount.core.amount.A, so you can "from beancount.core.amount import A". I now commit to using this constructor and converted all the tests to use it. This is great for writing tests. Amount really is a fundamental piece that deserves a very short alias like this. In summary, how to create Amount instances: * If you use explicit values, use Amount(Decimal, str), e.g., Amount(D('100.00'), 'USD'), Amount(number, currency) * If you need a convenience constructor, use A(str), e.g., A('100.00 USD'), A('32,000 KRW') 2015-09-29 - Added a new column 'description' to the SQL client in order to produce more compact reports. Oftentimes the payee is left to an empty value and journals have a lot of wasted whitespace. Concatenating the payee and the narration in a single field goes a long way towards producing reports that aren't nearly as wide. I could have added a CONCAT() function but this is so common and convenient that a dedicated 'description' alias is justified. You can use it in posting context ("SELECT description ..." or "SELECT ... WHERE description ~ '...'") or transaction context ("SELECT ... FROM description ~ '...'").. 2015-09-27 - Implemented a plugin that can book price conversions using a particular booking method, in this case, FIFO. This idea stemmed from a suggestion I made on the mailing-list a few weeks ago, which was to process a particular account's price conversions to insert cost basis specifications and in doing so to book reducing postings to FIFO order automatically. This could all be done via a plugin, so I did it. (See tag {c395de691449} on the mailing-list for more context.) You can find it under "beancount.plugins.book_conversions". You use it like this: first you load the plugin: plugin "beancount.plugins.book_conversions" "Assets:Bitcoin,Income:Bitcoin" The configuration requires you provide two accounts: 1. An assets account to process, whereby all price conversions (in any currency) occurring in that account will have their cost basis automatically inserted, and, 2. An Income account that is inserted on reducing transactions to absorb the profit or loss of the sale. Then you book your transactions as simple price conversions, without worrying about cost basis, for example, this input: 2015-09-04 * Assets:Bank -750.00 USD Assets:Bitcoin 3.000000 BTC @ 250.00 USD 2015-09-05 * Assets:Bank -780.00 USD Assets:Bitcoin 3.000000 BTC @ 260.00 USD 2015-09-20 * Assets:Bitcoin -2.000000 BTC @ 300.00 USD Expenses:Something 2015-09-21 * Assets:Bitcoin -2.000000 BTC @ 310.00 USD Expenses:Something 2015-09-22 * Assets:Bitcoin -2.000000 BTC @ 330.00 USD Expenses:Something Would get translated by the plugin into the equivalent of this: 2015-09-04 * Assets:Bitcoin 3.000000 BTC {250.00 USD} @ 250.00 USD trades: "trade-ff4c3e592147,trade-6b9c8689ed50" Assets:Bank -750.00 USD 2015-09-05 * Assets:Bitcoin 3.000000 BTC {260.00 USD} @ 260.00 USD trades: "trade-91e2245de593,trade-d622a89c614d" Assets:Bank -780.00 USD 2015-09-20 * Assets:Bitcoin -2.000000 BTC {250.00 USD} @ 300.00 USD trades: "trade-ff4c3e592147" Income:Bitcoin -100.00000000 USD Expenses:Something 600.00000000 USD 2015-09-21 * Assets:Bitcoin -1.000000 BTC {250.00 USD} @ 310.00 USD trades: "trade-6b9c8689ed50" Assets:Bitcoin -1.000000 BTC {260.00 USD} @ 310.00 USD trades: "trade-91e2245de593" Income:Bitcoin -110.00000000 USD Expenses:Something 620.00000000 USD 2015-09-22 * Assets:Bitcoin -2.000000 BTC {260.00 USD} @ 330.00 USD trades: "trade-d622a89c614d" Income:Bitcoin -140.00000000 USD Expenses:Something 660.00000000 USD Note that the price annotations are kept as in the input file. The plugin only inserts the cost basis and splits postings where required. Secondly, a unique trade identifier is allocated for each matching pair of postings that were booked and is added to the metadata of those postings. Its value is a comma-separated list. This metadata is useful because it allows you to load the entries and from it, to build a list of trades to produce a suitable report for reporting. So the plugin also contains code to do this. You can invoke it as a main program, like this: python3 -m beancount.plugins.book_conversions On the file above, it should generate the following output, which I believe is the report you're really after for reporting purposes: Units Currency Cost Currency Buy Date Buy Price Sell Date Sell Price P/L -------- -------- ------------- ---------- --------- ---------- ---------- ------ 2.000000 BTC USD 2015-09-04 250.00 2015-09-20 300.00 100.00 1.000000 BTC USD 2015-09-04 250.00 2015-09-21 310.00 60.00 1.000000 BTC USD 2015-09-05 260.00 2015-09-21 310.00 50.00 2.000000 BTC USD 2015-09-05 260.00 2015-09-22 330.00 140.00 -------- -------- ------------- ---------- --------- ---------- ---------- ------ Use the '--format=csv' option if you want to produce output suitable for including in a spreadsheet. If you need to customize the output further, see the source code (file:book_conversions.py). It is not possible to generate this list of trades using the SQL client yet, to change the output columns you would have to create a standalone script; you can copy just the main() function in the plugin if you need to do that and import the rest. (Note: The function that extracts the list of matches provides both the posting and its parent Transaction object, so you can output data from the transaction if you want, e.g. the narration of the selling transaction.) 2015-09-14 - Merged more changes to beancount.el (Emacs support) by Stefan Monnier: * Use syntax-tables for comments and strings. * Make ido use optional in Emacs config. * Quote filenames on command lines Thank you Stefan! 2015-09-12 - Added to beancount.plugins.split_expenses a new script that is able to generate the typical kinds of reports you would need to provide to participants in a shared project or travel on their specific expenses and contributions, as well as a breakdown of their expenses per category (expenses account) and final balances for each participants. Invoke it like this: python3 -m beancount.plugins.split_expenses The script lives in the same file as the split_expenses plugin, because it is tightly related to it. It presumes the input file will process through this plugin. Try to run it on the example files under beancount/examples/sharing. - Added a new file "examples/sharing/duxbury2015.beancount" which contains a new example of sharing expenses on a trip between three people. - Starter cleanup of beancount.el contributed by Stefan Monnier. More changes to come, this is basically a no-op patch. It just moves code around and makes other such cosmetic changes to silence some compiler warnings, prepare for subsequent changes, better follow coding conventions etc. Detail: * beancount.el: Add some standard file headers, license, and sectioning comments. Sprinkle a few FIXMEs while I'm here. (beancount-font-lock-defaults): Rename from beancount-font-lock-keywords, since this corresponds to font-lock-keywords and not to font-lock-defaults. (beancount-directive-names, beancount-font-lock-keywords, beancount-accounts): Move variables before their first use (to silence the compiler). (beancount-mode-map): Use a proper defvar instead of define-minor-mode's half-broken :keymap. (beancount-mode): Merge the make-local-variables into their corresponding `set'. (beancount-accounts): Move make-variable-buffer-local to top-level since it's a global operation. (beancount): Declare this group, already used in define-minor-mode. (beancount-hash-keys): Mark unused arg `v' with an underscore. (beancount-transaction-set-flag): Don't use replace-string (designed for interactive use) from Elisp. (beancount-for-line-in-region): Use copy-marker and eobp. (beancount-align-numbers): Break lines to avoid hiding arguments. (compilation-read-command): Declare variable to silence byte-compiler. 2015-09-06 - Filter out platform-specific comments from the code generated by GNU Bison, in order to avoid unnecessary diffs when I switch between Linux and Mac OS X. 2015-09-05 - Changed the semantics of the account_rounding account, so that it matches that of the other account options, that is, they all specify a sub-account of the Equity account. So if you had option "account_rounding" "Equity:RoundingError" in your input file, change this to: option "account_rounding" "RoundingError" To ease the transition, the parser now checks if the first component is one of the five categories and if so, issues a warning and removes the prefix. (One side-effect of this change is that the rounding error account may only be an Equity account, but I'm hoping this won't be a problem for anyone.) 2015-08-31 - The column "weight" was broken; fixed by yagebu@. 2015-08-30 - Fixed a very minor bug in split_expenses plugin whereby the generated postings did not contain the __automatic__ metadata field, and in some particular situations, their automatically calculated values would end up being used for inferring the tolerances. 2015-08-15 - Changed the semantics of the parsing stage, in a fairly profound way. This should have no visible changes to users, but people writing scripts should revise their code if they were using beancount.parser.parser.parse_*() functions. Just to be clear: beancount.loader.load_*() has not changed. If you just use the loader, there are no changes. Changes are only at the parser level. Here's what's going on and why: The parser used to carry out interpolation of missing numbers on postings for each transaction locally, while parsing the transactions. This was done by calling by calling beancount.core.interpolate.balance_incomplete_postings(), here: https://bitbucket.org/blais/beancount/src/ee2073aae080aaa8e260abe8a501abf872948f0e/src/python/beancount/parser/grammar.py?at=default#grammar.py-803 Loading a list of entries was carried out in two steps: ,-----------------------load----------------------. (recursively) ,---------------------. ,---------. ,------------. (input)--->| parse + interpolate |-->| plugins |-->| validation |--> entries `---------------------' `---------' `------------' First, the parser would run on the input and process all the input files recursively (processing includes). "Interpolation", the process of filling in missing numbers, was carried out at that stage, during parsing, and only locally, that is, for each transaction in isolation. "Booking" of lots, that is, selecting which of an account's inventory lots to match and reduce, was explicit. This booking could not take advantage of the accumulated inventories in order to vary its behavior. You had to specify the entire lot information unambiguously. After this, in a second stage, the plugins were run on the entries and a final validation step was run at the end. To implement the booking proposal (http://furius.ca/beancount/doc/proposal-booking), we want for the user to be able to provide a partial specification of lots to be matched against an account's accumulated inventory at the date the transaction is to be applied. The idea is that if there is no ambiguity, the user should be able to specify very little information about a lot (for example if there is a single lot in the account when processing the transaction an empty spec of "{}" should be sufficient). Moreover, where the specification is ambiguous, we also want to support automatic selection of lots according a method specified by the user, e.g., FIFO booking, LIFO booking, etc. For this to work, we need to have parsed all the inputs to some sort of incomplete specification, a representation of inputs that hasn't yet been resolved to specific lots, in order to carry out booking. The parser has been modified to output such incomplete postings: ,-------------------------load-------------------------. (recursively) ,-------. ,-------------. ,---------. ,------------. (input)--->| parse |-->| booking |-->| plugins |-->| validation |--> entries `-------' | + | `---------' `------------' | interpolate | `-------------' incomplete entries Because "interpolation" runs on the result of specific lot values, "booking" must run before it, and so they are inter-related. Thus, booking and inteprolation has been moved to a dedicated step that runs on the list of incomplete entries and resolves them to the regular entries to be processed further by plugins and validation. This also has a nice side-effect: the booking step is where all the complexity is, and it is now isolated and I will be able to test and experiment on it in isolation. This is where all the fun will be. A description of the incomplete specifications output by the parser can be found here in the parser.py file, this is the description of the intermediate state of postings whose lots haven't yet been matched and resolved to specific inventory lots: https://bitbucket.org/blais/beancount/src/18282452e265959b69d3d10c6d9cf32e5815c522/src/python/beancount/parser/parser.py?at=booking Essentially, a posting's 'lot' attribute contains a "LotSpec" tuple instead of a "Lot" tuple, and several numbers may be left unfilled (for interpolate values). I don't imagine anyone will ever have to manipulate such intermediate entries, only the booking code. The previous booking algorithm has been moved to the booking stage and the semantics should be identical to what they used to be. This is still the default algorithm--it just runs in its own dedicated stage, still operating locally on each transation. You should observe no difference in behaviour. I've merged these changes now in order to minimize the differences between the booking and default branches and because I was able to do it without changing any of the semantics (despite the large number of lines and tests modified). This was a necessary refactoring. Because that code is now isolated to its own stage I should be able to begin implementing the more complex state-dependent booking algorithms in the 'booking' branch. (I'm excited about this.) I could even implement different booking heuristics and switch between them. Because the parser used to spit out regular, complete entries, parser.parsedoc() was used in much of the tests instead of loader.loaddoc(), so that many of those tests would not have to concern themselves with making sure the input passed the validation stage run by the loader. For example, creating Open directives just to create some Transaction test object wasn't necessary in the tests. All of those tests had to be revised and I made them all depend on beancount.loader.load_*() instead of the now weakened parser.parse_*() functions which output incomplete, unbooked entries. This cleans up the dependencies a bit as well. If you wrote your own unit tests and were using parser.parsedoc(), you should convert them to use loader.loaddoc(). In order for this change not to go unnoticed (and for naming consistency with parse_string() and parse_file()) I'll probably rename parser.parsedoc() to parser.parse_doc() and ditto with the loader. (Note: In some cases I've had to specifically setup the input of some tests in "raw" plugin processing mode to avoid triggering some unwanted and unrelated errors. I'm tempted to remove even more from the default plugins. This may happen in a future CL.) - Renamed beancount.parser.parser.parsedoc() to parse_doc() and beancount.loader.loaddoc() to load_doc(), for consistency with the other parse_*() and load_*() functions. Kept a stub that will issue a warning if you use it. 2015-07-23 - In the conversion to the new booking syntax, I had inadventently removed support for the total cost "{{ ... }}" syntax. I brought this back next to the new booking syntax, and uncommented tests that had been made to skip. 2015-07-21 - Merged ongoing work from the 'booking' branch that will eventually change the semantics of inventory booking; in the interest of a smooth transition, and for me to be able to use either branch interchangeably, I introduced a few changes that should have no user-visible effect on 'default': * The parser.parse_string() and parser.parse_file() routines don't interpolate anymore. For this reason, the tests all had to be adjusted not to include interpolation. The interpolation of incomplete postings (grammar.interpolation()) has moved to a separate phase and is now run by loader._load() *AFTER* parsing. This is key to implementing fuzzy matching semantics for matching reducing lots: we need to have all the incomplete transactions parsed and sorted in order to select matching lots in date order. This only affects you if you wrote scripts against the parser interface directly (this is highly unlikely). * For writing unit tests, parser.parsedoc() and loader.loaddoc() are now decorator factories. This allowed me to add options to parser.parsedoc() to perform interpolation, and when not specified, to check that entries with interpolation are not present in the tests. Also, I coudl merge the functionality of parser.parsedoc_noerrors() and loader.loaddoc_noerrors() in their respective equivalents. The docstring tests now validate that there are no errors in the docstrings by default. This makes the tests tighter (a few bugs in the tests themselves were found and fixed). * The new syntax for cost specification that will be in effect for the inventory booking proposal is now supported. The syntax is backward-compatible with the previous one. Previously, the following syntaxes were supported for specifying the cost and optionally a lot acquisition date: Assets:Investments 1 HOOL {123.00 USD} Assets:Investments 1 HOOL {123.00 USD / 2013-07-20} Instead of these options, the new syntax supports a comma-separated list of cost-specifiers which can be one of -> e.g. 123.00 USD -> e.g. 2015-07-20