3.8.2 Enhancements: * The MergerConfig class now accepts overrides for config values as "keys" and "rules" keyword arguments to the constructor. Credit and my thanks go to https://github.com/leviem1! BREAKING CHANGES: * Support for Python 3.6 has been dropped. This is forced by incompatibilities discovered with the latest version of pytest and because dependencies like dateutil and ruamel-yaml-clib no longer support Python 3.6. Support for Python 3.7 is tepid. While pytest is still working with Python 3.7, other dependencies are no longer supporting Python 3.7; however, the extensive tests for yamlpath show no issues with them, so far. For now, Python 3.12 support is pending, waiting for the dateutil library to resolve a DeprecationWarning regarding its use of datetime.datetime.utcfromtimestamp(). 3.8.1 Bug Fixes: * Fixed issue #220 reported by https://github.com/abramov-oleg wherein novel mergeat paths given to the yaml-merge tool with Array and Array-of-Hash data in the RHS document would cause an interminable loop that would max out CPU and eventually exhaust RAM. 3.8.0 Enhancements: * The yaml-set and yaml-merge command-line tools now support a new option: --json-indent/-J. This applies only to JSON output. By default, JSON documents are written/printed as single-line documents. This new option enables users to vertically expand JSON output with a specified indentation size for nest levels. * Additional *YAMLPathExceptions have been added to increase the specificity of various error conditions. You do not need to change any of your code as a result of this change; all new exceptions are subclasses of the original YAMLPathException. The new exceptions include: * UnmatchedYAMLPathException: No matching nodes * DuplicateKeyYAMLPathException: Duplicate key creation attempted * TypeMismatchYAMLPathException: Data Type mismatch * BadAliasYAMLPathException: Anchor/Alias/YMK error * NoDocumentYAMLPathException: Attempt to delete the entire document * RecursionYAMLPathException: Recursion nesting error * The Processor class now enables consumers to verify whether a YAMLPath would resolve to at least one node in the present document. The result is a simple Boolean (True when the path resolves; False, otherwise). The document is not modified by this test. See: Processor.exists(yaml_path) Bug Fixes: * Pursuant to the addition of the intersection operator (&) added in 3.7.0, the text of one of the YAMLPathExceptions has been updated: Former: Adjoining Collectors without an operator has no meaning; try + or - between them, {yaml_path} New: Adjoining Collectors without an operator has no meaning; try +, -, or & between them, {yaml_path} 3.7.0 Enhancements: * Support for Python 3.11 has been added. * A new Search Keyword has been added, [unique([NAME])]. This operates against collections to return only values which have no duplicates within the collection; i.e.: [1, 2, 2, 3] has unique values, [1, 3]. The NAME argument is required when operating against Hashes (maps/dicts) and Arrays-of-Hashes (sequences/lists of maps/dicts) to identify which field/property to evaluate for uniqueness. This can be inverted to return only results which are duplicated within the collection. * A new Search Keyword has been added, [distinct([NAME])]. This operates against collections to return exactly one of every value within the collection, discarding duplicates; i.e.: [1, 2, 2, 3] has distinct values, [1, 2, 3]. The NAME argument is required when operating against Hashes (maps/dicts) and Arrays-of-Hashes (sequences/lists of maps/dicts) to identify which field/property to evaluate for uniqueness. This cannot be inverted. * Added a new Collector Math Operator: & (intersection) As in set mathematics, this yields only elements of two collections which are common to both collections. Unlike set mathematics, collections allow duplicate elements. If you need to enforce distinctness of the intersected results, use the [distinct([NAME])] Search Keyword against the collected result, as in `((list1)&(list2))[distinct([NAME])]`. Bug Fixes: * A typographical error in yamlpath.enums has been corrected with backward- compatible adapters in place to support both the correct and incorrect spelling of PathSeparators (formerly PathSeperators). If the PathSeperators version appears in your own code, please update to the new spelling. The incorrectly spelled version of this enumeration is now deprecated and will be removed in a future release. Thanks go entirely to https://github.com/AndydeCleyre for working so hard to submit this fix! * Processor.get_nodes would emit a nonobvious error message when mustexist is left at its default value (False) and yaml_path contained ** or * segments. Now, these segment types are excluded from generating "missing" nodes; only nodes which exist can be matched by * and **. Credit and my thanks go to https://github.com/gdubicki for discovering and reporting this issue. 3.6.9: Enhancements: * Partial support for updating bare Python dict and Python's native collections.OrderedDict data structures was removed in version 3.6.8 because compatible YAML/EYAML/JSON data never presented as these data types and if anyone ever attempted to update a key by reference in a dict or collections.OrderedDict, it would cause a Python stack dump due to neither supporting the required insert method, which is provided only by ruamel.yaml. This version not only restores this capability, but also solves the issue of missing support for the insert logic, where applicable. It also adds support for the ruamel.yaml.compat.ordereddict type. Thanks to https://github.com/tsinggggg for requesting this feature be added! 3.6.8 Bug Fixes: * Changes to format and value of child nodes under Anchored Hashes (maps/dicts) caused unexpected expansion of the same key-value pair wherever a YAML Merge Key Aliased the affected Anchored Hash. Thanks to https://github.com/hemnstill for finding and reporting this issue! Enhancements: * Support for Python 3.10 has been added, thanks to https://github.com/mechie! 3.6.7 Bug Fixes: * Release 3.6.6 had a broken package; it was unexpectedly missing the required ruamel.yaml patch found in yamlpath.patches.timestamp. Thanks to https://github.com/tsinggggg and https://github.com/kaniblu for reporting it! 3.6.6 Enhancements: * Support ruamel.yaml up to version 0.17.21. Bug Fixes: * YAML timestamp values could not be created via yamlpath tools or its library, per http://yaml.org/type/timestamp.html. * CAUTION 1: ruamel.yaml seems to force all timestamps to UTC while it loads YAML/JSON/Compatible data. So, when the timestamp value contains time-zone data, it will be stripped off after it is applied to the timestamp value. The yamlpath library reverses this when emitting the affected values but if you attempt to load the timestamp values directly in the DOM, you'll end up with the UTC-converted value, stripped of any time-zone specification. If you need the original, unmodified data as a time-zone aware datetime.datetime value, pass it through the helper method, Nodes.get_timestamp_with_tzinfo(data: AnchoredTimeStamp). * CAUTION 2: In order to support timestamp parsing, this project now depends on the python-dateutil library. Be sure to install it! Thanks again go to https://github.com/AndydeCleyre! 3.6.5 Bug Fixes: * When using EYAML with block formatted values on Windows, the block formatting was broken when written to YAML, corrupting the YAML file. Non-block formatted values were unaffected and this issue only affected Windows EYAML users. * EYAML values were not being decrypted when the result appeared in a list. * Boolean values were being unexpectedly output as base-10 representations of the binary values 1 (True) and 0 (False) when emitting to JSON after setting the Boolean. This change also brings this project into compilance with https://peps.python.org/pep-0632/. Many thanks to https://github.com/AndydeCleyre! 3.6.4 Enhancements: * Support ruamel.yaml up to version 0.17.17. Bug Fixes: * Refactored single-star wildcard segment (*) handling to enable filtering matches when subsequent segments exist; this fixes Issue #154. 3.6.3 Bug Fixes: * The eyaml-rotate-keys command-line tool failed to preserve block-style EYAML strings after key rotation. 3.6.2: Bug Fixes: * The eyaml-rotate-keys command-line tool would generate a stack-dump when the key for an encrypted value contained dots. The underlying library for this tool now safely generates the internal YAMLPaths it uses. * The default encoding when opening files is now set to utf-8 to support extended character sets on Windows. 3.6.1: Enhancements: * Enable verified support for ruamel.yaml up to version 0.17.10. 3.6.0: Bug Fixes: * Some peculiar data constructions could cause the ConsolePrinter to stack-dump when writing debug messages. * A NotImplementedError stack-dump was generated whenever a YAML Path was evaluated when written improperly for Collector Math operations. This specifically occurred when the RHS term was not demarcated with a parenthesis pair. Now, a YAML Path parsing error is generated, indicating where the missing character must appear. * Use of an Array Element index in square-bracket notation ([N]) within a sub-path -- such as in a Search Expression or Collector -- caused incorrect YAML Path parsing. This usually manifested as a "not an integer index" error. * Byte strings were causing stack-dumps during JSON serialization; they are now serialized as double-demarcated strings (a ' pair within a " pair) with a b prefix, like: {"byte_value": "b'BYTEVAL'"}. * Bare Anchor name references were treated as Hash key names when & was not the very first non-separator character of a YAML Path or immediately following a `[`. So, /my_hash/&my_anchor was not working as expected. * The Merger (and thus, the yaml-merge command-line tool) would only return the LHS document when neither LHS nor RHS documents were Hashes, no matter what the merge options were set to. This did not affect content which was children of Hashes. Enhancements: * YAML Path parsing errors are now a little more specific, indicating at which character index the issue occurs. API users who have been scanning error messages will need to update their code to accommodate the new text. * Collector subtraction now handles Hash-of-Hashes and Array-of-Array results, which were not possible before. * Array-of-Hash nodes can now be searched for the presence of a given key in its Hash elements using the . search operand, yielding matching elements (the entire Hash elements having the given key). The difference can be illustrated by contrasting these now-equivalent YAML Paths (where "books" is an Array-of-Hashes; imagine only some Hash elements have an "isbn" key): 1. `/books/*[.=isbn]` or `books.*[.=isbn]` 2. `/books[.=isbn]/isbn` or `books[.=isbn].isbn` 3. `/books/*[has_child(isbn)]/isbn` or `books.*[has_child(isbn)].isbn` 4. `/books[has_child(isbn)]/isbn` or `books[has_child(isbn)].isbn` All four of those queries yield exactly the same data. Note that example 2 behaves like examples 3 and 4. Examples 2-4 yield the entire matching Hash, not just the "isbn" value. This enables access to other keys of the Hash without necessitating use of a `[parent()]` search keyword, which would be necessary for example 1 if you wanted to access any key other than "isbn" from the matches. * YAML Merge Keys can now be accessed directly by Anchor name, yielding the entire original -- pre-merged -- reference Hash. This has _very limited_ utility. Using this in isolation will only reveal the default values for any referenced keys, ignoring -- perhaps confusingly -- any local overrides. It can however be helpful when reverse-engineering very complex merge arrangements. * The yaml-merge command-line tool (and the underlying Merger class) now offer an option -- --preserve-lhs-comments (-l) -- that will attempt to preserve LHS document comments. USE WITH CAUTION. At present, comment handling during a merge is unwieldy, so some comments or new-line characters may appear to become divorced from nodes they should obviously be attached to. As such, the default behavior of the merge engine will continue to be removal of all comments. At this time, RHS document comments will still be discarded during merge operations. This will be revisited when ruamel.yaml refactors how YAML comments are handled. * The yaml-merge command-line tool now offers a new option, --multi-doc-mode (-M), which accepts one of the following modes: * CONDENSE_ALL: This is the default, which merges all multi-documents up into single documents during the merge. * MERGE_ACROSS: Condense no multi-documents; rather, only merge documents "across" from right to left such that the first document in the RHS multi- document merges only into the first document in the LHS multi-document, the second across similarly, and so on. * MATRIX_MERGE: Condense no multi-documents; rather, merge every RHS document in a multi-document RHS into every LHS document in a multi- document LHS. * The [has_child(NAME)] Search Keyword now accepts an &NAME form of its first (only) parameter. This switches the function to match against Anchor/Alias names, including YAML Merge Keys. * YAML Merge Keys can now be deleted by their Anchor/Alias name via the yaml-set command-line tool and the underlying Processor class. * YAML Merge Keys can now be created, offering run-time merging of same-document Hash data. The yaml-set command-line tool offers a new option, --mergekey, which applies to --change targets the new YAML Merge Key, as long as each target is a Hash. WARNING: As a consequence of adding this capability to the yaml-set command- line tool, it is no longer possible to implicitly alias scalar nodes by passing only the --change and --anchor parameters. The operation must now be explicit by setting --aliasof or --mergekey along with --change and optionally with --anchor. * The yaml-diff tool now supports multi-document sources. Only one document of any multi-document source on each side of the LHS-RHS comparison can be selected for the operation (diffs are performed only between two documents). Such selection is made via two new options, --left-document-index|-L and --right-document-index|-R. An error is emitted whenever a multi-document source is detected without an appropriate document index selection. * YAML Unordered Sets -- https://yaml.org/type/set.html -- are now fully supported in YAML Paths, this project's API, and the reference command-line tools. Because an Unordered Set is effectively a Hash (map/dict) where the entries are key-value pairs all having null (None) values, their entries are accessible only by their exact key. While they look in YAML data like Arrays (sequences/lists) with a leading `?` rather than a `-`, they are not; their entries cannot be accessed by a numerical index because they are defined in the YAML specification as deliberately unordered. API Changes: * The common.nodes utility class now has a generally-useful static method which accepts any String data and safely converts it to its native Python data-type equivalent with special handling for case-insensitive Booleans via ast.literal_eval: typed_value. * The common.searches utility class now requires both terms to be of the same data-type for comparisons. When they types materially differ -- int and float are treated as similar enough -- a String comparision is performed. This is how it has always been excepting that types were lazily coalesced in older versions; they are now converted before the comparison is considered. * The NodeCoords wrapper now supports more utility properties and methods: * .unwrapped_node is the same output as calling NodeCoords.unwrap_node_coords(data) except it can be called directly upon instances of NodeCoords rather than as a static method call. The static method is still available. * .deepest_node_coord returns whichever NodeCoord instance is most deeply wrapped by an instance of NodeCoords. Such wrapping comes from nesting Collectors. This method simplfies getting to the original data element(s). * .wraps_a(Type) indicates whether the deepest wrapped data element is of a given data-type. 3.5.0: Bug Fixes: * Search expressions against Boolean values, [key=True] and [key=False], were impossible. Now, they are working and are not case-sensitive, so [key=True], [key=true], [key=TRUE], and such all work as expected. * When null values were present, Deep Traversal (**) segments would always return every node with a null value even when they would not match filter conditions after the ** segment. When mustexist=False, this would also cause a YAMLPathException. * Descendent searches were considering only the first child of the search ancestor. Now, ANY matching descendent node will correctly yield the ancestor. * Some Python-generated complex data types were escaping JSONification, leading to unexpected stack-dumps when writing out JSON data for data types like date and datetime. Enhancements: * An entirely new segment type has been added to YAML Path and is now supported by the library and reference implementation command-line tools: Keyword Searches. Similar to programming language keywords, these reserved Keywords work much like functions, accepting parameters and performing algorythmic operations or returning data not otherwise accessible to other YAML Path segment types. These new capabilities -- explored on the project Wiki -- include: * [has_child(NAME)] * [name()] * [max(NAME)] * [min(NAME)] * [parent([STEPS])] * When stringified, YAML Paths with a solitary * wildcard segment were printed using their internal RegEx variant, [.=~/.*/]. They are now printed as they are entered, using a solitary *. As a consequence, any deliberate RegEx of [.=~/.*/] is also printed as its equivalent solitary *. * The yaml-paths command now allows printing YAML Paths without protective escape symbols via a new --noescape option. While this makes the output more human-friendly, the unescaped paths will not be suitable for use as YAML Path input to other YAML Path processors where special symbols require escaping. * [API] The NodeCoords class now tracks ancestry and the last YAML Path segment responsible for triggering its generation. The ancestry stack -- List[AncestryEntry] -- was necessary to support the [parent()] Search Keyword. The responsible YAML Path segment tracking was necessary to enable Hash/map/dict key renaming via the [name()] Search Keyword. These optional attributes may be set when the NodeCoords is generated. * [API] YAMLPath instances now have a pop() method. This mutates the YAMLPath by popping off its last segment, returning that segment. 3.4.1: Bug Fixes: * yaml-set (and the underlying Processor class) were unable to change nodes having a null (None) value to anything else. This changes how null/None values are handled by the Processor during node retrieval; they are no longer discarded, so you will receive None as the data of any retrieved NodeCoords for appropriate null/None leaf nodes. Enhancements: * Python 3.9 is now supported (because common testing tools finally work with Python 3.9). * The node deletion capability of the yaml-set command is now part of the library. See Processor::delete_nodes(...) and Processor::delete_gathered_nodes(...) for details. * The node aliasing capability of the yaml-set command is now part of the library. See Processor::alias_nodes(...) and Processor::alias_gathered_nodes(...) for details. * The node tagging capability of the yaml-set command is now part of the library. See Processor::tag_nodes(...) and Processor::tag_gathered_nodes(...) for details. * The library now supports loading YAML from String rather than only from file. Simply pass a new `literal=True` keyword parameter to Parsers::get_yaml_data(...) or Parsers::get_yaml_multidoc_data(...) to indicate that `source` is literal serialized (String) YAML data rather than a file-spec. This mode is implied when reading from STDIN (source is "-"). * The emitter_write_folded_fix.py patch file for ruamel.yaml has been removed in favor of an author-supplied solution to the problem -- https://sourceforge.net/p/ruamel-yaml/tickets/383/ -- for which the patch was originally written. Known Issues: * ruamel.yaml version 0.17.x is a major refactoring effort by the project's owner. As such, only select versions will be marked as compatible with yamlpath. Such marking occurs in this project's dependencies list via the setup.py file. This is necessary because I use yamlpath in production environments where stability is paramount; I need the freedom to update yamlpath at-will without incurring any unexpected failures due to incompatible ruamel.yaml changes. I will try to test some -- but not all -- ruamel.yaml releases from time to time and update yamlpath dependency compatibilities accordingly. * ruamel.yaml version 0.17.4 somewhat resolves a previously reported issue -- https://sourceforge.net/p/ruamel-yaml/tickets/351/ -- wherein certain arrangements of comments or new-lines within YAML files near aliased hash keys would cause a total loss of data when the stream was written to file. Now, the data is no longer entirely lost. However, the preceding comment or new-line is deleted when the stream is written to file. This is deemed to be an acceptable compromise, for now, because the alternative is to either lose the entire document or lose all attempted changes to the affected document. Until the issue is properly fixed, an XFAIL test will continue to be in the yamlpath unit test suite. 3.4.0: Bug Fixes: * For the yaml-diff command-line tool, custom identity keys for specific records in AoHs of the RHS document were not being considered for comparison. Any identity key set for the whole AoH was being applied to every record in the set. Enhancements: * The yaml-get command-line tool -- and the underlying Processor::get_nodes method -- now retrives nodes with `null` values. Non-JSON null results from yaml-get are printed using the ASCII NULL control-character (Hexadecimal 00) wherever encountered. While most terminals and shells won't visibly print this character, it is there and can be picked up by downstream parsers of STDOUT. When the output is rendered as JSON, the unquoted "null" identifier is used. * The yaml-set command-line tool now enables assigning null values using a new --null|-N input option. * The yaml-set command-line tool now supports assigning YAML Aliases to target --change|-g nodes, referencing any other node via --aliasof|-A whether it is already Anchored, or not. Should the Anchor not already exist, a unique name will be automatically assigned. This automatic name can controlled via --anchor|-H. Setting --change|-g and --aliasof|-A to the same node along with a new --anchor|-H explicitly renames an existing Anchor and its Aliases. The same is implicitly possible by specifying --change|-g and --anchor|-H without --aliasof|-A. Using these against non-YAML files merely duplicates the target value to the indicated --change|-g nodes. * With a new --tag|-T auxilliary option, the yaml-set command-line tool now supports assigning custom YAML tags (data-type specifiers) to nodes created or updated during the operation. * The yaml-merge and yaml-diff command-line tools now also handle YAML tags. * The single-star wildcard segment translation was changed from [.!=""] to [.=~/.*/] which enables it to match any key or value (including empty-string and null). API Changes: * The yamlpath.func library is too big and some very useful general code has become attached to special-purpose classes. As such, it's time for refactoring before this becomes untennable. New, better-organized static classes were created under a new yamlpath.common namespace. Since this would be destructive to other Python code which depends on the old organization, this is a heads-up. Your existing code will continue to work without any changes, but only until 4.x (which is not yet planned). Until then, you must update your own code to use the new static classes for the following functions and methods so as to avoid fatal errors down the road: - yamlpath.func.append_list_element -> yamlpath.common.Nodes.append_list_element - yamlpath.func.build_next_node -> yamlpath.common.Nodes.build_next_node - yamlpath.func.clone_node -> yamlpath.common.Nodes.clone_node - yamlpath.func.create_searchterms_from_pathattributes -> yamlpath.common.Searches.create_searchterms_from_pathattributes - yamlpath.func.ensure_escaped -> yamlpath.YAMLPath.ensure_escaped - yamlpath.func.escape_path_section -> yamlpath.YAMLPath.escape_path_section - yamlpath.func.get_node_anchor -> yamlpath.common.Anchors.get_node_anchor - yamlpath.func.get_yaml_data -> yamlpath.common.Parsers.get_yaml_data - yamlpath.func.get_yaml_editor -> yamlpath.common.Parsers.get_yaml_editor - yamlpath.func.get_yaml_multidoc_data -> yamlpath.common.Parsers.get_yaml_multidoc_data - yamlpath.func.make_float_node -> yamlpath.common.Nodes.make_float_node - yamlpath.func.make_new_node -> yamlpath.common.Nodes.make_new_node - yamlpath.func.search_anchor -> yamlpath.common.Searches.search_anchor - yamlpath.func.search_matches -> yamlpath.common.Searches.search_matches - yamlpath.func.stringify_dates -> yamlpath.common.Parsers.stringify_dates NOTE that this method is deprecated and will be eliminated in favor of using its more comprehensive replacement, yamlpath.common.Parsers.jsonify_yaml_data - yamlpath.func.unwrap_node_coords -> yamlpath.wrappers.nodecoords.NodeCoords.unwrap_node_coords - yamlpath.func.wrap_type -> yamlpath.common.Nodes.wrap_type - yamlpath.merger.Merger.combine_merge_anchors -> yamlpath.common.Anchors.combine_merge_anchors - yamlpath.merger.Merger.delete_all_comments -> yamlpath.common.Anchors.delete_all_comments - yamlpath.merger.Merger.rename_anchor -> yamlpath.common.Anchors.rename_anchor - yamlpath.merger.Merger.replace_anchor -> yamlpath.common.Anchors.replace_anchor - yamlpath.merger.Merger.scan_for_anchors -> yamlpath.common.Anchors.scan_for_anchors - yamlpath.merger.Merger.set_flow_style -> yamlpath.common.Anchors.set_flow_style Until you update your code, a deprecation warning will be printed to STDERR every time the yamlpath.func file is imported and the first time one of the relocated Merger static methods is called. To be rid of the message, update your code to use the new sources of the deprecated functions/methods and remove every import of and from yamlpath.func. 3.3.0: Bug Fixes: * It was impossible to install yamlpath 3.x without first installing ruamel.yaml via pip for Python 3.x. Not only has this been fixed but explicit tests have been created to ensure this never happens again. Enhancements: * A new command-line tool, yaml-diff, now compares exactly two YAML/JSON/Compatible documents, producing a GNU diff-like report of any differences in the data they present to parsers. Along with diff's "a" (added), "c" (changed), and "d" (deleted) report entries, affected YAML Paths are printed in lieu of line numbers. Further, a report entry of "s" (same) is available and can be enabled via command-line options. This tool also features optional special handling of Arrays and Arrays-of-Hashes, which can be configured as CLI options or via an INI file for distinct settings per YAML Path. See --help or the Wiki for more detail. API Changes: * NodeCoords now employ a new `path` attribute. This is an optional parameter which is assigned during construction to later report the translated origin YAML Path; this is where the node was found or created within the DOM. Note that Collector segments work against virtual DOMs, so the YAML Path of an outer Collector will be virtual, relative to its parent at construction; when nested, this will be a bare list index. Any NodeCoords in the virtual container which point to real nodes in the DOM will have their own concrete YAML Paths. * YAMLPath instances now support nonmutating addition of individual segments via the + operator. Whereas the append() method mutates the YAMLPath being acted upon, + creates a new YAMLPath that is the original plus the new segment. In both cases, the orignal YAMLPath's separator is retained during both operations. As with .append(), new segments added via + must also be properly escaped -- typically via path.escape_path_section -- before being added. 3.2.0: Enhancements: * Expanded YAML Path Search Expressions such that the OPERAND of a Search Expression may be a sub-YAML Path. This enables searching descendent nodes -- without moving the document pointer -- to yield ancestors with matching descendants. This has more utility when searching against Arrays-of-Hashes. Bug Fixes: * Date values in YAML could not be written to JSON streams; the JSON renderer would generate an incompatibility error. Now, dates are written as Strings to JSON. This affected: yaml-get, yaml-set (in stream mode), yaml-merge, and yaml-paths. 3.1.0: Enhancements: * yaml-set can now delete nodes when using --delete rather than other input arguments. * A new command-line tool has been created, yaml-validate, which validates YAML/JSON/compatible single- and multi-documents (files or STDIN). Bug Fixes: * The yaml-merge command-line tool wasn't allowing Scalar values supplied via STDIN -- with no structure, just bare Scalar values -- to be appended to exising Array data structures; rather, it was wholly overwriting the destination, deleting all pre-exisiting elements. * The yaml-merge command-line tool wasn't accepting empty-strings as STDIN documents; it was reporting a document-read error, instead. This turns out to be useful when you want to use yaml-merge instead of yaml-set to deliberately write an empty-string value at some --mergeat location within the LHS document. * The yaml-merge command would not accept any variation of "false" as a Scalar value input; it was instead reporting a document-read error. This turns out to be useful when using yaml-merge as if it were yaml-set to write a false Boolean value to the LHS document at some --mergeat location. API Changes: * The func.get_yaml_data and func.get_yaml_multidoc_data functions now return/yield tuples. The first field is the desired yaml_data (can be None for empty documents) and the second field is a Boolean which indicates True when the document loaded with no errors or False when an error occurred. This is necessary in order to accept a bare "false" Scalar value as a STDIN-supplied document. 3.0.0: Enhancements: * Added a new YAML Path Segment Type: * This is identical to a Search segment where the search term is `[.!=""]`. This translates to "match every Hash key for which its name is not empty and every Array element which is not empty". This operator also vertically expands results from Collectors, effectively breaking them out from an Array- per-line to one-Scalar-per-line. If you place this inside a Collector, the results will still be collected into an Array. * The * character now also serves as a wildcard character for key-names, Hash values, and Array value comparisons, converting the segment to a Search. For example, a YAML Path like `abc.d*` becomes `abc[.^d]`, `abc.*f` becomes `abc[.$f]`, and `abc.*e*` becomes `abc[.=~/^.*e.*$/]`, and so on. * Added a new YAML Path Segment Type: ** This new type is a "Traversal" segment which causes YAML Path operations to deeply traverse the document from that point. When there are no further segments in the YAML Path, every leaf node (Scalar value) is matched. When the YAML Path has at least one further segment, it (and all further segments) must match subsequent nodes (anywhere deeper than that point in the document) or none are matched. Results can be collected. * The yaml-merge and yaml-get command-line tools now treat the - pseudo-file as implicit when NOT specified AND the session is non-TTY. This can be blocked with --nostdin|-S. This enables, for example, piping into these commands without being forced to specify the - pseudo-file as an argument to them. * The yaml-merge command now enables users to force the merged document to be written out as YAML or JSON via a new --document-format (-D) command-line argument. When unset, the format will be based on the file-name extension of the --output file when provided, or (last-resort) that of the first document (AUTO). * The yaml-merge command now accepts multi-document YAML files, created when the YAML standard-specified End-of-Document, Start-of-Document marker pair (... followed by ---) is present, like: ```yaml --- document: 1 ... --- document: 2 ``` * The yaml-merge command now accepts multi-document JSON files, created when there are multiple root-level entities, like: ```json {"document": 1} {"document": 2} ``` * Because any document to yaml-merge can be a multi-document, it no longer requires at least 2 YAML_FILEs be supplied on the command-line. If users pass only a single file or stream that is not a multi-document file, its content will merely be written out without any merging into it. This can be useful for trivially converting any file from YAML to JSON or JSON to YAML, like `yaml-merge --document-format=json file.yaml` or `yaml-merge --document-format=yaml file.json`. * The `yaml-set` command-line tool can now write changes to empty or minimally- viable files, enabling users to build up new data files from scratch. The file must already exist, even if completely empty. A non-empty, minimally- viable file depends on document type. For example: A minimally-viable YAML file: ```yaml --- # The triple-dash is required. ``` Two versions of a minimally-viable JSON file: ```json {} ``` or: ```json [] ``` However, minimally-viable structure is necessary only for files with unusual file-name extensions. When the file-name extension is one of yaml, yml, or json (case-insensitive), the file can be completely empty and still result in a YAML or JSON data structure. * The `yaml-set` command-line tool now accepts empty-String values. * The `yaml-merge` command-line tool now permits overwriting one of its input files as long as `--overwrite` is used instead of `--output`; these are mutually-exclusive options. To help users protect against accidental change, a new `--backup` flag will cause the to-be-overwritten file to be renamed with a ".bak" file-name extension. A pre-existing backup file with the same name will be unceremoniously replaced. * The `yaml-set` command-line tool now accepts an arbitrary set of characters from which to derive `--random` values via a new `--random-from` argument. This is especially useful when you need to limit or expand the characters used and when you wish to favor some characters more than others (simply repeat the favored characters more than other characters in the argument value but do so under caution because doing so reduces randomness). * The `yaml-set` command-line tool now accepts documents from STDIN, causing it to write the resulting changes to STDOUT. This enables `yaml-set` to operate as a stream editor, like `yaml-get` and `yaml-merge`. * The `yaml-paths` command-line tool now accepts documents from STDIN. It also now accepts multi-document YAML and JSON as file or STDIN input. Because it is impossible to determine whether a file or stream contains multi-document data without reading through the entire file more than once, output now always displays the file-name (or STDIN) and -- new -- the document-index in which matches were found. As before, users can turn off file-name display by setting --nofile|-F. In previous, single-document versions, the file-name display was automatically muted when there was only one YAML_FILE to process. Bug Fixes: * Collectors were breaking search nodes with Regular Expressions, making it impossible for collected searches to return expected matches. * Fixed the Known Issue which was logged at version 2.4.0; setting values which override aliased key-value pairs now correctly adds the new key-value pair to the DOM. * When the left-most document was JSON, yaml-merge and yaml-set would both improperly write out a YAML document start mark (---) and then a hybrid JSON/YAML result rather than valid JSON. * The yaml-merge command would "explode" LHS Anchored Hashes wherever they were aliased when the RHS document modified the same Hash into which the alias was used. * Setting a Python-style Boolean value via `yaml-set` (True or False) without setting --format=boolean would cause an error because ruamel.yaml was expecting an integer, instead. It is no longer necessary to set --format in this case. However, --format=boolean can still be useful to convert more "Boolean like" values into true|false, like on, off, yes, no, true, false, True, False, 1, 0. Non-Breaking API Changes: * The various protected _get_* methods of Processor were changed to reduce the number of positional parameters while also allowing for new special-use parameters for future changes. The formerly optional positional `parent` and `parentref` parameters are now optional keyword arguments by the same names. Because these were all protected methods and the affected parameters were optional anyway, this is not deemed a breaking change; no one should have been directly calling them. * The get_yaml_editor function now supports several keyword arguments which provide for some customization of the returned ruamel.yaml.YAML instance. See its documentation for details. This is a non-breaking change as the defaults for each new keyword argument set the behavior identical to what it was before this change. * The get_yaml_data function now returns False rather than None when there is an issue attempting to load data. This is because an empty-but-viable document correctly returns None but there is no valid YAML or JSON document which can be comprised only of a Scalar Boolean. This is a non-breaking change because None and False are equivalent for code like: ```python data = get_yaml_data(get_yaml_editor(), ConsoleLogger(), "file.yaml") if not data: print("No data") ``` However, you can now differentiate between "No data" and "Invalid document" like so: ```python data = get_yaml_data(get_yaml_editor(), ConsoleLogger(), "file.yaml") if data is None: print("No data") elif not data and isinstance(data, bool): print("Invalid document") else: print("Got a non-empty document") ``` * The ConsolePrinter's debug method now prints vastly more detail and allows for customization of the output. Read its documentation for details. From this release forward, the version reported by all command-line tools is synchronized with the version of the overall yamlpath installation. 2.4.3: Bug Fixes: * Array-of-Hashes were not being detected for the purpose of applying merge rules defined via the INI-style configuration file. * Array-of-Hashes identity key inference was looking first to the LHS document. This was backwards for an RHS-to-LHS merge and has been corrected. The yaml-merge command now reports version 0.0.4 to reflect these changes. 2.4.2: Enhancements: * In the INI file's [rules] section, different merge rules can now be applied to specific parts -- no matter how deeply nested -- of the same Hash structure. Bug Fixes: * The 3rd-party Python INI file parser had no way of differentiating between the key and value of a YAML Path entry containing an = sign in its key, like "/path[.=name]/key = left". This update reconstitutes such lines and correctly parses an affected YAML Path from the merge rule. The yaml-merge command now reports version 0.0.3 to reflect these changes. 2.4.1: Bug Fixes: * The yaml-merge tool (and underlying Merger class) incorrectly assigned "None" Anchors to all floating-point numbers. This prevented all merging when both the LHS and RHS documents contained at least one floating-point number, each. The yaml-merge command now reports version 0.0.2 to reflect this change. 2.4.0: Enhancements: * Added new reference command-line tool: yaml-merge. This is a very complex tool, so a comprehensive treatise will be added to the project Wiki to explore its capabilities. Along with those of its component classes, its unit tests also provide many examples of the same. * YAMLPath instances now support arbitrary changes to separator. * YAMLPath instances now support equality testing (against the stored path), immune to differences in separator. * The get_yaml_data function now supports "-" as a source file. This is interpreted as a read from STDIN. * Due to the change to the get_yaml_data function, the yaml-get reference command-line tool now supports retrieving nodes from YAML/Compatible data passed to it via STDIN when its YAML_FILE argument is set to -. The new yaml-merge reference command-line tool also reads from STDIN when one of its YAML_FILE input arguments is -. No other reference command-line tools support this change at this time. Known Issues: 1. Neither yaml-set nor yaml-merge will add override keys to a Hash which uses the YAML merge operator (<<:) and which does not already have a matching override key. This issue has existed for a very long time but was only discovered during preparation for this release. This will be logged and tracked as a Known Issue for this release -- to be fixed at another time -- because no one (not even myself) has yet encountered/reported this issue, it is non-trivial to fix, and it is an edge-case. Here is an example of this issue: For ex.yaml: --- anchored_hash: &its_anchor ah_key: Base value merging_hash: <<: *its_anchor mh_key: Implementation value ... both of these commands: `yaml-set --change=/merging_hash/ah_key --value='Override value' ex.yaml` `echo 'Override value' | yaml-merge -m /merging_hash/ah_key ex.yaml -` ... will fail to affect the expected change. The expectation would be: --- anchored_hash: &its_anchor ah_key: Base value merging_hash: <<: *its_anchor mh_key: Implementation value ah_key: Override value ... but the actual result is (without any indication of an error): --- anchored_hash: &its_anchor ah_key: Base value merging_hash: <<: *its_anchor mh_key: Implementation value 2.3.7: Bug Fixes: * Setting negative floats could cause the leading "-" symbol to be replaced with an unexpcted "0" when specifying a float format, or crash when using the default format. 2.3.6: Bug Fixes: * When using yaml-set with --format=folded and --eyamlcrypt, the encrypted value was being mistakenly appended with a spurious newline character at its end. Note that this affected only Puppet's Hiera lookup and not EYAML itself; on the command-line, the eyaml command would not show the extra newline character. The yaml-get command was also not affected, making it very difficult to set up a unit-test for this case. 2.3.5: Bug Fixes: * Certain YAML constructs trigger AssertionErrors in ruamel.yaml during YAML data writes. This was causing yaml-set to generate empty files. Until https://sourceforge.net/p/ruamel-yaml/tickets/351/ is fixed, this patch will revert the file contents to mitigate data loss under these conditions. A specific test has been created to detect when the upstream issue is fixed. 2.3.4: Bug Fixes: * Minor security patch: Python already makes non-shell subprocess calls safe (mitigating shell command injection). This patch makes that already-present protection explicit rather than implicit. 2.3.3: Bug Fixes: * Subtraction Collector math had no effect when the RHS was a list of scalar values (because LHS was a list of NodeCoords, so comparison was always false). Also reduced O(3N) to O(2N) during Collector subtraction. Enhancements: * The console logger's debug method now includes the type of each element in a list while it is being dumped. 2.3.2: Bug Fixes: * Subtraction Collector math crashed when the RHS result was non-iterable. 2.3.1: Bug Fixes: * Under certain conditions, some YAML changes were affecting unexpected nodes which had identical original values. * YAMLValueFormats.DEFAULT was identifying all CLI-supplied values as String rather than their most-likely data-type. API COMPATIBILITY WARNING: Previous versions of the library returned only selected data nodes. Now, instead of data nodes, NodeCoords objects are returned. This is necessary in order to resolve Issue #44, which was caused by Python over-aggressively optimizing memory, making non-String data nodes references to any other node with the same value. This made changing exactly one of those data nodes impossible because all nodes with the same original value would change with it. Now, the NodeCoords class carries DOM tracking data along with each data node making precise node changes possible. However, your queries will now return this more complex additional data. In order to evaluate just the data, please refer to this before-and-after example: BEFORE: for node in processor.get_values(yaml_path): do_something_with(node) AFTER: for node_coordinate in processor.get_values(yaml_path): do_something_with(node_coordinate.node) If you need to recursively remove DOM tracking data from the results, a new utility function is available: func.unwrap_node_coords(data). Note however that you need that tracking data in order to change data within the DOM. This does not affect the output of the sample command-line utilities. 2.3.0: Bug Fixes: * The get_yaml_data helper function now contains ruamel.yaml errors/warnings without disrupting calling context handlers. Enhancements: * yaml-paths version 0.2.0 now has more detailed output control, trading --pathonly for --values, --nofile, --noexpression, and --noyamlpath. 2.2.0: Bug Fixes: * YAML construction errors are now caught and more cleanly reported by all command-line tools. Enhancements: * yaml-paths version 0.1.0 now has more specific flags, enabling: * more precise handling of anchors in YAML data, and * expanding parent node results to instead return all their child leaf nodes. 2.1.1: Bug Fixes: * yaml-paths was recursing into nodes for which the name had already matched, causing unnecessary search results. Version 0.0.2 fixes this; when a node is matched by name, any children are ignored because they will have already been yielded as the parent node's value. 2.1.0: Enhancements: * Added a new yaml-paths command-line tool. In short, it enables searching YAML/Compatible files, returning YAML Paths for any matches. As an Alpha- grade tool, it is being released at version 0.0.1. Feedback welcome! * All command-line tools which accept --pathsep now accept symbolic separators rather than only names; so, --pathsep=/ is idental to --pathsep=fslash, etc. Minor changes were also made to all command-line tools to consolidate some repeat code. Each has a version bump to reflect this minor refactoring effort. 2.0.2: Bug Fixes: * eyaml-rotate-keys was broken by the refactoring for 2.0.0. eyaml-rotate-keys v1.0.2 restores functionality. Enhancements: * Command-line tools are now managed via pip as entry_points/console_scripts rather than external binaries. This enables superior cross-platform compatibility as well as unit testing. As such, all of the CLI tools have been updated pursuant to (generally trivial, excepting eyaml-rotate-keys) issues discovered during their newfound CI tests. 2.0.1: Bug Fixes: * yaml-set v1.0.4 lost track of EYAML block formatting between read and write, causing replacement values to use unexpected formatting. This is fixed in yaml-set v.1.0.5. 2.0.0: Enhancements: * Added Collectors to YAML Path expressions. These take the form of "(YAML Path)" -- parenthesis () are used to demarcate each Collector -- resulting in a list of zero or more matches for the sub-query. Operators between Collectors include + and -, like "(...)+(...)", "(...)-(...)", and nested Collectors are possible, like "(...)-((...)+(...))". Collectors may appear anywhere within the outer YAML Path, effectively setting the point within the data at which each Collector is rooted. * A major code refactoring was undertaken to break YAMLPath out as its own class and improve code quality (per mypy and pylint). Bug Fixes: * yaml-set v1.0.4 now implies --mustexist when --saveto is set. Otherwise, --saveto was trying to save nothing when --change (without --mustexist) pointed nowhere. 1.2.5: Bug Fixes: * yaml-set v1.0.3 no longer requires --privatekey unless decryption is requested via enabling --check. As a side-effect, the script will also no longer ignore requests to set the same value as was already set. 1.2.4: Bug Fixes: * yaml-set v1.0.2 now preserves newlines for pre-folded EYAML values when saving the old encrypted value to another node. * ruamel.yaml v0.15.96 is now the minimum acceptable version in order to adopt better round-trip editing. 1.2.3 Enhancements: * Set minimum compatible versions of Python and ruamel.yaml in order to ensure users experience the expected behaviors. 1.2.2 Enhancements: * Array element selection can now be specified by either the usual [#] notation or a bare integer. Thus, the following are all identical: array[1] array.1 /array/1 1.2.1 Enhancements: * Some exception/error messages have been updated to print the entire original -- albeit parsed -- YAML Path in addition to the present segment under evaluation. Bug Fixes: * yaml-get version 1.0.2 now converts new-lines into "\n" character sequences when writing output so that multi-line values remain one-result-per-line. * Use of escape symbols for unusual characters (where demarcation would usually be more intuitive) is now preserved. Thus, these two search phrases are now identical: array[.%" can't "] array[.%\ can\'t\ ] * The issue preventing some YAML Paths from being printable after parsing has been fixed. Valid, parsed YAML Paths now correctly print into a re-parsable form even with weird sequences and escapes. Note that superfluous whitespace and other symbols are still removed or escaped when the YAML Path is printed, so: term [ key == "Superfluous spaces aren\'t kept." ] correctly parses and prints as: term[key=Superfluous\ spaces\ aren\'t\ kept.] 1.2.0 Enhancements: * A new search operator, :, now enables capturing slices of Arrays (by 0-based element number) and Hashes (by alphanumeric key-name). This looks like: "some::array[2:15]" or "some::hash[beta:gamma]". * yaml-get now returns JSON instead of "pretty Python" data objects when the search returns complex data types (Arrays and Hashes). This change makes the result more portable to non-Python consumers and ensures the result will be one per line. * The separator used for identifying Hash sub-keys can now be customized. If you prefer your paths to look like "/hash/sub/key" rather than "hash.sub.key", you can now have it your way. For now, only . and / are allowed. The separator can be either strictly specified or automatically inferred by whether the first character of a given YAML Path is /. Command-line tools like yaml-get and yaml-set have a new --pathsep argument for this; the default is "auto" and can be set to "fslash" (/) or "dot" (.). Bug Fixes: * EYAML on Windows now works when a batch file is used to wrap the Ruby `eyaml` command. Known Issues: * Escape symbols in YAML Paths parse correctly and will be properly processed, resulting in retriving or setting the expected data. However, the parsed path cannot be stringified back to its original form (with escape symbols). This issue affects only logging/printing of the post-parsed path. A unit test has been created to track this issue, but it is marked xfail until such time as someone is willing to tackle this (very) low priority issue. Until then, developers should try to print the pre-parsed version of their paths rather than rely exclusively on Parser.str_path(). Further, don't do this: 1. Accept or take a string path that has escaped characters. 2. Parse that path. 3. Stringify the parsed path. 4. Parse the stringified, parsed path. This is silly, anyway because you already have the first (good) parsed result at step 2. 5. Try to use this parsed-stringified-parsed path result for anything. Instead, only use the first parsed result that you got at step 2. 1.1.2 Bug fixes: * When the YAML Path is fully quoted -- a known side-effect of using Jenkins and trying to responsibly quote any part of an argument to a shell command -- the visual feedback failed to show the user that the parsed version of the YAML Path was (correctly) treated as one whole KEY. While this is not what most users should expect, it is correct because YAML Path cannot safely make any assumptions as to whether the quoting was deliberate or unintentional. Now, the stringified version of affected YAML Paths very clearly injects escape symbols for otherwise intentional special symbols, clearly alerting the user that a probable issue is afoot. * When instructed to `--save` old values, the yaml-set binary was saving only decrypted versions of the original values. It now preserves the original encrypted form while still performing other requested or necessary tests against the decrypted value. 1.1.1 Bug fixes: * The . search operand was limited to only Hash key-names; it now also enables searching Array elements. 1.1.0 Added support for Regular Expressions as a search mechanism against Hash keys and values. Also increased unit test coverage and fixed some bugs in that effort. 1.0.0 Initial release. All features are belived to be working as expected, backed by comprehensive unit tests and my own in-production use of the bundled command-line tools. Of course, not all edge-cases can be predicted, so if you manage to find a bug, please report it! Feature requests are always welcome, as well. That said, Pull Requests are always the best way to contribute!