# 1.6.0 (2026-05-17) ## πŸ’« New features - `iter_gitdiff`: - add support for pathspec-constrained reporting [[76d8db7f]](https://github.com/datalad/datalad-next/commit/76d8db7f) - atomic: - describe atomicity property [[f07a613b]](https://github.com/datalad/datalad-next/commit/f07a613b) - atomic upload for `url_operations.file` [[4a8aa14f]](https://github.com/datalad/datalad-next/commit/4a8aa14f) - atomic upload for url_operations.ssh [[116ac16f]](https://github.com/datalad/datalad-next/commit/116ac16f) - delete: - add delete to HttpUrlOperations [[dfbf1e50]](https://github.com/datalad/datalad-next/commit/dfbf1e50) - add delete to `SshUrlOperations` [[fb4aeec8]](https://github.com/datalad/datalad-next/commit/fb4aeec8) ## πŸ› Bug Fixes - guard handle_401 against IndexError when token meets Basic-only server [[d947636b]](https://github.com/datalad/datalad-next/commit/d947636b) - include ZipArchiveOperations in Archivist's handlers [[54d783f8]](https://github.com/datalad/datalad-next/commit/54d783f8) - proper import of version information to the top-level module [[ed1ca140]](https://github.com/datalad/datalad-next/commit/ed1ca140) - delay import to have import subject be impacted by patches [[5ee1a7b8]](https://github.com/datalad/datalad-next/commit/5ee1a7b8) - `CreateSiblingRia`: - canonify push-url correctly [[4f1f578b]](https://github.com/datalad/datalad-next/commit/4f1f578b) - `decode_bytes`: - error handling led to data loss in subsequent chunks [[dfa1d20b]](https://github.com/datalad/datalad-next/commit/dfa1d20b) - handle multiple errors [[7a73d8ce]](https://github.com/datalad/datalad-next/commit/7a73d8ce) - always backslashreplace when asked to [[0ac7974a]](https://github.com/datalad/datalad-next/commit/0ac7974a) - `gitworktree`: - decode bytes before itemization [[5fe82ead]](https://github.com/datalad/datalad-next/commit/5fe82ead) - `iter_gitdiff`: - adjust conditional acknowledging that `GitTreeItem.name` is a `str` [[21db7511]](https://github.com/datalad/datalad-next/commit/21db7511) - enable multi-level submodule recursion [[94bd3c50]](https://github.com/datalad/datalad-next/commit/94bd3c50) - yield copy of container item when it is also further processed [[78d71e82]](https://github.com/datalad/datalad-next/commit/78d71e82) - correct `git diff-index|tree` command for submodule recursion [[c37dd315]](https://github.com/datalad/datalad-next/commit/c37dd315) - corrected type annotation in internal helper [[0eb4ffb4]](https://github.com/datalad/datalad-next/commit/0eb4ffb4) - decode bytes before itemization [[8b188142]](https://github.com/datalad/datalad-next/commit/8b188142) - `iter_gittree`: - decode bytes before itemization [[c8598545]](https://github.com/datalad/datalad-next/commit/c8598545) - delete: - unify semantics of delete-operations [[80bc38c3]](https://github.com/datalad/datalad-next/commit/80bc38c3) - set write-permission in `SshUrlOperations.delete` [[86c728fd]](https://github.com/datalad/datalad-next/commit/86c728fd) - tests: - bring back datalad core wtf-test [[6d2d44a5]](https://github.com/datalad/datalad-next/commit/6d2d44a5) - remove print-statement from test [[67acb453]](https://github.com/datalad/datalad-next/commit/67acb453) - remove align_pattern performance test [[3642879a]](https://github.com/datalad/datalad-next/commit/3642879a) - types: - add type for `prop` [[f96dd3cf]](https://github.com/datalad/datalad-next/commit/f96dd3cf) - fix exception arguments [[7fa458d3]](https://github.com/datalad/datalad-next/commit/7fa458d3) - fix type annotations in alig_pattern [[247ee261]](https://github.com/datalad/datalad-next/commit/247ee261) - uncurl: - add close-handler to uncurl [[c9b2a4c8]](https://github.com/datalad/datalad-next/commit/c9b2a4c8) ## 🏠 Refactorings - import `PRE_INIT_COMMIT_SHA` from `datalad_core` [[43c72541]](https://github.com/datalad/datalad-next/commit/43c72541) - homogenize deprecation pattern for `datalad_next.iterable_subprocess` [[cc846f5b]](https://github.com/datalad/datalad-next/commit/cc846f5b) - homogenize deprecation pattern for `datalad_next.iterools` [[7bd83268]](https://github.com/datalad/datalad-next/commit/7bd83268) - use `gitpathspec` module from `datasalad` library [[337187ff]](https://github.com/datalad/datalad-next/commit/337187ff) - `iter_gitdiff`: - establish new top-level internal helper [[f922c7bd]](https://github.com/datalad/datalad-next/commit/f922c7bd) - new dedicated helper to yield from a submodule [[fbb0318e]](https://github.com/datalad/datalad-next/commit/fbb0318e) - discontinue use of `kwargs` dict [[232dba4b]](https://github.com/datalad/datalad-next/commit/232dba4b) - flip conditional to simplify internal helper [[39505865]](https://github.com/datalad/datalad-next/commit/39505865) - dedicated function for Git output -> `GitDiffItem` [[a513b8c8]](https://github.com/datalad/datalad-next/commit/a513b8c8) - dissolve kwarg-dict to improve type annotation [[b88e21f3]](https://github.com/datalad/datalad-next/commit/b88e21f3) - `iter_gitstatus`: - reduce variable recycling to avoid confusing type-checker [[2aa77052]](https://github.com/datalad/datalad-next/commit/2aa77052) - `iterable_subprocess`: - removed package and use from `datasalad` [[4ee9c272]](https://github.com/datalad/datalad-next/commit/4ee9c272) - `itertools`: - removed package and use from `datasalad` [[a2a992d5]](https://github.com/datalad/datalad-next/commit/a2a992d5) ## πŸ“ Documentation - add maintainability badge [[abd0e4b7]](https://github.com/datalad/datalad-next/commit/abd0e4b7) - add `datalad_next.shell.shell()` to docs [[e49ca134]](https://github.com/datalad/datalad-next/commit/e49ca134) - Add official Hatch badge to README.md [[18fa8c63]](https://github.com/datalad/datalad-next/commit/18fa8c63) - trim module API documentation to an overview table of content [[704aeedf]](https://github.com/datalad/datalad-next/commit/704aeedf) - (temporarily) disable the inclusion of manpages in the docs [[e3e588af]](https://github.com/datalad/datalad-next/commit/e3e588af) - remove unused intersphinx extension [[761f232f]](https://github.com/datalad/datalad-next/commit/761f232f) - correct spelling in doc-strings [[6511cbd9]](https://github.com/datalad/datalad-next/commit/6511cbd9) - exclude memory addresses from documentation [[0d2440de]](https://github.com/datalad/datalad-next/commit/0d2440de) - `iter_gitdiff`: - add docstring for `eval_submodule_state` argument [[e786921b]](https://github.com/datalad/datalad-next/commit/e786921b) - Contributing guide: - cheat sheet on `hatch` [[8f2f7d69]](https://github.com/datalad/datalad-next/commit/8f2f7d69) ## πŸ›‘ Tests - cover token auth with Basic-only challenge [[1e33929d]](https://github.com/datalad/datalad-next/commit/1e33929d) - enable zip testing [[2d2b1526]](https://github.com/datalad/datalad-next/commit/2d2b1526) - simplify hatch test setup [[e8bc0268]](https://github.com/datalad/datalad-next/commit/e8bc0268) - fix test_tree_with_broken_symlinks on Python 3.13 [[25ac6ef0]](https://github.com/datalad/datalad-next/commit/25ac6ef0) - fix ruff setup (used wrong path) [[865665ac]](https://github.com/datalad/datalad-next/commit/865665ac) - disable test with erratic behavior after hatch-based execution [[1c3f3130]](https://github.com/datalad/datalad-next/commit/1c3f3130) - disable test_wtf of datalad-core-4 [[4477de88]](https://github.com/datalad/datalad-next/commit/4477de88) - `iter_gitdiff`: - adjust type-annotation to match reality [[5cb96362]](https://github.com/datalad/datalad-next/commit/5cb96362) - new test to show that recursion is broken [[858036fa]](https://github.com/datalad/datalad-next/commit/858036fa) - type-annotate internal helper [[b4bec180]](https://github.com/datalad/datalad-next/commit/b4bec180) - `test_gitdiff`: - disable kwargs type-checking [[04d56150]](https://github.com/datalad/datalad-next/commit/04d56150) # 1.5.0 (2024-06-16) ## πŸ’« New features - new support subpackage for Git's pathspecs [[8d05f1cd]](https://github.com/datalad/datalad-next/commit/8d05f1cd) - add pathspec support to `iter_(gitworktree|submodules)()` [[7622267c]](https://github.com/datalad/datalad-next/commit/7622267c) - `GitWorktreeFileSystemItem`: - method to convert `GitWorktreeItem` [[30858780]](https://github.com/datalad/datalad-next/commit/30858780) - `iter_gitworktree`: - added submodule recursion with pathspec support [[2851ec4c]](https://github.com/datalad/datalad-next/commit/2851ec4c) - add ability to report untracked content only [[2763b10f]](https://github.com/datalad/datalad-next/commit/2763b10f) - `iter_submodules`: - add `match_containing` mode [[8025b7bd]](https://github.com/datalad/datalad-next/commit/8025b7bd) ## πŸ› Bug Fixes - `iter_gitstatus`: - support newly added submodules [[fd197ad6]](https://github.com/datalad/datalad-next/commit/fd197ad6) - `iter_gitstatus`: - rectify usage of unsupported `untracked=no` mode in tests [[673e6c22]](https://github.com/datalad/datalad-next/commit/673e6c22) - Patches for DataLad (core): - fix `create_sibling_ria` patching [[78e29b5d]](https://github.com/datalad/datalad-next/commit/78e29b5d) ## πŸ“ Documentation - Contributing guide: - expand description of expected commit messages [[04f68b49]](https://github.com/datalad/datalad-next/commit/04f68b49) # 1.4.1 (2024-05-22) ## πŸ› Bug Fixes - dependencies: limit test patch import to test runs [[905b99bd]](https://github.com/datalad/datalad-next/commit/905b99bd) ## πŸ“ Documentation - add note of Git >= v2.31 requirement for next-status [[093575d8]](https://github.com/datalad/datalad-next/commit/093575d8) - state conventional-commits requirement [[a9180fc0]](https://github.com/datalad/datalad-next/commit/a9180fc0) ## πŸ›‘ Tests - fixture: add missing import (for non-WebDAV fallback) [[ddd66799]](https://github.com/datalad/datalad-next/commit/ddd66799) - markers: extend list of implicit datalad-core markers [[59abbcfb]](https://github.com/datalad/datalad-next/commit/59abbcfb) - markers: add missing datalad-core marker "integration" [[e4e60b99]](https://github.com/datalad/datalad-next/commit/e4e60b99) # 1.4.0 (2024-05-17) ## πŸ› Bug Fixes - RIA over SSH access from Mac clients to Linux server was broken due to an inappropriate platform check that assumed that local and remote platform are identical. Fixes https://github.com/datalad/datalad/issues/7536 via https://github.com/datalad/datalad-next/pull/653 (by @mih) - `next-status` has received a number of fixes: - It no longer issues undesirable modification reports that are based on `mtime` changes alone (i.e., no content change). Fixes https://github.com/datalad/datalad-next/issues/639 via https://github.com/datalad/datalad-next/pull/650 (by @mih) - It now detects staged changes in repositories with no commit. Fixes https://github.com/datalad/datalad-next/issues/680 via https://github.com/datalad/datalad-next/pull/681 (by @mih) - `next-status -r mono` now reports on new commits in submodules. Previously this was ignored, leading to the impression of clean datasets despite unsaved changes. Fixes https://github.com/datalad/datalad-next/issues/645 via https://github.com/datalad/datalad-next/pull/679 (by @mih) - `iter_annexworktree()` can now also be used on plain Git repos, and would behave exactly as if reporting on non-annexed files in a git-annex repo. Previously, a cryptic `iterable did not yield matching item for route-in item, cardinality mismatch?` error was issued in this case. Fixes https://github.com/datalad/datalad-next/issues/670 via https://github.com/datalad/datalad-next/pull/673 (by @mih) ## πŸ’« Enhancements and new features - `datalad_next.shell` provides a context manager for (long-running) shell or interpreter subprocesses. Within the context any number of commands can be executed in such a shell, and each command can process input (iterables), and yield output (iterables). This feature is suitable for running and controlling "remote shells" like a login shell on a server via SSH. A range of utilities is provided to employ this functionality for special purpose implementations (e.g., accept fixed-length or variable-length process output). A suite of operations like download/upload file to a remote shell is provided for POSIX-compliant shells `datalad_next.shell.operations.posix`. https://github.com/datalad/datalad-next/pull/596 (by @christian-monch) - A rewrite of `SSHRemoteIO`, the RIA SSH-operations implementation from datalad-core is provided as a patch. It is based on the new `shell` feature, and provides more robust operations. It's IO performance is at the same level as `scp`-based down/uploads. In contrast to the original implementation, it support fine-grained progress reporting for uploads and downloads. Via https://github.com/datalad/datalad-next/pull/655 (by @mih) - The `SpecialRemote` base class in datalad-core is patched to support a standard `close()` method for implementing resource release and cleanup operations. The main special remote entry point has been altered to run implementations within a `closing()` context manager to guarantee execution of such handlers. Via https://github.com/datalad/datalad-next/pull/655 (by @mih) - A new `has_initialized_annex()` helper function is provided to test for a locally initialized annex in a repo. Via https://github.com/datalad/datalad-next/pull/673 (by @mih) - `iter_annexworktree()` can now also be used on plain Git repositories, and it yields the same output and behavior as running on a git-annex repository with no annex'ed content (just tracked with Git). Fixes https://github.com/datalad/datalad-next/issues/670 via https://github.com/datalad/datalad-next/pull/673 (by @mih) - `next-status` and `iter_gitstatus()` have been improved to report on further modifications after a file addition has been originally staged. Fixes https://github.com/datalad/datalad-next/issues/637 via https://github.com/datalad/datalad-next/pull/679 (by @mih) - `next-status` result rendering has been updated to be more markedly different than git-status's. Coloring is now exclusively determined by the nature of a change, rather than being partially similar to git-status's index-updated annotation. This reduces the chance for misinterpretations, and does not create an undesirable focus on the Git index (which is largely ignored by DataLad). Fixes https://github.com/datalad/datalad-next/issues/640 via https://github.com/datalad/datalad-next/pull/679 (by @mih) - A large 3k-line patch set replaces almost the entire RIA implementation, including the ORA special remote, and the `create-sibling-ria` command. The new implementation brings uniform support for Windows clients, progress reporting for uploads and downloads via SSH, and a faster and more robust behavior for SSH-based operations (based on the new remote shell feature). Fixes https://github.com/datalad/datalad-next/issues/654 via https://github.com/datalad/datalad-next/pull/669 (by @christian-monch) ## πŸ“ Documentation - Git-related subprocess execution helpers are now accessible in the rendered documentation, and all supported file collections are now mentioned in the `ls-file-collection` command help. Fixes https://github.com/datalad/datalad-next/issues/668 via https://github.com/datalad/datalad-next/pull/671 (by @mih) ## πŸ›‘ Tests - Test setup has been improved to support a uniform, datalad-next enabled environment for subprocesses too. This extends the scope of testing to special remote implementations and other code that is executed in subprocesses, and relies on runtime patches. See https://github.com/datalad/datalad-next/pull/i665 (by @mih) # 1.3.0 (2024-03-19) ## πŸ’« Enhancements and new features - Code organization is adjusted to clearly indicate what is part of the package's public Python API. Anything that can be imported directly from the top-level of any sub-package is part of the public API. As an example: `from datalad_next.runners import iter_git_subproc` imports a part of the public API, but `from datalad_next.runners.git import iter_git_subproc` does not. See `README.md` for more information. Fixes https://github.com/datalad/datalad-next/issues/613 via https://github.com/datalad/datalad-next/pull/615 (by @mih) https://github.com/datalad/datalad-next/pull/617 (by @mih) https://github.com/datalad/datalad-next/pull/618 (by @mih) https://github.com/datalad/datalad-next/pull/619 (by @mih) https://github.com/datalad/datalad-next/pull/620 (by @mih) https://github.com/datalad/datalad-next/pull/621 (by @mih) https://github.com/datalad/datalad-next/pull/622 (by @mih) https://github.com/datalad/datalad-next/pull/623 (by @mih) - New `patched_env` context manager for patching a process' environment. This avoids the for importing `unittest` outside test implementations. Via https://github.com/datalad/datalad-next/pull/633 (by @mih) - `call_git...()` functions received a new `force_c_locale` parameter. This can be set whenever Git output needs to be parsed to force running the command with `LC_ALL=C`. Such an environment manipulation is off by default and not done unconditionally to let localized messaging through in a user's normal locale. ## πŸ› Bug Fixes - `datalad-annex::` Git remote helper now tests for a repository deposit, and distinguishes an absent remote repository deposit vs cloning from an empty repository deposit. This rectifies confusing behavior (successful clones of empty repositories from broken URLs), but also fixes handling of subdataset clone candidate handling in `get` (which failed to skip inaccessible `datalad-annex::` URLs for the same reason). Fixes https://github.com/datalad/datalad-next/issues/636 via https://github.com/datalad/datalad-next/pull/638 (by @mih) ## πŸ“ Documentation - API docs have been updated to include all top-level symbols of any sub-package, or in other words: the public API. See https://github.com/datalad/datalad-next/pull/627 (by @mih) ## 🏠 Internal - The `tree` command no longer uses the `subdatasets` command for queries, but employs the recently introduced `iter_submodules()` for leaner operations. See https://github.com/datalad/datalad-next/pull/628 (by @mih) - `call_git...()` functions are established as the only used abstraction to interface with Git and git-annex commands outside the use in DataLad's `Repo` classes. Any usage of DataLad's traditional `Runner` functionality is discontinued. Fixes https://github.com/datalad/datalad-next/issues/541 via https://github.com/datalad/datalad-next/pull/632 (by @mih) - Type annotations have been added to the implementation of the `uncurl` git-annex remote. A number of unhandled conditions have been discovered and were rectified. # 1.2.0 (2024-02-02) ## πŸ› Bug Fixes - Fix an invalid escape sequence in a regex that caused a syntax warning. Fixes https://github.com/datalad/datalad-next/issues/602 via https://github.com/datalad/datalad-next/pull/603 (by @mih) ## πŸ’« Enhancements and new features - Speed up of status reports for repositories with many submodules. An early presence check for submodules skips unnecessary evaluation steps. Fixes https://github.com/datalad/datalad-next/issues/606 via https://github.com/datalad/datalad-next/pull/607 (by @mih) ## 🏠 Internal - Fix implementation error in `ParamDictator` class that caused a test failure. The class itself is unused and has been scheduled for removal. See https://github.com/datalad/datalad-next/issues/611 and https://github.com/datalad/datalad-next/pull/610 (by @christian-monch) ## πŸ›‘ Tests - Promote a previously internal fixture to provide a standard `modified_dataset` fixture. This fixture is sessions-scope, and yields a dataset with many facets of modification, suitable for testing change reporting. The fixture verifies that no modifications have been applied to the testbed. (by @mih) - `iterable_subprocess` tests have been robustified to better handle the observed diversity of execution environments. This addresseses, for example, https://bugs.debian.org/1061739. https://github.com/datalad/datalad-next/pull/614 (by @christian-monch) # 1.1.0 (2024-01-21) -- Iterate! ## πŸ’« Enhancements and new features - A new paradigm for subprocess execution is introduced. The main workhorse is `datalad_next.runners.iter_subproc`. This is a context manager that feeds input to subprocesses via iterables, and also exposes their output as an iterable. The implementation is based on https://github.com/uktrade/iterable-subprocess, and a copy of it is now included in the sources. It has been modified to work homogeneously on the Windows platform too. This new implementation is leaner and more performant. Benchmarks suggest that the execution of multi-step pipe connections of Git and git-annex commands is within 5% of the runtime of their direct shell-execution equivalent (outside Python). See https://github.com/datalad/datalad-next/pull/538 (by @mih), https://github.com/datalad/datalad-next/pull/547 (by @mih). With this change a number of additional features have been added, and internal improvements have been made. For example, any use of `ThreadedRunner` has been discontinued. See https://github.com/datalad/datalad-next/pull/539 (by @christian-monch), https://github.com/datalad/datalad-next/pull/545 (by @christian-monch), https://github.com/datalad/datalad-next/pull/550 (by @christian-monch), https://github.com/datalad/datalad-next/pull/573 (by @christian-monch) - A new `itertools` module was added. It provides implementations of iterators that can be used in conjunction with `iter_subproc` for standard tasks. This includes the itemization of output (e.g., line-by-line) across chunks of bytes read from a process (`itemize`), output decoding (`decode_bytes`), JSON-loading (`json_load`), and helpers to construct more complex data flows (`route_out`, `route_in`). - The `more_itertools` package has been added as a new dependency. It is used for `datalad-next` iterator implementations, but is also ideal for client code that employed this new functionality. - A new `iter_annexworktree()` provides the analog of `iter_gitworktree()` for git-annex repositories. - `iter_gitworktree()` has been reimplemented around `iter_subproc`. The performance is substantially improved. - `iter_gitworktree()` now also provides file pointers to symlinked content. Fixes https://github.com/datalad/datalad-next/issues/553 via https://github.com/datalad/datalad-next/pull/555 (by @mih) - `iter_gitworktree()` and `iter_annexworktree()` now support single directory (i.e., non-recursive) reporting too. See https://github.com/datalad/datalad-next/pull/552 - A new `iter_gittree()` that wraps `git ls-tree` for iterating over the content of a Git tree-ish. https://github.com/datalad/datalad-next/pull/580 (by @mih). - A new `iter_gitdiff()` wraps `git diff-tree|files` and provides a flexible basis for iteration over changesets. - `PathBasedItem`, a dataclass that is the bases for many item types yielded by iterators now more strictly separates `name` property from path semantics. The name is a plain string, and an additional, explicit `path` property provides it in the form of a `Path`. This simplifies code (the `_ZipFileDirPath` utility class became obsolete and was removed), and improve performance. Fixes https://github.com/datalad/datalad-next/issues/554 and https://github.com/datalad/datalad-next/issues/581 via https://github.com/datalad/datalad-next/pull/583 (by @mih) - A collection of helpers for running Git command has been added at `datalad_next.runners.git`. Direct uses of datalad-core runners, or `subprocess.run()` for this purpose have been replaced with call to these utilities. https://github.com/datalad/datalad-next/pull/585 (by @mih) - The performance of `iter_gitworktree()` has been improved by about 10%. Fixes https://github.com/datalad/datalad-next/issues/540 via https://github.com/datalad/datalad-next/pull/544 (by @mih). - New `EnsureHashAlgorithm` constraint to automatically expose and verify algorithm labels from `hashlib.algorithms_guaranteed` Fixes https://github.com/datalad/datalad-next/issues/346 via https://github.com/datalad/datalad-next/pull/492 (by @mslw @adswa) - The `archivist` remote now supports archive type detection from `*E`-type annex keys for `.tgz` archives too. Fixes https://github.com/datalad/datalad-next/issues/517 via https://github.com/datalad/datalad-next/pull/518 (by @mih) - `iter_zip()` uses a dedicated, internal `PurePath` variant to report on directories (`_ZipFileDirPath`). This enables more straightforward `item.name in zip_archive` tests, which require a trailing `/` for directory-type archive members. https://github.com/datalad/datalad-next/pull/430 (by @christian-monch) - A new `ZipArchiveOperations` class added support for ZIP files, and enables their use together with the `archivist` git-annex special remote. https://github.com/datalad/datalad-next/pull/578 (by @christian-monch) - `datalad ls-file-collection` has learned additional collections types: - The new `zipfile` collection type that enables uniform reporting on the additional archive type. - The new `annexworktree` collection that enhances the `gitworktree` collection by also reporting on annexed content, using the new `iter_annexworktree()` implementation. It is about 15% faster than a `datalad --annex basic --untracked no -e no -t eval`. - The new `gittree` collection for listing any Git tree-ish. - A new `iter_gitstatus()` can replace the functionality of `GitRepo.diffstatus()` with a substantially faster implementation. It also provides a novel `mono` recursion mode that completely hides the notion of submodules and presents deeply nested hierarchies of datasets as a single "monorepo". https://github.com/datalad/datalad-next/pull/592 (by @mih) - A new `next-status` command provides a substantially faster alternative to the datalad-core `status` command. It is closely aligned to `git status` semantics, only reports changes (not repository listings), and supports type change detection. Moreover, it exposes the "monorepo" recursion mode, and single-directory reporting options of `iter_gitstatus()`. It is the first command to use `dataclass` instances as result types, rather than the traditional dictionaries. Git v2.31 or later is required. - `SshUrlOperations` now supports non-standard SSH ports, non-default user names, and custom identity file specifications. Fixed https://github.com/datalad/datalad-next/issues/571 via https://github.com/datalad/datalad-next/pull/570 (by @mih) - A new `EnsureRemoteName` constraint improves the parameter validation of `create-sibling-webdav`. Moreover, the command has been uplifted to support uniform parameter validation also for the Python API. Missing required remotes, or naming conflicts are now detected and reported immediately before the actual command implementation runs. Fixes https://github.com/datalad/datalad-next/issues/193 via https://github.com/datalad/datalad-next/pull/577 (by @mih) - `datalad_next.repo_utils` provide a collection of implementations for common operations on Git repositories. Unlike the datalad-core `Repo` classes, these implementations do no require a specific data structure or object type beyond a `Path`. ## πŸ› Bug Fixes - Add patch to fix `update`'s target detection for adjusted mode datasets that can crash under some circumstances. See https://github.com/datalad/datalad/issues/7507, fixed via https://github.com/datalad/datalad-next/pull/509 (by @mih) - Comparison with `is` and a literal was replaced with a proper construct. While having no functional impact, it removes an ugly `SyntaxWarning`. Fixed https://github.com/datalad/datalad-next/issues/526 via https://github.com/datalad/datalad-next/pull/527 (by @mih) ## πŸ“ Documentation - The API documentation has been substantially extended. More already documented API components are now actually renderer, and more documentation has been written. ## 🏠 Internal - Type annotations have been extended. The development workflows now inform about type annotation issues for each proposed change. - Constants have been migrated to `datalad_next.consts`. https://github.com/datalad/datalad-next/pull/575 (by @mih) ## πŸ›‘ Tests - A new test verifies compatibility with HTTP serves that do not report download progress. https://github.com/datalad/datalad-next/pull/369 (by @christian-monch) - The overall noise-level in the test battery output has been reduced substantially. INFO log messages are no longer shown, and command result rendering is largely suppressed. New test fixtures make it easier to maintain tidier output: `reduce_logging`, `no_result_rendering`. The contribution guide has been adjusted encourage their use. - Tests that require an unprivileged system account to run are now skipped when executed as root. This fixes an issue of the Debian package. https://github.com/datalad/datalad-next/pull/593 (by @adswa) # 1.0.2 (2023-10-23) -- Debianize! ## 🏠 Internal - The `www-authenticate` dependencies is dropped. The functionality is replaced by a `requests`-based implementation of an alternative parser. This trims the dependency footprint and facilitates Debian-packaging. The previous test cases are kept and further extended. Fixes https://github.com/datalad/datalad-next/issues/493 via https://github.com/datalad/datalad-next/pull/495 (by @mih) ## πŸ›‘ Tests - The test battery now honors the `DATALAD_TESTS_NONETWORK` environment variable and downgrades by skipping any tests that require external network access. (by @mih) # 1.0.1 (2023-10-18) ## πŸ› Bug Fixes - Fix f-string syntax in error message of the `uncurl` remote. https://github.com/datalad/datalad-next/pull/455 (by @christian-monch) - `FileSystemItem.from_path()` now honors its `link_target` parameter, and resolves a target for any symlink item conditional on this setting. Previously, a symlink target was always resolved. Fixes https://github.com/datalad/datalad-next/issues/462 via https://github.com/datalad/datalad-next/pull/464 (by @mih) - Update the vendor installation of versioneer to v0.29. This resolves an installation failure with Python 3.12 due to the removal of an ancient class. Fixes https://github.com/datalad/datalad-next/issues/475 via https://github.com/datalad/datalad-next/pull/483 (by @mih) - Bump dependency on Python to 3.8. This is presently the oldest version still supported upstream. However, some functionality already used 3.8 features, so this is also a bug fix. Fixes https://github.com/datalad/datalad-next/issues/481 via https://github.com/datalad/datalad-next/pull/486 (by @mih) ## πŸ’« Enhancements and new features - Patch datalad-core's `run` command to honor configuration defaults for substitutions. This enables placeholders like `{python}` that point to `sys.executable` by default, and need not be explicitly defined in system/user/dataset configuration. Fixes https://github.com/datalad/datalad-next/issues/478 via https://github.com/datalad/datalad-next/pull/485 (by @mih) ## πŸ“ Documentation - Include `gitworktree` among the available file collection types listed in `ls-file-collection`'s docstring. Fixes https://github.com/datalad/datalad-next/issues/470 via https://github.com/datalad/datalad-next/pull/471 (by @mslw) - The renderer API documentation now includes an entrypoint for the runner-related functionality and documentation at https://docs.datalad.org/projects/next/en/latest/generated/datalad_next.runners.html Fixes https://github.com/datalad/datalad-next/issues/466 via https://github.com/datalad/datalad-next/pull/467 (by @mih) ## πŸ›‘ Tests - Simplified setup for subprocess test-coverage reporting. Standard pytest-cov features are not employed, rather than the previous approach that was adopted from datalad-core, which originated in a time when testing was performed via nose. Fixes https://github.com/datalad/datalad-next/issues/453 via https://github.com/datalad/datalad-next/pull/457 (by @mih) # 1.0.0 (2023-09-25) This release represents a milestone in the development of the extension. The package is reorganized to be a collection of more self-contained mini-packages, each with its own set of tests. Developer documentation and guidelines have been added to aid further development. One particular goal is to establish datalad-next as a proxy for importing datalad-core functionality for other extensions. Direct imports from datalad-core can be minimized in favor of imports from datalad-next. This helps identifying functionality needed outside the core package, and guides efforts for future improvements. The 1.0 release marks the switch to a more standard approach to semantic versioning. However, although a substantial improvements have been made, the 1.0 version nohow indicates a slowdown of development or a change in the likelihood of (breaking) changes. They will merely become more easily discoverable from the version label alone. Notable high-level features introduced by this major release are: - The new `UrlOperations` framework to provide a set of basic operations like `download`, `upload`, `stat` for different protocols. This framework can be thought of as a replacement for the "downloaders" functionality in datalad-core -- although the feature list is not 100% overlapping. This new framework is more easily extensible by 3rd-party code. - The `Constraints` framework elevates parameter/input validation to the next level. In contrast to datalad-core, declarative input validation is no longer limited to the CLI. Instead, command parameters can now be validated regardless of the entrypoint through which a command is used. They can be validated individually, but also sets of parameters can be validated jointly to implement particular interaction checks. All parameter validations can now be performed exhaustive, to present a user with a complete list of validation errors, rather then the fail-on-first-error method implemented exclusively in datalad-core. Validation errors are now reported using dedicated structured data type to aid their communication via non-console interfaces. - The `Credentials` system has been further refined with more homogenized workflows and deeper integration into other subsystems. This release merely represents a snapshot of continued development towards a standardization of credential handling workflows. - The annex remotes `uncurl` and `archivist` are replacements for the datalad-core implementations `datalad` and `datalad-archive`. The offer substantially improved configurability and leaner operation -- built on the `UrlOperations` framework. - A growing collection of iterator (see `iter_collections`) aims to provide fast (and more Pythonic) operations on common data structures (Git worktrees, directories, archives). The can be used as an alternative to the traditional `Repo` classes (`GitRepo`, `AnnexRepo`) from datalad-core. - Analog to `UrlOperations` the `ArchiveOperations` framework aims to provide an abstraction for operations on different archive types (e.g., TAR). The represent an alternative to the traditional implementations of `ExtractedArchive` and `ArchivesCache` from datalad-core, and aim at leaner resource footprints. - The collection of runtime patches for datalad-core has been further expanded. All patches are now individually documented, and applied using a set of standard helpers (see http://docs.datalad.org/projects/next/en/latest/patches.html). For details, please see the changelogs of the 1.0.0 beta releases below. ## πŸ’« Enhancements and new features - `TarArchiveOperations` is the first implementation of the `ArchiveOperations` abstraction, providing archive handlers with a set of standard operations: - `open` to get a file object for a particular archive member - `__contains__` to check for the presence of a particular archive member - `__iter__` to get an iterator for processing all archive members https://github.com/datalad/datalad-next/pull/415 (by @mih) ## πŸ› Bug Fixes - Make `TarfileItem.name` be of type `PurePosixPath` to reflect the fact that a TAR archive can contain members with names that cannot be represent unmodified on a non-POSIX file system. https://github.com/datalad/datalad-next/pull/422 (by @mih) An analog change is done for `ZipfileItem.name`. https://github.com/datalad/datalad-next/pull/409 (by @christian-monch) - Fix `git ls-file` parsing in `iter_gitworktree()` to be compatible with file names that start with a `tab` character. https://github.com/datalad/datalad-next/pull/421 (by @christian-monch) ## πŸ“ Documentation - Expanded guidelines on test implementations. - Add missing and fix wrong docstrings for HTTP/WebDAV server related fixtures. https://github.com/datalad/datalad-next/pull/445 (by @adswa) ## 🏠 Internal - Deduplicate configuration handling code in annex remotes. https://github.com/datalad/datalad-next/pull/440 (by @adswa) ## πŸ›‘ Tests - New test fixtures have been introduced to replace traditional test helpers from datalad-core: - `datalad_interactive_ui` and `datalad_noninteractive_ui` for testing user interactions. They replace `with_testsui`. https://github.com/datalad/datalad-next/pull/427 (by @mih) - Expand test coverage for `create_sibling_webdav` to include recursive operation. https://github.com/datalad/datalad-next/pull/434 (by @adswa) # 1.0.0b3 (2023-06-09) ## πŸ› Bug Fixes - Patch `CommandError`, the standard exception raised for any non-zero exit command execution to now reports which command failed with `repr()` too. Previously, only `str()` would produce an informative message about a failure, while `repr()` would report `CommandError('')`, unless a dedicated message was provided. (by @mih) - Some error messages (in particular from within git-annex special remotes) exhibited uninformative error messages like `CommandError('')`. This is now fixed by letting `CommandError` produce the same error rendering in `__str__` and `__repr__`. Previously, `RuntimeError.__repr__` was used, which was unaware of command execution details also available in the exception. https://github.com/datalad/datalad-next/pull/386 (by @mih) - The `datalad-annex` Git remote helper can now handle the case where a to-be-clone repository has a configured HEAD ref that does not match the local configured default (e.g., `master` vs `main` default branch). Fixes https://github.com/datalad/datalad-next/issues/412 via https://github.com/datalad/datalad-next/pull/411 (by @mih) - Patch `create_sibling_gitlab` to work with present day GitLab deployments. This required adjusting the naming scheme for the `flat` and `collection` layouts. Moreover, the `hierarchy` layout is removed. it has never been fully implemented, and conceptually suffers from various corner-cases that cannot be (easily) addressed. Consequently, the `collection` layout is the new default. It's behavior matches that of `hierarchy` as far as this was functional, hence there should be no breakage for active users. https://github.com/datalad/datalad-next/pull/413 ## πŸ’« Enhancements and new features - Patch the process entrypoint of DataLad's git-annex special remote implementations to funnel internal progress reporting to git-annex via standard `PROGRESS` protocol messages. This makes it obsolete (in many cases) to implement custom progress reporting, and the use of the standard `log_progress()` helper (either directly or indirectly) is sufficient to let both a parent DataLad process or git-annex see progress reports from special remotes. Fixes https://github.com/datalad/datalad-next/issues/328 via https://github.com/datalad/datalad-next/pull/329 (by @mih) - The `HttpUrlOperations` handler now supports custom HTTP headers. This makes it possible to define custom handlers in configuration that include such header customization, for example to send custom secret or session IDs. Fixes https://github.com/datalad/datalad-next/issues/336 (by @mih) - `Constraint` implementations now raise `ConstraintError` consistently on a violation. This now makes it possible to distinguish properly handled violations from improper implementation of such checks. Moreover, `raise_for()` is now used consistently, providing uniform, structured information on such violations. `ConstraintError` is derived from `ValueError` (the exception that was previously (mostly) raised. Therefore, client-code should continue to work without modification, unless a specific wording of an exception message is relied upon. In few cases, an implicit `TypeError` (e.g., `EnsureIterableof`) has been replaced by an explicit `ConstraintError`, and client code needs to be adjusted. The underlying exception continues to be available via `ConstraintError.caused_by`. (by @mih) - New `MultiHash` helper to compute multiple hashes in one go. Fixes https://github.com/datalad/datalad-next/issues/345 (by @mih) - As a companion of `LeanGitRepo` a `LeanAnnexRepo` has been added. This class is primarily used to signal that particular code does not require the full `AnnexRepo` API, but works with a much reduced API, as defined by that class. The API definition is not final and will grow in future releases to accommodate all standard use cases. https://github.com/datalad/datalad-next/pull/387 (by @mih) - Dedicated dataclasses for common types, such as git-annex keys (`AnnexKey`) and `dl+archives:` URLs (`ArchivistLocator`) have been added. They support parsing and rendering their respective plain-text representations. These new types are now also available for more precise type annotation and argument validation. (by @mih) - `datalad_next.archive_operations` has been added, and follows the pattern established by the `UrlOperations` framework, to provide uniform handling to different archive types. Two main (read) operations are supported: iteration over archive members, and access to individual member content via a file-like. (by @mih) - New `archivist` git-annex special remote, as a replacement for the `datalad-archives` remote. It is implemented as a drop-in replacement with the ability to also fall-back on the previous implementation. In comparison to its predecessor, it reduces the storage overhead from 200% to 100% by doing partial extraction from fully downloaded archives. It is designed to be extended with support for partial access to remote archives (thereby reducing storage overhead to zero), but this is not yet implemented. - New `datalad_next.iter_collections` module providing iterators for items in particular collections, such as TAR or ZIP archives members, the content of a file system directory, or the worktree of a Git repository. Iterators yield items of defined types that typically carry information on the properties of collections items, and (in the case of files) access to their content. - New command `ls_file_collection()` is providing access to a select set of collection iterators via the DataLad command. In addition to the plain iterators, it provide uniform content hashing across all supported collection types. - The `datalad-annex` Git remote helper can now recognize and handle legacy repository deposits made by its predecessor from `datalad-osf`. https://github.com/datalad/datalad-next/pull/411 (by @mih) ## 🏠 Internal - Remove DataLad runner performance patch, and all patches to clone functionality. They are included in datalad-0.18.1, dependency adjusted. - New `deprecated` decorator for standardized deprecation handling of commands, functions, and also individual keyword arguments of callables, and even particular values for such arguments. Inspired by https://github.com/datalad/datalad/issues/6998. Contributed by @adswa - Use the correct type annotation for `cfg`-parameter of `datalad_next.utils.requests_auth.DataladAuth.__init__()` https://github.com/datalad/datalad-next/pull/385 (by @christian-monch) - The patch registry has been moved to `datalad_next.patches.enabled`, and the `apply_patch()` helper is now located in `datalad_next.patches` directly to avoid issues with circular dependencies when patching core components like the `ConfigManager`. The documentation on patching has been adjusted accordingly. https://github.com/datalad/datalad-next/pull/391 (by @mih) - The `main()` entrypoint of the `datalad-annex` Git remote helper has be generalized to be more reusable by other (derived) remote helper implementations. https://github.com/datalad/datalad-next/pull/411 (by @mih) # 1.0.0b2 (2023-03-17) ## πŸ’« Enhancements and new features - `CredentialManager` - The Credential Manager gained a new helper, ``obtain()``, that supports a credential selection by name/ID, falls back to querying with a set of properties, and would finally resort to an interactive credential query from the user. ([#216](https://github.com/datalad/datalad-next/pull/216) by @mih) - All optional arguments of the CredentialManager are now keyword-argument-only ([#230](https://github.com/datalad/datalad-next/pull/230) by @mih) - Users no longer need to provide type hints for legacy credentials in "provider" configurations ([#247](https://github.com/datalad/datalad-next/pull/247) by @mih) - Credential reporting supports a ``cred_type`` annotation ([#257](https://github.com/datalad/datalad-next/pull/257) by @mih) - Credential errors for GitHub-like remotes were improved to hint users how to update or set new credentials ([#235](https://github.com/datalad/datalad-next/pull/235) by @mih) - `UrlOperations` - The URL handler can now load configurations from config files ([#222](https://github.com/datalad/datalad-next/pull/222) by @mih) - Improved messaging within `URLOperationsRemoteError` ([#308](https://github.com/datalad/datalad-next/pull/308) by @mih) - `Parameter validation` - A new `validate_defaults` parameter of ``EnsureCommandParameterization`` allows opt-in parameter validation, which causes processing of any specified parameter's default. ([#227](https://github.com/datalad/datalad-next/pull/227) by @mih) - A new base class ``ConstraintError`` can communicate parameter validation errors and can associate constraint violations with a particular context. ``CommandParametrizationError`` uses it to communicate violations for a full command parameterization at once and is used in an improved `EnsureCommandParametrization` constraint. Callers can now also decide whether to perform an exhaustive parameter validation, or fail on first error. ([#234](https://github.com/datalad/datalad-next/pull/234) by @mih) - A new ``ConstraintWithPassthrough`` constraint exposes `EnsureParameterConstraint`'s pass-through feature ([#244](https://github.com/datalad/datalad-next/pull/244) by @mih) - `EnsureCommandParameterization` learned a `tailor_for_dataset()` parameter that can be used to identify which parameters' constraints should be tailored for which dataset. This allows tailoring constraints for particular datasets ([#260](https://github.com/datalad/datalad-next/pull/260) by @mih) - ``EnsurePath`` can be tailored to dataset instances to resolve paths against a given Dataset ([#271](https://github.com/datalad/datalad-next/pull/271) by @mih) - The ``EnsureDataset`` constraint learned an optional check for a valid dataset ID ([#279](https://github.com/datalad/datalad-next/pull/279) by @adswa) - A ``WithDescription`` meta constraints paves the way for custom docs for parameters: If given, it replaces the original parameter documentation, and can be used to tailor descriptions for specific use cases. ([#294](https://github.com/datalad/datalad-next/pull/294) by @mih) - Parameter violations gained structured error reporting and customized rendering of parameter violations ([#306](https://github.com/datalad/datalad-next/pull/306) by @mih) - ``EnsureGeneratorFromFileLike`` became more suitable for batch mode use by learning to yield instead of raise internal exceptions, if configured by the caller ([#278](https://github.com/datalad/datalad-next/pull/278) by @mih) ## πŸ› Bug Fixes - Previously, the last used credential matching a ``realm`` was used unconditionally. Now, credentials without secrets are excluded. ([#248](https://github.com/datalad/datalad-next/pull/248) by @mih) - ``AND`` and ``OR`` compounds for Constraints do not modify Constraints in place anymore, but return a new instance. ([#292](https://github.com/datalad/datalad-next/pull/292) by @mih) - Even though the ``EnsureDataset`` constraint returns ``DatasetParameter`` objects, ``_execute_command`` that would patch up DataLad commands wasn't able to work with them ([#269](https://github.com/datalad/datalad-next/pull/269) by @adswa) ## πŸͺ“ Deprecations and removals - The URL operation ``sniff`` was renamed to ``stat``. ([#231](https://github.com/datalad/datalad-next/pull/231) by @adswa) - `serve_path_via_webdav()` that came with 0.2 was deprecated in favor of the `webdav_server` fixture ([#301](https://github.com/datalad/datalad-next/pull/301) by @mih) ## πŸ“ Documentation - A dedicated Developer Guide section of the docs was introduced ([#304](https://github.com/datalad/datalad-next/pull/304) by @adswa) - The README mentions the `uncurl` special remote, and the documentation now provide installation information - ``CONTRIBUTING.md`` was updated on patching ([#262](https://github.com/datalad/datalad-next/pull/262/) by @mih) ## 🏠 Internal - Package dependencies were made explicit ([#212](https://github.com/datalad/datalad-next/pull/212) by @mih) - Misc. code reorganization: - The CredentialManager was elevated to a top-level module ([#229](https://github.com/datalad/datalad-next/pull/220) by @mih) - Dataset-lookup behavior of the ``credentials`` command became identical to ``download`` ([#256](https://github.com/datalad/datalad-next/pull/256) by @mih) - The DataLad runner performance patch and all patches to clone functionality were removed as they are included in datalad-0.18.1; The dependency was adjusted accordingly. ([#218](https://github.com/datalad/datalad-next/pull/218) by @mih) - Compound constraints got a comprehensive ``__repr__`` to improve debugging ([#276](https://github.com/datalad/datalad-next/pull/276) by @mih) - Discontinue legacy code ([#300](https://github.com/datalad/datalad-next/pull/300/) by @mih) ## πŸ›‘ Tests - Automatic CI builds were disabled for changes constrained to the following files and directories: `.github/`, `CHANGELOG.md`, `CITATION.cff`, `CONTRIBUTORS`, `LICENSE`, `Makefile`, `README.md`, `readthedocs.yml` - Coverage reports for the uncurl special remote ([#220](https://github.com/datalad/datalad-next/pull/220) by @mih) - Tests will not fail if coverage uploads fail ([#241](https://github.com/datalad/datalad-next/pull/241/files) by @mih) - GitHub actions use the `datalad-installer` to install git-annex ([#239](https://github.com/datalad/datalad-next/pull/239/files) by @mih) - A bug in DataLad's test setup causes configuration managers to leak across datasets (https://github.com/datalad/datalad/issues/7297). Next implemented test isolation for keyring and config as a fix ([#263](https://github.com/datalad/datalad-next/pull/263) by @mih) - A number of new pytest fixtures were introduced: - `memory_keyring` ([#254](https://github.com/datalad/datalad-next/pull/254) by @mih), which was then replaced by ``tmp_keywing`` ([#264](https://github.com/datalad/datalad-next/pull/264)) - `dataset` and `existing_dataset` ([#296](https://github.com/datalad/datalad-next/pull/296) by @mih) - `webdav_server` ([#297](https://github.com/datalad/datalad-next/pull/297/) by @mih) - `httpbin` ([#313](https://github.com/datalad/datalad-next/pull/313) by @jwodder) - 100% coverage for constraints ([#259](https://github.com/datalad/datalad-next/pull/259/)) # 1.0.0b1 (2022-12-23) ## πŸ’« Enhancements and new features - Improved composition of importable functionality. Key components for `commands`, `annexremotes`, `datasets` (etc) are collected in topical top-level modules that provide "all" necessary pieces in a single place. - Add patch to `ThreadedRunner` to use a more optimal buffer size for its read thread. This was previously fixed to 1024 bytes, and now uses the value of `shutil.COPY_BUFSIZE` as a platform-tailored default. This can boost the throughput from several tens to several hundreds MB/s. - A new `download` command aims to replace any download-related functionality in DataLad. It supports single-pass checksumming, progress reporting for any supported URL scheme. Currently support schemes are `http(s)://`, `file://`, and `ssh://`. The new command integrates with the `datalad-next` credential system, and supports auto-discovery, interactive-prompt-on-demand, and (optional) save-on-success of credentials. Additional URL scheme handlers can be provided by extension packages. Unlike the datalad-core downloaders framework, they can be fully standalone, as long as they implement a lean adaptor class (see `datalad_next.url_operations`). The `AnyUrlOperations` is provided to enable generic usage in client code where an underlying handler is auto-selected based on the URL scheme. `datalad_next.url_operations.any._urlscheme_handler` contains a (patch-able) mapping of scheme identifiers to handler classes. The `uncurl` special remote makes this URL operations framework accessible via `git-annex`. It provides flexible means to compose and rewrite URLs (e.g., to compensate for storage infrastructure changes) without having to modify individual URLs recorded in datasets. It enables seamless transitions between any services and protocols supported by the framework. - A `python-requests` compatible authentication handler (`DataladAuth`) that interfaces DataLad's credential system has been added. - A substantially more comprehensive replacement for DataLad's `constraints` system for type conversion and parameter validation has been developed and is included in this release. This includes all types of the predecessor in the DataLad core package, and a large number of additions, including - `EnsureMapping` (aids handling of key-value specification) - `EnsureGeneratorFromFileLike` (aids reading inputs from, e.g. STDIN; see the `download` command for how reading JSON-lines input can be supported in addition with virtually no changes to the actual command implementation) - `EnsurePath` (existing or not, particular formats, etc.) - `EnsureJSON` (automatic validation and loading) - `Ensure(Parsed)URL` (pattern matching, requiring/forbidding components) - `EnsureGitRefName` (check for compliance with Git's naming rules) - Commands can now opt-in to receive fully validated parameters. This can substantially simplify the implementation complexity of a command at the expense of a more elaborate specification of the structural and semantic properties of the parameters. This specification is achieved by declaring an `EnsureCommandParameterization`, in a `_validator_` member of a command's `ValidatedInterface` class. This feature is introduced as a patch to the command execution in datalad-core. With this patch, commands are now exclusively called with keyword-style parameters only. This feature is in an early stage of development (although all included commands have already been ported to use it) that will likely undergo substantial changes in the coming releases. - A new `EnsureDataset` constraint is provided that returns a `DatasetParameter` on successful validation. This return value contains the original input specification, and the `Dataset` class instance. The `resolve_path()` utility is adjust to support this parameter-type, thereby making the use of the `require_dataset()` utility obsolete. - As a companion for the `http(s)://` URL handling for the new `download` command, a `requests`-compatible authentication handler has been implemented that integrates with the `datalad-next` credential system. ## πŸ“ Documentation - All runtime patches are now documented and included in the readthedocs-hosted documentation. ## 🏠 Internal - No code uses `Constraint` implementations from the DataLad core package anymore. - Further expand type annotations of the code base. # 0.6.3 (2022-10-26) -- Tests only ## πŸ› Bug Fixes - Small change in the tests of the `tree` command for more robust behavior across Python and pytest versions. https://github.com/datalad/datalad-next/pull/117 (by @bpoldrack) # 0.6.2 (2022-10-14) -- Hidden secrets ## πŸ› Bug Fixes - `CredentialManager` no longer splits a credential input prompt into a prompt message (`ui.message()`) and the actual input (`ui.question()`) this enables DataLad Gooey to properly render this jointly as an input dialog with a description. https://github.com/datalad/datalad-next/pull/113 (by @bpoldrack) ## πŸ’« Enhancements and new features - `CredentialManager.get()` and the `credentials` command now also report credential fragments for which there is no secret on record. This enables the discovery of DataLad's legacy credentials, and for setting a secret for them for use with the next credential system. Moreover, it reports half-configured credentials, and facilitates their clean-up or completion, for example with DataLad Gooey's credential management GUI. # 0.6.1 (2022-09-27) ## πŸ’« Enhancements and new features - A new patch set break up the implementation of `clone_dataset()` into its procedural components, and makes it more accessible for extension patches. There are no behavior changes associated with this internal reorganization. # 0.6.0 (2022-08-25) ## πŸ› Bug Fixes - Fixed datalad-push always reporting success when pushing to an export remote. Fixes https://github.com/datalad/datalad-next/issues/88 via https://github.com/datalad/datalad-next/pull/93 (by @bpoldrack) - Token secrets entered for GitHub-like sibling creation are now stored by default under a name matching the API endpoint hostname (e.g. 'api.github.com'), rather than a confusing and conflict-prone 'None'. Using the `--credential` option, an alternative name can be given, as before. Fixes https://github.com/datalad/datalad-next/issues/97 via https://github.com/datalad/datalad-next/pull/98 (by @mih) ## πŸ’« Enhancements and new features - The `configuration` command now indicates the absence of a particular configuration setting queried via `get` with a `status='impossible'` result. This change enables the distinction of an unset configuration item from an item set to an empty string with the default CLI result renderer. Fixes https://github.com/datalad/datalad/issues/6851 via https://github.com/datalad/datalad-next/pull/87 by @mih - The default of the configuration item `datalad.annex.retry` (in effect when not explicitly configured otherwise) is changed from `3` to `1`. This prevents a set of performance and user experience issues resulting from, e.g., repeated download attempts, even when no change in outcome can be expected (e.g., a wrong or no credential supplied). This change can cause a loss of robustness in download behavior for services that indeed experience spurious failures. Its is recommended to specifically parametrize such command calls (e.g., downloads in CI runs) with an appropriate configuration override. Fixes https://github.com/datalad/datalad/issues/6969 and https://github.com/datalad/datalad/issues/6509 (by @mih) - New `tree` command for traversing a directory hierarchy. Like the UNIX equivalent, it can visualize a directory tree. Additionally, it annotates the output with DataLad-related information, like the location of dataset, and their nesting depth. Besides visualization, `tree` also reports structured data in the form of result records that enable other applications to use `tree` for gathering data from the file system. Fixes https://github.com/datalad/datalad-next/issues/78 via https://github.com/datalad/datalad-next/pull/92 (by @catetrai) ## πŸ“ Documentation - Add an example of adding a `user_password`-type credentials, with a given `user` property, to the examples in the `credentials` command. https://github.com/datalad/datalad-next/pull/103 (by @mslw) # 0.5.0 (2022-07-19) ## πŸ’« Enhancements and new features - The `configuration` command no longer requires a datasets to be present for a `get` operation to retrieve a configuration item from scope `global`. Fixes [#6864](https://github.com/datalad/datalad/issues/6854) via [#86](https://github.com/datalad/datalad-next/pull/86) (by @mih) # 0.4.1 (2022-07-14) ## πŸ› Bug Fixes - Fix a missing import in the credential retrieval for GitHub-like sibling creation, which made it impossible to discover credentials without providing an explicit credential name. # 0.4.0 (2022-07-08) -- datalad-annex:: for all #### πŸ’« Enhancements and new features - `datalad-annex::` Git remote helper now uses `git annex transferkey` instead of `fsck` to "probe" for `XDLRA` repository keys. This avoid problems due to a behavior change in git-annex 10.20220525, and can also speed-up operation for slow special remotes, by avoiding a dedicated probe-request. [#76](https://github.com/datalad/datalad-next/pull/76) (by @mih) - `datalad-annex::` Git remote helper now fully compatible with the Windows platform, by working around [a git-annex issue](https://git-annex.branchable.com/bugs/Fails_to_drop_key_on_windows___40__Access_denied__41__) [#77](https://github.com/datalad/datalad-next/pull/77) (by @mih) #### πŸ› Bug Fixes - Prevent docstring duplication in patched `push` command [#71](https://github.com/datalad/datalad-next/pull/71) (by @mih) #### πŸ“ Documentation - Bibliographic information on authorship was added [#80](https://github.com/datalad/datalad-next/pull/80) (by @mslw) #### πŸ›‘ Tests - The test battery is now using `pytest`. This change required bumping the dependency on DataLad to version 0.17.0. [#73](https://github.com/datalad/datalad-next/pull/73) (by @mih) #### 🏠 Internal - Reduced code duplication by consolidating on a common helper for sibling identification, now available from DataLad 0.17.0 [#82](https://github.com/datalad/datalad-next/pull/82) (by @adswa) #### Authors: 3 - Michael Hanke (@mih) - MichaΕ‚ Szczepanik (@mslw) - Adina Wagner (@adswa) # 0.3.0 (2022-05-25) -- Optimized push #### πŸ’« Enhancements and new features - Make push avoid refspec handling for special remote push targets. See PR https://github.com/datalad/datalad-next/pull/64 for details on the associated behavior changes that are introduced with this new patch. # 0.2.2 (2022-04-29) -- More docs! #### πŸ“ Documentation - Adjusted documentation of patched datalad-core commands now also shows properly in Python sessions. - Extended the documentation on collaborative workflows with ``datalad-annex::``Git remotes and WebDAV siblings. # 0.2.1 (2022-04-28) -- User experience #### πŸ’« Enhancements and new features - Disable auto-enabling of webdav storage remotes on clone. Datalad does not yet support the needed inspection to determine the necessary credentials automatically. Instead an explicit `datalad sibling enable` call is required. This is now also added to the documentation. - Make sure that `create-sibling-webdav` does not ask users to input the internal `realm` property, when prompting for credentials. - `CredentialManager` now displays more appropriate labels when prompting for a secret, e.g. `password` instead of `user_password`. # 0.2.0 (2022-04-28) -- WebDAV This release primarily brings the ability to store DataLad datasets on a WebDAV server. This is done in a way that allows for cloning such dataset with `datalad clone` from such a WebDAV server too. This feature enables full-featured DataLad-based collaborative workflows on widely available cloud storage systems, such as owncloud/next/cloud -- which are also the basis for several institutional services like the European Open Science Cloud's (EOSC) B2DROP service. #### πŸ’« Enhancements and new features - A `create-sibling-webdav` command for hosting datasets on a WebDAV server via a sibling tandem for Git history and file storage. Datasets hosted on WebDAV in this fashion are cloneable with `datalad-clone`. A full annex setup for storing complete datasets with historical file content version, and an additional mode for depositing single-version dataset snapshot are supported. The latter enables convenient collaboration with audiences that are not using DataLad, because all files are browsable via a WebDAV server's point-and-click user interface. - Enhance `datalad-push` to automatically export files to git-annex special remotes configured with `exporttree=yes`. - Enhance `datalad-siblings enable` (`AnnexRepo.enable_remote()` to automatically deploy credentials for git-annex special remotes that require them. - `git-remote-datalad-annex` is a Git remote helper to push/fetch to any location accessible by any git-annex special remote. - `git-annex-backend-XDLRA` (originally available from the `mihextras` extension) is a custom external git-annex backend used by git-remote-datalad-annex. A base class to facilitate development of external backends in Python is also provided. - `serve_path_via_webdav` test decorator that automatically deploys a local WebDAV server. - `with_credential` test decorator that temporarily deploy a credential to the local credential system. - Utilities for HTTP handling - `probe_url()` to discover redirects and authentication requirements for an HTTP URL - `get_auth_realm()` return a label for an authentication realm that can be used to query for matching credentials - Utilities for special remote credential management: - `get_specialremote_credential_properties()` inspects a special remote and return properties for querying a credential store for matching credentials - `update_specialremote_credential()` updates a credential in a store after successful use - `get_specialremote_credential_envpatch()` returns a suitable environment "patch" from a credential for a particular special remote type # 0.1.0 (2022-03-31) -- Credentials, please! #### πŸ’« Enhancements and new features - A new credential management system is introduced that enables storage and query of credentials with any number of properties associated with a secret. These properties are stored as regular configuration items, following the scheme `datalad.credential..`. The special property `secret` lives in a keystore, but can be overridden using the normal configuration mechanisms. The new system continues to support the previous credential storage setup. Fixes [#6519](https://github.com/datalad/datalad/issues/6519) ([@mih](https://github.com/mih)) - A new `credentials` command enables query, modification and storage of credentials. Legacy credentials are also supported, but may require the specification of a `type`, such as (`token`, or `user_password`) to be discoverable. Fixes [#396](https://github.com/datalad/datalad/issues/396) ([@mih](https://github.com/mih)) - Two new configuration settings enable controlling how the interactive entry of credential secrets is conducted for the new credential manager: `datalad.credentials.repeat-secret-entry` can be used to turn off the default double-entry of secrets, and `datalad.credentials.hidden-secret-entry` can turn off the default hidden entry of secrets. Fixes [#2721](https://github.com/datalad/datalad/issues/2721) ([@mih](https://github.com/mih)) #### Authors: 1 - Michael Hanke ([@mih](https://github.com/mih)) ---