This major release 4.0.0 of ELS adds the ELS Navigator desktop application and a variety of related enhancements and changes.
The Navigator is purpose-built to make building and on-going maintenance of collections of all types of media across multiple storage devices easier. Originally built for home media collections ELS 4.0 adds a new dimension to library-based, multi-device, cross-platform file management. Using the built-in capabilities of ELS the Navigator provides a visual tool for curating a collection either locally or remotely. Tools and jobs are provided to make performing repetitive tasks easier. And the original back-up tool is built-in of course.
A work in progress.
Features:
Like the rest of ELS the new Navigator is a general tool for anyone manipulating media across multiple storage devices and is also compatible with modern media systems such a Plex Media Server. Works on Windows, Linux and Mac.
It's all built-in with the -n | --navigator option.
ELS 4.0.0 is significantly different than previous versions. Changes and bug fixes have been made throughout the code, too numerous to list every change. Read these notes thoroughly when upgrading for changes, additions and enhancements.
ELS Navigator.
New ELS project download options including an all-in-one with an embedded Java JRE.
The remote communication paradigm has been changed to provide more reliability.
The -n | --rename option has been removed in favor of the rename tool in the new ELS Navigator.
The -n option has been repurposed for the Navigator and the --navigator option has been added.
Added option -y | --preserve-dates to retain original file dates.
Added option -z | --decimal-scale to format numeric (B, KB, MB, GB, TB) values with a 1000 decimal scale instead of a 1024 binary scale.
Added option -j | --job to execute a previously-defined ELS Job. If the name contains whitespace enclose it in quotations. In this mode the job controls ELS actions.
Added option -A | --auth-keys for subscriber and publisher listeners. This is the same format as Hint Keys. Authentication keys are used to authenticate both workstations and publishers instead of requiring a specific system defined by -s|S.
Added option -g | --listener-keep-going. For a Publisher the "keep going" option skips sending the quit command to the subscriber when the backup operation is complete. For a subscriber it skips ending with a fault on an unexpected disconnect (EOL) and ignores quit commands. To stop a subscriber in this mode use the --listener-quit command.
Added option -G | --listener-quit that only sends the quit command to a remote subscriber, then exits. Similar to the -Q | --force-quit option.
Added option -B | --blacklist that uses a text file of one-line IP addresses to filter and block incoming connections. The blacklist supports # style comments and blank lines. Each IP address is an IPv4 dotted address, e.g. 127.0.0.1, on separate lines.
Added option -I | --ip-whitelist to filter and allow incoming connections. Similar to the -B | --blacklist file.
Changed the behavior of -u | --duplicates where duplicates are now only logged when this option is enabled. Otherwise only a total number is reported in the statistics. Previously duplicates were always reported in a back-up or dry-run.
Added option -E | --empty-directories where empty directories are logged when this option is enabled. Otherwise only a total number is reported in the statistics.
Added option -N | --ignored to log ignored files. For backup runs and the --duplicates option.
Implemented detailed logging of communications-related steps using the "trace" log level for the --console-level and --debug-level options.
Added remote mode J to the -r|--remote option. This is to support command-line use of the "Any Subscriber" origin option of Job tasks. Combined with -j|--job the remote subscriber defined with -s|-S is used. This is in contrast to the "Specific Subscriber" origin option where the subscriber defined for the task overrides the -s|-S option.
Added --dump-system that prints all JVM System.getProperties() values then exits.
Added -C | --config to set the location of the ELS configuration directory. Use "-C ." for the current directory.
The ELS Navigator has necessitated the introduction of a formal user-based directory structure to hold the various preference, bookmark, library, tool, job, etc. files. All these items are kept in each user's HOME/.els/ directory.
IMPORTANT: When upgrading from ELS versions earlier than 4.0.0 copy your existing library JSON files to your HOME/.els/libraries/ directory. If that directory does not exist create it.
The listener daemons "bye" command behavior has changed. Now bye will leave the daemon running instead of shutting down. Quit, exit and logout still perform a shutdown.
When using the ELS interactive terminal (not to be confused with ELS Navigator) the "bye" command has been changed to end the terminal session but leave the remote listener running. Commands quit, exit and logout will shutdown the remote listener.
Added JSON library elements for temporary files:
Removed JSON library element "renaming" and the related Java code.
Changed the JSON library "ignore_patterns" behavior:
Added a new authentication technique for subscriber and publisher listeners.
Modified the code for methodical exit code status values. Exit code 0 is normal, 1 indicates a fault occurred. Exit code 130 is returned if Ctrl-C is hit on the command line. Useful for error handling in multi-step automation batch files or scripts.
Changed free space checking when backing-up a group of files so the value checked is reduced as each item in the group is copied. GitHub Issue #55.
Added JSON "timeout" element for the stty protocol in minutes. This provides a mechanism to avoid process hangs and the implementation uses an internal heartbeat to keep the connection alive during long-running operations. The heartbeat is not an actual ping.
Changed Hint syntax handling to use the more formal syntax generated by the Navigator. Important: When upgrading to ELS 4.0.0 from previous versions be sure to complete all Hint processing and make sure for any Hint Tracking being used, local or remote, the datastore is empty.
When running Navigator with a remote Subscriber and executing a backup Job that would normally stop the listener when done be sure to start the remote subscriber listener with the -g | --listener-keep-going option to avoid a connection fault in Navigator when the backup is complete.
When performing long copy/move operations multiple copy/paste and drag 'n drop operations may be batched. Each operation is added to the existing batch(es) of running operations and are processed in order.
When running a backup operation or copying/moving content in Navigator the target path is determined dynamically when the target is a library. Because of this the available free space is checked during the copy/move operation and cannot be checked before the copy/move begins.
The ELS Navigator was built using JFormDesigner.
This inexpensive plug-in for IntelliJ allowed the creation of the Navigator
much faster and with far fewer mistakes.
For existing user-written scripts add the "-C ." option to set the working directory to the current directory. See Command Line Changes, item 17.
Release 3.1.0 of ELS adds the Hint Status Tracker and a new mode - the Hint Status Server (HSS). The Tracker coordinates hint completion status locally. The Server tracks hint completion status for remote operations. These are needed when two or more back-ups are being used with hints.
The Tracker and new HSS mode are optional. All previous features and behavior remain the same. The HSS is an additional separate process that is executed before any remote operation requiring hint coordination. Options are available to allow the HSS to run continuously or "ordered" to quit by a publisher or subscriber when an operation is completed. A separate TCP/IP port is required for the status server listener.
-h has been repurposed and -H added for hint support.
Previously the -h | --version options were used for help that only displayed the version. The --version option still does that.
The -h option is now -h | --hints [file] : Hints Status Server file to enable connection to the new ELS Hint Status Server.
Added -H | --hint-server [file] : Hints Status Server to execute continuous hint status server daemon
Added -q | --quit-status : Send quit command to hint status server when operation is complete. Allows either a publisher or subscriber to tell the HSS to shutdown.
The execution sequence must be the HSS, then subscriber, then publisher. The publisher commands the subscriber to quit automatically when the operation is done. So it is best to add the --quit-status option to the subscriber so when it shuts down it will command the Hint Status Server to quit - if desired.
Added -Q | --force-quit : Special option that only connects to the HSS to send a quit command, then it ends. Requires a --hint file and -p | -P publisher file to specify the to/from connection ends respectively.
Issue #30 'Fix terminal_allowed handling'.
Added the logic necessary to use the terminal_allowed value in the JSON file.
Issue #34 'Fix empty -t | -T handling'.
Fixed the issue when using an empty -t | -T to use the sources as targets.
Issue #35 'Fix --remote M'.
Fixed the automated login issue when using --remote M.
With the addition of the Hint Status Server where a remote ELS session is employing 3 ELS processes - hint server, subscriber, and publisher - it was necessary to rearrange the disconnect/shutdown logic and sequences. These changes implement a more formal, and less brute-force, disconnect and quit approach allowing for future n-way connection possibilities.
For IntelliJ to run and debug the multiple processes the Multirun plugin has been added with a variety of configurations in the .idea project.
The mock directory has been completely rearranged to support testing and provide a completely self-contained development and test environment. In addition a mock/scripts/linux/ directory has been added with many scripts to perform application-level tests using pre-set publisher and subscriber collections and hint files.
These scripts show many of the various ways ELS may be executed using different combinations of options. See the README in that directory for more information and a description of the testing sequence.
For IntelliJ users several run/debug configurations have been added that match the scripts in the mock/scripts/linux/ directory organized in the same way and use the same mock/ data.