Version 4.0

This major release 4.0 of ELS adds the ELS Navigator desktop application and a variety of related enhancements and changes.

The Navigator is purpose-built to make building and on-going maintenance of collections of all types of data across multiple storage devices easier. Originally built for home media collections ELS 4.0 adds a new dimension to library-based, multi-device, cross-platform file management.

Using the built-in capabilities of ELS the Navigator provides a visual tool for curating a collection either locally or remotely. Tools and jobs are provided to make performing repetitive tasks easier. And the original back-up tool is built-in of course.

Features:

• Modes (where the Navigator is running)
  • On a media collection where Hints are tracked
  • On a separate workstation where Hints are not tracked
• Browser
  • Split-pane Publisher/Subscriber view
  • Collection and System tabs for each
  • Local or remote subscriber
  • Drag 'n Drop and Copy, Cut, Paste
  • Automatic ELS Hint Tracking
    • Renames
    • Moves
    • Deletes
  • Multiple named tool configurations
    • Duplicate Finder
    • Empty Directory Finder
    • Junk Remover
    • Operations
    • Renamer
    • Sleep
  • Named jobs of sequenced tools to automate repetitive tasks
• BackUp
  • Configure named ELS back-ups with different configurations
  • Execute and monitor back-up runs
  • Generate scripts for command line and/or background scheduled execution
• Libraries
  • Create and edit ELS Publisher, Subscriber, and Hint Server JSON files
  • Create and edit ELS Authentication keys, Hint keys, blacklist and whitelist

Like the rest of ELS the new Navigator is a general tool for anyone manipulating large volumes of data across multiple storage devices and is also compatible with modern media systems such a Plex Media Server and Jellyfin. Works on Windows, Linux and Mac.

It's all built-in.

Upgrade Notes

ELS 4.0 is significantly different from previous versions. Changes and bug fixes have been made throughout the code, too numerous to list every change. Read these notes thoroughly when upgrading for changes, additions and enhancements.

1. When upgrading to ELS 4.0 from previous versions be sure to complete all Hint processing prior to the upgrade. Ensure the Hint Tracking datastore is empty. The syntax used in the .els Hint files has been updated.
   
   
2. With the Navigator a formal directory structure is needed to store the various configuration files. The location may be defined using the -C | --config option. The default is the user's home directory in subdirectory .els/

Enhancements

1. ELS Navigator.

   1. Navigator is a publisher in ELS terms, by design.
   2. Navigator supports two scenarios:
      1. On a media collection such as a media server system, and optionally connected to a back-up running an ELS subscriber listener.
      2. On a separate workstation, and optionally connected to a media collection system or back-up running an ELS subscriber listener.
   3. The File, Open Publisher dialog shows radio buttons for Collection or Workstation.
      1. This setting determines whether Hints are stored when Hint Tracking is enabled.
         1. If Collection then the Hints are added, and tracked if enabled.
         2. If Workstation then Hints are not used during operations.
   4. Command-line behavior changes when using -n | --navigator:
      1. -P sets the Navigator for running on a Collection
      2. -p sets the Navigator as a Workstation
      3. -S sets the Navigator for a remote Subscriber
      4. -s sets the Navigator as a local Subscriber
      5. Note: The distinction of collection or library is not used with the Navigator. All data displayed are from active storage scans.

2. New ELS project download options including an all-in-one with an embedded Java JRE.

3. The remote communication paradigm has been changed to provide more reliability.

Command Line Changes

1. The -n | --rename option has been removed in favor of the rename tool in the Navigator.

2. The -n option has been repurposed for the Navigator and the --navigator option has been added.

3. Added option -y | --preserve-dates to retain original file dates.

4. Added option -z | --decimal-scale to format numeric (B, KB, MB, GB, TB) values with a 1000 decimal scale instead of a 1024 binary scale.

5. Added option -j | --job to execute a previously-defined ELS Job. If the name contains whitespace enclose it in quotations. In this mode the job controls ELS actions.

6. Added option -A | --auth-keys for subscriber and publisher listeners. This is the same format as Hint Keys. Authentication keys are used to authenticate both workstations and publishers instead of requiring a specific system defined by -s|S.

7. Added option -g | --listener-keep-going. For a Publisher the "keep going" option skips sending the quit command to the subscriber when the backup operation is complete. For a subscriber it skips ending with a fault on an unexpected disconnect (EOL) and ignores quit commands. To stop a subscriber in this mode use the --listener-quit command.

8. Added option -G | --listener-quit that only sends the stop command to a remote subscriber, then exits. Similar to the -Q | --force-quit option.

9. Added option -B | --blacklist that uses a text file of one-line IP addresses to filter and block incoming connections. The blacklist supports # style comments and blank lines. Each IP address is an IPv4 dotted address, e.g. 127.0.0.1, on separate lines.

10. Added option -I | --ip-whitelist to filter and allow incoming connections. Similar to the -B | --blacklist file.

11. Changed the behavior of -u | --duplicates where duplicates are now only logged when this option is enabled. Otherwise only a total number is reported in the statistics. Previously duplicates were always reported in a back-up or dry-run.

12. Added option -E | --empty-directories where empty directories are logged when this option is enabled. Otherwise only a total number is reported in the statistics.

13. Added option -N | --ignored to log ignored files. For backup runs and the --duplicates option.

14. Implemented detailed logging of communications-related steps using the "trace" log level for the --console-level and --debug-level options.

15. Added remote mode J to the -r|--remote option. This is to support command-line use of the "Any Subscriber" origin option of Job tasks. Combined with -j|--job the remote subscriber defined with -s|-S is used. This is in contrast to the "Specific Subscriber" origin option where the subscriber defined for the task overrides the -s|-S option.

16. Added --dump-system that prints all JVM System.getProperties() values then exits.

17. Added -C | --config to set the location of the ELS configuration directory. Use "-C ." for the current directory. Default is user home directory in .els/

18. Added --marker "arg" to aid in identifying a running process. Any "arg" is ignored. This option has no effect on ELS and is intended to help with scripting.

19. Added -O | --override-subscriber-server [address:port]. If address:port is defined that is used, otherwise the Listen port is used for out-going connections to a Subscriber instead of the server. Subscriber listeners use the Listen address unless the [address:port] is defined. If Listen is not defined the Host is used. Outgoing connections use the Host address unless the -O option for Listen or the [address:port] are defined.

20. Added -J | --override-hint-server to use the listen address:port for out-going connections to a Hint Server instead of the server. Used for outgoing LAN connections where server is used over the Internet. Hint Server uses the Listen address. There is no optional [address:port] because there can be only one.

21. Added the --logger option for the -j | --job action to execute the Job in the foreground with ELS operating in Logger mode to display the Job log. It is used internally when executing a Job in the foreground.

22. Added -V option to check for updates from the command line (terminal) and a checkForUpdate script in the installation bin/ subdirectory.

23. Added -Y option to install updates from the command line and an installUpdate script in the installation bin/ subdirectory. Always installs update even if versions match.

Other Changes

1. The ELS Navigator has necessitated the introduction of a formal user-based directory structure to hold the various preference, bookmark, library, tool, job, etc. files. All these items are kept in each user's HOME/.els/ directory.

   IMPORTANT: When upgrading from ELS versions earlier than 4.0 copy your existing library JSON files to your HOME/.els/libraries/ directory. If that directory does not exist create it.

2. When using the ELS interactive terminal (not to be confused with ELS Navigator) the "bye" command has been changed to end the terminal session but leave the remote listener running. Commands quit, exit and logout will shutdown the remote listener.

3. Added JSON library elements for temporary files:

   1. temp_dated "true/false" : If temporary files such as received collection files have date and time embedded in the filename. If false the same file is overwritten.
   2. temp_location "path" : Where to place temporary files. An empty string "" is the location of the ELS Jar file. If the path begins with "/" the user's home directory is substituted for the "".

4. Removed JSON library element "renaming" and the related Java code.

5. Changed the JSON library "ignore_patterns" behavior:

   1. If the pattern contains the path separator literal for that repository the full path is matched.
      1. For example pattern: ".\/Plex Versions." will exclude the directory "/Plex Versions" and any subdirectories and files.
   2. If the pattern does not contain the path separator literal only the right-end directory or file name is matched.

6. Added a new authentication technique for subscriber and publisher listeners.

   1. Normally the publisher and subscriber are specific.
      1. If a connection is made to a listener and it is not the specific system expected the listener will fail and exit.
   2. However if the listener is running with the -A|--auth-keys Authentication Keys file option authentication matches against all the keys. So a single subscriber listener can connect to one or more remote ELS publishers or ELS Navigators concurrently.
      1. The current limit is 100 concurrent connections.
      2. This is the same technique used by the Hint Status Server using a Hint keys file.
   3. The reasons for separate listener authentication keys and hint keys:
      1. A listener may want to allow Navigator sessions in addition to back-up systems.
      2. Hint keys control:
         1. Which back-up systems are setup to process hints.
         2. Which systems are tracked and status maintained in the Hint datastore.

7. Modified the code for methodical exit code status values. Exit code 0 is normal, 1 indicates a fault occurred. Exit code 130 is returned if Ctrl-C is hit on the command line. Useful for error handling in multi-step automation batch files or scripts.

8. Changed free space checking when backing-up a group of files so the value checked is reduced as each item in the group is copied. GitHub Issue #55.

9. Added JSON "timeout" element for the stty protocol in minutes. This provides a mechanism to avoid process hangs and the implementation uses an internal heartbeat to keep the connection alive during long-running operations. The heartbeat is not an actual ping.

10. Changed Hint syntax handling to use the more formal syntax generated by the Navigator. Important: When upgrading to ELS 4.0 from previous versions be sure to complete all Hint processing and make sure for any Hint Tracking being used, local or remote, the datastore is empty.

Operational Notes

1. When running Navigator with a remote Subscriber and executing a backup Job that would normally stop the listener when done be sure to start the remote subscriber listener with the -g | --listener-keep-going option to avoid a connection fault in Navigator when the backup is complete.

2. When performing long copy/move operations multiple copy/paste and drag 'n drop operations may be batched. Each operation is added to the existing batch(es) of running operations and are processed in order.

3. When running a backup operation or copying/moving content in Navigator the target path is determined dynamically when the target is a library. Because of this the available free space is checked during the copy/move operation and cannot be checked before the copy/move begins.

Developer Notes

1. The ELS Navigator was built using JFormDesigner.
   This inexpensive plug-in for IntelliJ allowed the creation of the Navigator much faster and with far fewer mistakes.

2. For existing user-written scripts add the "-C ." option to set the working directory to the current directory. See Command Line Changes.

3. With the addition of the Hint Status Server where a remote ELS session is employing 3 ELS processes - hint server, subscriber, and publisher - it was necessary to rearrange the disconnect/shutdown logic and sequences. These changes implement a more formal, and less brute-force, disconnect and quit approach allowing for future n-way connection possibilities.

4. For IntelliJ to run and debug the multiple processes the Multirun plugin has been added with a variety of configurations in the .idea project.

5. The mock directory has been completely rearranged to support testing and provide a completely self-contained development and test environment. In addition a mock/scripts/linux/ directory has been added with many scripts to perform application-level tests using pre-set publisher and subscriber collections and hint files.

   These scripts show many of the various ways ELS may be executed using different combinations of options. See the **README** in that directory for more information and a description of the testing sequence.

6. For IntelliJ users several run/debug configurations have been added that match the scripts in the mock/scripts/linux/ directory organized in the same way and use the same mock/ data.