# Changelog All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] ## [0.11.0] - 2025-02-19 ### Changed - Fixed an issue with CA data paths being written to the credential store which could affect users working with copies of the client from multiple virtual environments. - Removed unwanted formatting when various message types are written out by `hop subscribe`. ### Added - Offloading of messages too large to be accepted by the Kafka broker to a companion REST service. This feature requires configuration of such a service by broker administrators, but can then operate transparently. - Support for sending messages with keys, which allows a degree of control over how messages are routed to topic partitions. See the [Kafka documentation](https://kafka.apache.org/documentation/#intro_concepts_and_terms) for details. ## [0.10.1] - 2024-09-23 ### Added - `setuptools` is explicitly listed as a dependency to work better in environments where it is not provided by default. ## [0.10.0] - 2024-07-31 ### Changed - The `hop.io.Producer` can be used to publish to more than one topic with a single instance. To support this capability, the `write` method now accepts an optional `topic` argument to direct to which topic to publish. - The error message resulting when attempting to open a stream with ambiguous credentials has been further clarified. ### Added - Users can now explicitly stop the `hop.io.Consumer`'s processing loop. (Thanks to @myNameIsPatrick) - A message model for GCN text format notices has been added. The original notice text is preserved unmodified, but parsing is performed to provide convenient programmatic access to the message fields in memory. ## [0.9.0] - 2023-11-10 ### Changed - Create Blob objects instead of crashing on binary data when deserializing - Fix handling of IPv6 addresses and default port numbers - Fix authentication procedure in RobustPublisher ### Added - Allow messages to be consumed without deserialization - Expose ability to commit offsets synchronously ## [0.8.0] - 2022-12-20 ### Changed - The command line tool now supports sending data which is piped to it either as JSON or as the raw BLOB format. ### Added - Messages are now sent with UUIDs, attached as the `_id` header, and the username of the sender as the `_sender` header. ## [0.7.0] - 2022-08-19 ### Added - Support for authentication using the "OAUTHBEARER" mechanism. - An optional `timeout` parameter for `hop.io.list_topics` which can be used to be used to prevent overly long or indefinite delays from this function when brokers are offline. ## [0.6.0] - 2022-06-28 ### Changed - All messages returned from `hop.io.Consumer` will now be instances of message model classes. This means that for receiving unstructured JSON messages (now handled by the `JSONBlob` class), users should retrieve the deserialized object(s) by accessing the `content` property of the message, rather than expecting the message to directly be the serialized object(s). - `hop.io.Producer` will now use a Kafka message header to indicate the format used for encoding each message it sends. `hop.io.Consumer` will read these headers, but also has backwards compatibility with the JSON envelope format used in previous versions. ### Added - Support for sending messages encoded using the [Avro container format](https://avro.apache.org/docs/current/spec.html#Object+Container+Files). - Messages not formatted as JSON can now be safely accepted by `hop.io.Consumer`, although it will not necessarily be able to provide useful deserialization of unregistered formats. Messages which are not recognized will be returned as instances of the `Blob` class, whose `content` property will contain the raw bytes of the message data. ## [0.5.1] - 2022-04-27 ### Changed - Remove message header encoding from `Producer` since confluent-kafka handles encoding. - Fix issue with parsing server hostname when user-provided hostname has a `kafka://` prefix. - Change `Producer.pack` from a private to public method - Remove Anaconda packaging in favor of conda-forge. ### Added - Add robust publisher functionality to allow the client to retry message sending by checking for message receipt by the server. - Add default client behavior to ignore messages designated as 'test' messages via a header key `_test`. - Add `flush` method to the `Producer`. - Add conda-forge packaging. ## [0.5.0] - 2021-10-11 ### Changed - Fix issue where subscribing to multiple topics only returned messages from a single topic. - Change naming for listening to messages forever from `--persist` to `--until-eos`. - Listen to messages forever by default within consumer (`until_eos=False`). - Change delivery callback to be on a per-message basis, rather than per stream. - Bump minimum adc-streaming version required to 2.x. - Remove correct conflicting cred in `add_credential()` when `--force` specified. ### Added - Add consistent accessors for all Auth properties. - Support read/write of optional credential fields. - Add topic listing as part of the python API. - Support message headers within producer. - Add headers to metadata provided by consumer. - Add logging throughout library. Within CLI, logging level is controllable via `--quiet` (no INFO) and `--verbose` (INFO + DEBUG). ## [0.4.1] - 2021-09-30 ### Changed - Pin adc-streaming to 1.x. - Support Sphinx 4.x for documentation. ## [0.4.0] - 2021-04-10 ### Changed - Consumer group IDs can no longer be specified as part of a broker URL, as the userinfo is used as a credential username. - Multiple brokers cannot be specified in a single URL, as this creates ambiguity in selecting credentials by hostname. ### Added - Give user directions when user has no configuration set up. - Add CLI subcommand `hop list-topics` to list available topics. - Allow multiple credentials in auth configuration. - Add command line interfaces for managing multiple credentials within `hop auth`. ## [0.3.0] - 2021-03-02 ### Changed - Generally increase user readability of error messages. - Fix issue in `hop subscribe` when printing JSON-formatted output. - Better handling of unstructured messages, avoiding excessive nesting when serializing messages. - Treat input from stdin as JSON for `hop publish`. - Create credential files with safe permissions by default. - Require that credential files have proper permissions, refusing to open configuration files which have over-broad permissions. - Modify how consumer group names are generated. When using proper authentication, prefix the group name with the credential username, instead of the local username. ### Added - Add documentation for optional parameters passed down to lower-level Kafka libraries when using `Stream`. ## [0.2] - 2020-09-16 ### Changed - Change `hop auth` to `hop configure`. - Return message content rather than a Blob model when unpacking unstructured messages. ### Added - Allow custom message formats via external plugins. - Add `hop configure web` to get credentials through web UI. - Add `hop configure setup -c <credentials>` to populate configuration with credentials. - Allow use of open Stream instances without context manager interface. - Expose manual committing of messages to allow fault tolerance within applications. ## [0.1] - 2020-07-24 ### Changed - Modify message serialization format passed through the wire to enable rapid message classification. - Simplify API both the CLI and python API, and provide consistency between both methods for publishing and subscribing to messages. - `Stream` now takes in various message models directly for publishing, and return the appropriate message model when grabbing messages, with a "catch-all" unstructured type. - Standardize API for message models throughout. - Authentication is now enabled by default for both the CLI and python API. Disabling authentication now needs to be done explicitly. - Update options in CLI and python API to reflect `adc-streaming` 1.0 release. ### Added - Add `hop version` to print out versions of hop-client dependencies. - Add `hop auth` for various authentication utilities, including setting up configuration for configuring various authentication options and showing where the default configuration location is. - Expose `Auth` class for providing custom configuration if not reading in auth configuration from a default location. ### Removed - Remove timeout option in subscribing to messages in CLI and python API, instead will listen to messages until an end-of-stream (EOS) is received. To listen to messages indefinitely, use persist option. - Remove options to pass in Producer/Consumer configuration options directly. Instead, specific options can be configured and a proper method for passing in authentication is exposed. ## [0.0.5] - 2020-05-02 ### Changed - Change package name from `scimma-client` to `hop-client` with entrypoint change from `scimma` to `hop`. ### Added - Allow subscribing of GCN circulars via Kafka via `scimma subscribe`. - Add `Stream` API to publish/subscribe to Kafka topics. - Define `VOEvent` and `GCNCircular` models to store, parse and serialize messages of their respective types. ## [0.0.4] - 2020-02-24 ### Added - Allow `scimma publish` to take in client configuration options via config file (-F /path/to/file) or config options (-X key=val). ## [0.0.2] - 2020-02-04 ### Added - Expose `scimma` CLI. - Allow publishing of GCN circulars via Kafka via `scimma publish`. - Expose API to read and parse an RFC 822 formatted GCN circular from a text file. ## [0.0.1] - 2020-01-24 ### Added - Initial Release. [Unreleased]: https://github.com/scimma/hop-client/compare/v0.5.1...HEAD [0.5.1]: https://github.com/scimma/hop-client/releases/tag/v0.5.1 [0.5.0]: https://github.com/scimma/hop-client/releases/tag/v0.5.0 [0.4.1]: https://github.com/scimma/hop-client/releases/tag/v0.4.1 [0.4.0]: https://github.com/scimma/hop-client/releases/tag/v0.4.0 [0.3.0]: https://github.com/scimma/hop-client/releases/tag/v0.3.0 [0.2]: https://github.com/scimma/hop-client/releases/tag/v0.2 [0.1]: https://github.com/scimma/hop-client/releases/tag/v0.1 [0.0.5]: https://github.com/scimma/hop-client/releases/tag/v0.0.5 [0.0.4]: https://github.com/scimma/hop-client/releases/tag/v0.0.4 [0.0.2]: https://github.com/scimma/hop-client/releases/tag/v0.0.2 [0.0.1]: https://github.com/scimma/hop-client/releases/tag/v0.0.1