New in version 0.12.0: Build System Changes: * Added CMake build infrastructure for Python wheel generation via scikit-build-core, enabling cross-platform builds and PyPI distribution. * Python wheels can now be built for Windows, Linux, and macOS using GitHub Actions CI/CD workflows. All wheels include the full set of encoding plugins (bzip2, gzip, lzma, flac, zzip) on every platform. * Added wheel packaging test (test_wheel.py) to verify encoding plugin availability in shipped wheels via cibuildwheel. * CMake build system modernization: updated to support modern CMake conventions, improved test harness integration, and better support for downstream projects using find_package(Getdata). Bindings Changes: * PYTHON: Updated to support NumPy 2.x. * PYTHON BUG FIX: Fixed encoding_support return codes. * PYTHON BUG FIX: Improved Python 3.x compatibility in test suite, including proper exception handling and path separators for cross-platform support. Miscellaneous: * Improved MSVC (Visual C++) compatibility for native Windows builds, including a fix for void pointer arithmetic in lzma.c (a GCC extension not supported by MSVC). * Updated build system to support gfortran with modern compiler flags (-std=legacy, -fallow-argument-mismatch). * Added CI/CD workflows for automated testing on Ubuntu, macOS, and Windows platforms. |=========================================================================| New in version 0.11.0: Library Changes: * BUG FIX: When using a bzipped /REFERENCE field, mixing gd_nframes() calls with reads of data from that field no longer results in subsequent reads to occur in the wrong place. Reported by Matthew Petroff. * BUG FIX FOR CVE-2021-20204: The first RAW field in a Dirfile having the same name as a previously defined field no longer results in a segmentation fault when trying to access the Dirifle's reference field. By default, the parser will return a syntax error (GD_E_FORMAT) when encountering a duplicate field name, making the DIRFILE invalid and avoiding the segmentation fault. To trigger the segmentation fault, the standard GD_E_FORMAT error return must be avoided, by doing one of the following: - The GD_IGNORE_DUPS flag must be used when invoking the format file parser. This stops the parser from generating an error when discarding out the duplicate field. - A caller-supplied parser callback must return GD_SYNTAX_IGNORE when passed the GD_E_FORMAT error generated by the duplicate field. (This is what happens in the checkdirfile utility, which is why it is affected by this bug.) API Changes: * A new function gd_open_limit() can be used to limit the number of open file descriptors used by the library to access RAW data in a given DIRFILE. GetData will try to keep under the limit by automatically closing RAW fields when necessary. Note: the library may exceed the limit in certain cases. See the man page for full details. * A number of symbols, which were deprecated in GetData-0.8, have been removed. This removal extends to the corresponding symbols in the bindings. The removed symbols, and their replacements, are: Removed symbol Replacement ------------------- ------------------- GD_FLOAT GD_FLOAT32 GD_DOUBLE GD_FLOAT64 GD_E_BAD_ENDIANNESS GD_E_ARGUMENT GD_E_BAD_PROTECTION GD_E_ARGUMENT GD_E_BAD_VERSION GD_E_ARGUMENT GD_E_FORMAT_NO_PARENT GD_E_FORMAT_NO_FIELD gd_bit_t int gd_spf_t unsigned int NOTE: The Python bindings have a different FLOAT type alias which corresponds to the native float type in Python. That symbol is not deprecated and has not been removed. * BUG FIX: Lingering references to the type gd_shift_t, which was deprecated in GetData-0.10.0 and replaced by gd_int64_t, have been removed from the API. Bindings Changes: * PYTHON BUG FIX: When first_sample is non-zero, and neither num_frames nor num_samples is indicated dirfile.getdata() no longer attempts to read past the end of the data file. Patch from Matthew Petroff. Miscellaneous * Running "make clean" no longer accidentally deletes most of the man pages from the man/ subdirectory. * New options --with-pcre and --without-pcre have been added to ./configure to adjust how the PCRE library is (or is not) used. * Modules now link to the main getdata library to avoid the potential for symbol look-up errors on load. Reported by Matthew Petroff. * Fortran bindings can now be built with gfortran version >= 10. |=========================================================================| New in version 0.10.0: Dirfile Changes: * Dirfile Standards Version 10 has been released. It adds three new field types: SARRAY, which is an array of STRING scalars (like a CARRAY, but with STRINGs instead of CONSTs), and INDIR and SINDIR, which are vector fields provide indexed look-ups from CARRAY (INDIR) or SARRAY (SINDIR) scalar arrays. It also adds field code namespaces, which can be specified with the new /NAMESPACE directive, or else as part of an /INCLUDE directive. * Some notes about namespaces: - Namespaces are separated from field names by a dot (.): namespace.name and can be nested arbitrarily deep, separating namespaces with intermediate dots: namespace.subspace.subsubspace.name Namespaces were created to provide an alternative to the prefix and suffix added to the /INCLUDE directive in Standards Version 9, which have some unfortunate side-effects due to their modifying field names directly. Because namespaces nest and are syntactically separate from field names, they do a better job of encapsulation. - If the namespace of a field is null (""), which is the default, then the dot separating the namespace from the field name may be omitted. - An /INCLUDE statement can specify a namespace which becomes the included fragment's "root namespace". The root namespace of the base (top-level) format file is always the nullspace (""), and cannot be changed. - In addition to root namespaces, there is also a "current namespace". At the top of a fragment, the current namespace is set to the root namespace. A /NAMESPACE directive can be used to change the current namespace to a subspace under the root namespace. That is, if the root namespace is "root", then the directive: /NAMESPACE subspace changes the current namespace to "root.subspace" regardless of what the current namespace was before. To change the current namespace back to the root namespace, use the null token ("") with the /NAMESPACE directive: /NAMESPACE "" - If no namespace is specified in an /INCLUDE line, then the current namespace becomes the root namespace of the included fragment. - Subnamespaces under the root namespace may also be specified directly in the name part of a field specification. As a result, it is never necessary to use the /NAMESPACE directive. - Every field code and name in a fragment implicitly gains either the fragment's root namespace or the current namespace. If the field code starts with an initial dot, then the root namespace is prepended to it, otherwise the current namespace is prepended to it. Because the current namespace is always a subspace of the root namespace, this means that metadata in a given fragment is never able to access other fields outside its own root namespace. - The exception to the above is the implicit INDEX vector, which ignores all namespaces attached to it, either implicitly, through the current namespace, or explicitly when specified in the metadata. This contrasts with the behaviour of the INDEX field in the presence of affixes, where it can appears or disappears based on the effects of the affixes creating or modifying the specific field name "INDEX". * Because syntactically the dot (.) now performs two functions, namely both separating namespaces from each other and field names, but also separating a field code from a representation suffix, there exists ambiguity in the syntax. To resolve the ambiguity, a new representa- tion suffix, .z has been added which does nothing. As an example, the field code: name.r is interpreted as the real part of the field named "name", assuming such a field exists. To indicate the field named "r" in the namespace "name", the field code: name.r.z must be used. Note that this ambiguity only exists in a Dirfile where both "name" and "name.r" are valid field names. If the field named "name" doesn't exist, then the first field code, "name.r" is unambig- uously interpreted as the field "r" in the namespace "name". * A note on the SINDIR field: Unlike every other vector field, this field produces character string data at the sample rate of it's input vector, which may be surprising, in certain instances. * GetData has supported FLAC compression since 0.9.0. Standards Version 10 now adds "flac" to the list of pre-defined encodings. Library Changes: * The function gd_array_len() no longer silently ignores a representation suffix in the field_code provided. In most cases, passing a representation suffix will now cause this function to fail with a GD_E_BAD_CODE error. * A couple of unnecessary malloc's have been removed from the field code search, reducing the chance of encountering a GD_E_ALLOC error. Notably, the functions gd_array_len(), gd_bof(), gd_entry_type(), gd_fragment_index(), and gd_spf() will no longer produce this error at all. * NOTE: The GD_VECTOR_ENTRIES symbol in gd_entry_list() and gd_nentries() calls does not match SINDIR entries, only numeric-valued vectors. This also affects the corresponding special case functions (gd_nvectors(), gd_vector_list() &c.) * BUG FIX: When building in ANSI-C mode, the computation of complex- valued RECIP fields is now correct. * BUG FIX: The gd_include() family of functions now correctly clean up after encountering an error. Previously, encountering a syntax error would result in these functions erroneously adding fields specified before the syntax error to the DIRFILE, with a bogus fragment index. Similarly, when these functions return GD_E_REFERENCE, they no longer add the included fragment (excluding the bad /REFERENCE directive) to the DIRFILE without telling the caller about it. * BUG FIX: gd_alter_protection() wasn't marking affected fragments as modified, meaning the change in protection level would be lost upon close unless other metadata changes were made to the fragment as well, or a flush of the fragment's metadata was triggered explicitly with a call to gd_rewrite_fragment(). * BUG FIX: The metadata update performed by gd_delete() now successfully updates all fields which used the deleted field as an input. Previously some fields could be skipped, leading to segfaults later when these fields were accessed. * BUG FIX: gd_add_spec() no longer creates an empty data file while failing with GD_E_PROTECT when operating in a fragment with data protection turned on. * BUG FIX: gd_putdata() now refuses to write to complex-valued LINTERP fields. Previously, the write would succeed as if the imaginary part of the field were zero. * BUG FIX: gd_nentries() now correctly rejects invalid values for the type parameter. * BUG FIX GetData now properly deals with circular series of aliases, turning them all into dangling aliases. Previously, the alias resolution would terminate at some arbitrary point around the loop, resulting in internal errors arising from attempts to use an alias as a field. * BUG FIX: Several bugs in the I/O positioning performed before reads and writes to compressed data have been fixed. Previously, reads and writes could occur in the wrong place in the data stream. Reported by S. J. Benton. * BUG FIX: Similarly, a number of bugs associated with random-access writes to compressed data files which were causing data corruption have been fixed. * BUG FIX: When reading LZMA-compressed data, gd_getdata() no longer hangs if liblzma returns only part of a multibyte sample. * BUG FIX: The FLAC encoding now works correctly with non-native endian data. * BUG FIX: Writes to ASCII (text) encoded files weren't properly updating the file's I/O position, leading to subsequent reads and writes occurring in the wrong place. * BUG FIX: Trying to open a non-existent, gzip-encoded data file now reports the correct error (GD_E_IO; "No such file or directory"), instead of segfaulting. Reported by Matthew Petroff. * BUG FIX: A segfault encountered when closing very large compressed files after writing to them has been fixed. * BUG FIX: Attempting to open a SIE-encoded data file a second time after a first attempt failed no longer results in a segfault. * BUG FIX: The parser no longer assumes a new string buffer returned by the parser callback (by assigning it to pdata->line) has the size given by pdata->buflen, which the callback is not required to update, but instead determines the buffer size directly. Previously, this assumption could result in a segfault within the parser. * BUG FIX: gd_add_polynom() and gd_add_cpolynom() no longer reject valid poly_ord values. * BUG FIX: The gd_[m]add() family of functions are now better at rejecting invalid data types. * BUG FIX: A segfault-on-error has been fixed in gd_[m]alter_spec. * BUG FIX: The gd_madd() functions longer accept aliases as parent field names. * BUG FIX: Setting n_fields (for LINCOMs) or poly_ord (for POLYNOMs) to an out-of-range value and then calling gd_free_entry_strings() no longer results in a segfault. * BUG FIX: A rare segfault has been fixed in gd_carrays() and gd_strings(). API Changes: * The new function gd_alloc_funcs() allows callers to change the memory manager used by GetData to allocate and de-allocate heap buffers that it returns. The functions gd_entry(), gd_error_string(), gd_fragment_affixes(), gd_linterp_tablename(), and gd_raw_filename() will use this memory manager to allocate the buffers they return. The function gd_free_entry_strings() will use this memory manager to free strings. Even if an alternate memory manager is specified, GetData will still use the Standard Library's malloc() and free() for much of it's internal storage. * A new function, gd_match_entries(), extends the functionality of gd_entry_list() and gd_nentries() by additionally allowing both searches restricted to a particular fragment and also regular expression matching against entry names. * A number of functions which used to return -1 on error now instead return an appropriate error code, previously only available through gd_error(). These error codes are all negative-valued. Functions whose error returns have changed and now do this are: gd_add(), gd_add_alias(), all the gd_add_() functions, gd_add_spec(), gd_alter_affixes(), all the gd_alter_(), functions, gd_alter_encoding(), gd_alter_endianness(), gd_alter_entry(), gd_alter_frameoffset(), gd_alter_protection(), gd_alter_spec(), gd_array_len(), gd_bof(), gd_delete(), gd_desync(), gd_dirfile_standards(), gd_discard(), gd_entry(), gd_entry_type(), gd_eof(), gd_flush(), gd_fragment_affixes(), gd_fragment_index(), gd_frameoffset(), gd_get_carray(), gd_get_carray_slice(), gd_get_constant(), gd_hidden(), gd_hide(), gd_include(), gd_include_affix(), gd_include_ns(), gd_madd(), gd_madd_alias(), all the gd_madd_() functions, gd_madd_spec(), gd_metaflush(), gd_move(), gd_nframes(), gd_open(), gd_parent_fragment(), gd_protection(), gd_put_carray(), gd_put_carray_slice(), gd_put_const(), gd_raw_close(), gd_rename(), gd_rewrite_fragment(), gd_seek(), gd_spf(), gd_sync(), gd_tell(), gd_unhide(), gd_uninclude(), gd_validate(), gd_verbose_prefix() * gd_add_indir(), gd_add_sarray(), gd_add_sindir(), gd_madd_indir(), gd_madd_sarray(), gd_madd_sindir(), gd_alter_indir(), gd_alter_sarray(), and gd_alter_sindir() have been added to manipulate metadata of the new field types. * gd_msarrays(), gd_get_sarray(), gd_get_sarray_slice(), gd_put_sarray(), gd_put_sarray_slice(), gd_sarrays() have been added to read and write SARRAY values. * gd_fragment_namespace() and gd_include_ns() have been added to read and write the root namespace of a fragment. * gd_put_string() now returns zero on success and a negative-valued error code on error, as well, following the lead of gd_put_constant() and gd_put_sarray(). Also note that the return type is now int, where previously it was size_t, despite the documentation having always claimed it returned int. * A new data type symbol, GD_STRING, exists to represent string data. The gd_native_type() function now returns GD_STRING for STRING fields (as well as for the new SARRAY and SINDIR fields). * The GD_FUNCTION_ALIASES block, and hence the long-deprecated GetData-0.6 API, has been removed from getdata.h. Anyone still using the GetData-0.6 API should modernise to avoid known problems with that API. * The gd_shift_t type, which was used for PHASE field shifts, has been deprecated. It has been replaced with gd_int64_t, which is what it was always typedef'd to. * BUG FIX: gd_getdata() and gd_putdata() now properly report GD_E_IO when an I/O error occurs while reading a LINTERP table. Previously GD_E_LUT would be erroneously returned, as if the table file were empty. Bindings Changes: * MATLAB: The FLAGS argument to GD_INCLUDE is now optional and defaults to zero. * C++, F77, F95, PERL: The bindings for gd_put_string() have changed to reflect changes in the C library. Fortran 77 no longer returns a n_wrote integer; Fortran 95 implements fgd_put_string a subroutine; C++ and Perl bindings now return integer zero on success, and negative on error. * IDL, MATLAB, PHP BUG FIX: Bindings for the C functions gd_raw_filename and gd_linterp_tablename now no longer leak the string returned by the C API. * PYTHON BUG FIX: A UnicodeEncodeError while assigning to dirfile.verbose_prefix no longer results in that attribute being set to None. Instead, it retains its former value, which reflects what actually happens in the underlying C library in this case. * PYTHON BUG FIX: Objects returned by dirfile.entry() and dirfile.fragment() weren't being initialised with the correct reference count, leading to memory leaks when they went out of scope. Reported by Alexandra Rahlin. Miscellaneous: * The --enable-assert configure option, which hasn't done anything for a long time, has been removed. * A new configure option, --disable-util, can be used to suppress building of the executables in the util/ subdirectory. * In the standard autotools build system, encodings which use external libraries for compression (gzip, bzip2, flac, lzma, slim, zzip, zzslim) are now by default built as dynamically loaded modules. To recover the old build behaviour, which put everything into the core GetData library binary, pass --disable-modules to ./configure. Using modules intro- duces a runtime dependency on GNU libltdl. The CMake-based build system used in the native Microsoft Windows source release retains the old (monolithic) behaviour. |=========================================================================| New in version 0.9.4: Library Changes: * BUG FIX: Arbitrarily-long reads of FLAC-encoded files now work. Previously, each FLAC frame was written to the start of the output buffer, overwriting the previous frame, and leaving most of the buffer uninitialised. Reported by S. J. Benton. * BUG FIX: Data read from FLAC-encoded 1-byte types are now correct. Previously, although all data requested was read, only the first half would be returned, encoded as 16-bit data (i.e., with a zero-byte between every sample). Bindings Changes: * PYTHON BUG FIX: An erroneous preprocessor definition which made the Python bindings unbuildable for Python 3 has been fixed. Reported by Akito Kusaka. Miscellaneous: * A work-around has been implemented in the configure script to avoid a bug in bash 4.1's parser. Reported by Akito Kusaka. * The way configure decides the Python module path has changed slightly to provide a better default under debian-based systems when using debhelper. Reported by S. J. Benton. |=========================================================================| New in version 0.9.3: Library Changes: * GetData can now read SIE files containing the optional nine-byte header (which can be created by daisie). When read by GetData, information in the header is completely ignored (because the GetData metadata contains all the necessary information). GetData never writes the header, but calls to gd_putdata() will preserve an existing header. Calls which re-code the file (like gd_alter_endianness(), gd_alter_raw(), &c.) will result in an existing header being deleted. * BUG FIX: Filenames and line numbers appearing in GD_E_FORMAT error strings returned by gd_error_string() are correct again. This bug also affected parser metadata sent to a registered parser callback. * BUG FIX: On platforms where char is signed, the library no longer rejects field names containing bytes with the top bit set. API Changes: * gd_verbose_prefix() can now be used on invalid dirfiles. Previously, this function would return GD_E_BAD_DIRFILE when passed an invalid DIRFILE pointer. * BUG FIX: When trying to access a LINTERP table file in a non-existent directory, GetData now reports the correct error (No such file or directory). Reported by Johanna Nagy. Bindings Changes: * PHP: PHP7 support has been added. * PYTHON: Python3 support has been added based on a patch from Matthew Petroff. The earliest supported Python3 version is 3.2. Python2 is still supported from version 2.4, but Unicode support must be enabled in Python2. Support for Python 2.3 has been dropped. * PYTHON: Under Python3, the bindings run into the issue of GetData not knowing the character encoding of Dirfile metadata (the C library just deals with bytes). As a result, under Python3, by default, most strings returned by the library are returned as encoded bytes() objects, instead of native Unicode str() objects. (This is true under Python2 as well, but less obvious since the native Python2 str() object is encoded.) To help deal with this, dirfile and entry objects now have a character_encoding attribute which can be set to inform pygetdata of the character encoding to use to decode the strings returned by the C library into Unicode strings. There is also a global pygetdata.character_encoding object which can be used to set the default encoding for newly-created pygetdata objects. See the module documentation for details. The default for character_encoding is None, implying no decoding should occur. In Python3, paths are handled separately (since the filesystem encoding may be different than the GetData metadata encoding). Error strings are decoded if possible, and then ASCII encoded to ensure they're always available, regardless of the capabilities of standard error. * PYTHON: As a side-effect to the above, pygetdata now accepts Unicode strings (both in Python2 and in Python3). The specified character_encoding will be used to encode them to C strings before being passed to the C library. If no character_encoding is specified, the current locale's default encoding will be used. * PYTHON: When using Python 2.6 or newer, calling repr() on a pygetdata.entry object now returns an eval()able string. * PYTHON: When using Python 2.7 or newer, a very small C API is produced for pygetdata. It is defined in pygetdata.h installed alongside the pygetdata module. This C API is needed by daisie, the stand-alone SIE encoding library. * C++ BUG FIX: Including another GetData header before getdata/dirfile.h no longer results in a fatal circular dependency. * PHP BUG FIX: The numbits parameter to gd_madd_sbit() is now optional and defaults to 1, as with other similar functions. * PHP BUG FIX: Fixed a memory leak on error in gd_getdata(). * PYTHON BUG FIX: Calling pygetdata.dirfile.[m]carrays no longer crashes if return_type=pygetdata.NULL. In this case, None is returned in place of the data arrays. * PYTHON BUG FIX: Entry objects returned by dirfile.entry() now properly represent scalar field codes. * PYTHON BUG FIX: An out-of-memory condition encountered by the underlying C library is now reported using the standard MemoryError exception. Previously, the bindings would attempt to instantiate a new pygetdata.AllocError exception when this happened, which wouldn't work if no memory was available. (The CPython interpreter pre-allocates an instance of MemoryError to deal with this situation.) For backwards compatibility, pygetdata.AllocError is now an alias for MemoryError. * PYTHON BUG FIX: a scalar field code string can now be assigned to the threshold attribute of WINDOW entries. * PYTHON BUG FIX: Attempting to delete various attributes from pygetdata objects now either succeeds or else raises an exception, instead of crashing. Miscellaneous: * The default install directory for the IDL, Perl, PHP, and Python bindings has changed: they are now all installed under ${exec_prefix} when possible. The install directories can still be overridden by ./configure options. |=========================================================================| New in version 0.9.2.1: Miscellaneous: * This release fixes one entry in the test suite (alter_entry_scalar3r) which was broken in the original 0.9.2 release. The library and bindings are unchanged (and report their version to be simply 0.9.2). The error in the test suite was reported by Dinar Valeev. New in version 0.9.2: Library Changes: * BUG FIX: When using a FLAC-encoded reference field, gd_nframes() no longer leaks file descriptors. * BUG FIX: A segfault in gd_entry_list() introduced in 0.9.1 has been fixed. Reported by Christian Trippe. * BUG FIX: A descriptor leak on error has been plugged in gd_desync(). * BUG FIX: A segfault triggered by encountering an I/O error while writing metadata has been fixed. * BUG FIX: Memory leaks in the FLAC and ZZIP encodings, plus single-byte leaks in gd_add_string() and gd_madd_string() have been plugged. * BUG FIX: gd_alter_entry() and gd_[m]alter_spec() no longer mangle the values of entry parameters when asked to remove scalar field codes. Reported by Dan Horák. Bindings Changes: * PYTHON BUG FIX: Fixed a potential segfault in dirfile.get_carray() on platforms where sizeof size_t != sizeof int. Reported by Dan Horák. * PHP BUG FIX: Calling gd_discard or gd_close on a persistent dirfile now does nothing, instead of closing the Dirfile and corrupting the persistent resource. |=========================================================================| New in version 0.9.1: Library Changes: * Functions returning entry lists (gd_entry_list(), &c.) or bulk scalar field data (gd_constants(), gd_carrays(), &c.) no longer run through the entry list twice, resulting in noticeable speed improvements when operating on large dirfiles. * BUG FIX: Top-level aliases pointing to metafields, and non-hidden aliases pointing to hidden entries are now included in the data returned by gd_nentries() and gd_entry_list() when they should be. * BUG FIX: Surprising combinations of hidden fields, meta fields, and aliases no long confuse the functions which return bulk scalar data (gd_constants, gd_strings, gd_carrays, &c.). Previously, given the right metadata, these functions would segfault, return incorrect data, or raise GD_E_INTERNAL_ERROR. * BUG FIX: The functions gd_delete(), gd_hide(), gd_hidden(), and gd_unhide() no longer segfault when asked to operate on a field that doesn't exist; instead, they properly report error (GD_E_BAD_CODE). * BUG FIX: gd_seek() and gd_tell() now always return -1 on error. Previously, they would return zero for some errors; they always still correctly set the dirfile error. * BUG FIX: Several memory or other resource leaks triggered by error returns have been fixed. * BUG FIX: GetData no longer ignores errors reported by zlib when writing to gzipped data files. In the past, this could, in some cases, cause GetData to wedge. API Changes: * BUG FIX: gd_constants() and gd_mconstants() now return NULL and set the dirfile error to GD_E_BAD_TYPE if passed GD_NULL as the return_type. Previously these functions would return NULL without setting an error when passed GD_NULL. * BUG FIX: gd_getdata() and gd_putdata() now truncate their operation to avoid overflow. If size_t is N-bytes wide, and GetData is operating on an M-byte wide data type, then truncation occurs at 2**(8N-M)-1 samples. Note, however, that truncation can also be triggered by intermediate data products, which may have wider data types (larger M) than the type specified by the caller. If truncation occurs, a short read/write will occur, without raising an error. Truncation occurs even when the data type is GD_NULL (i.e. when these functions have nothing to do). For GD_NULL, M is taken to be 1. Callers should check the return value of these functions, and make another call to handle the truncation, if necessary. * BUG FIX: Instead of seeking to a random place, gd_seek() now returns GD_E_RANGE when the specified offset, or the resultant position would overflow a 64-bit signed integer. Note, however, that, in most cases, the file size limit imposed by your filesystem or OS will be encountered first (in which case GD_E_IO will be returned). For comparison, the Windows NTFS driver limits files to 2**44 bytes, as does the ext4 filesystem. * BUG FIX: Similarly, gd_getdata() and gd_putdata() return GD_E_RANGE if asked to operate on data outside the addressable range (more than 2**63 samples past the start of the dirfile). Bindings Changes: * PYTHON: The return_type parameter to dirfile.get_constant() and dirfile.get_carray() is now optional, and will default to the native type of the data returned, as happens with dirfile.getdata(). * IDL BUG FIX: Using !GD.NULL with GD_GETDATA(), GD_GET_CONSTANT() and GD_GET_CARRAY() now works. In this case GD_GETDATA() returns the number of samples read (as in the C API), and the other functions return zero on success. * MATLAB BUG FIX: Using GD.NULL with gd_getdata(), gd_get_constant(), gd_get_carray(), and gd_get_carray_slice(), now works. In this case, gd_getdata() returns the number of samples read (as in the C API), and the other functions return zero on success. * PERL BUG FIX: Using $GetData::NULL with getdata(), get_constant(), get_carray(), and get_carray_slice() now works. In scalar context, getdata() returns the number of samples read, while the others return undef. In list context, they all return an empty array in this case. * PHP BUG FIX: Passing data with a GD_NULL data type now behaves as expected (i.e. the passed data is ignored). * PHP BUG FIX: Using GD_NULL with gd_getdata(), gd_get_constant(), gd_get_carray() now works. In this case, gd_getdata() returns the number of samples read (as in the C API), and the other functions return TRUE on success. * PYTHON BUG FIX: Using return_type=pygetdata.NULL with dirfile.getdata(), dirfile.get_constant() or dirfile.get_carray() now works. dirfile.getdata() will return the number of samples read (as in the C API), ignoring as_list. The other functions return None on success. * PYTHON BUG FIX: Reflecting the merge of GD_E_BAD_REPR into GD_E_BAD_CODE in the C API in 0.9.0, the BadRepr exception is now an alias of BadCode. Miscellaneous: * The build system now uses ExtUtils::MakeMaker instead of Module::Build to build the perl bindings. Module::Build was removed from the Perl 5 core in Perl 5.22. ExtUtils::MakeMaker is still a core module. The bindings are unchanged; the change in build prerequisites is the only difference users should notice due to this switch. * A new configure option, --disable-large-tests, will cause the test suite to skip running tests that write large amounts (>~10MB) of data. |=========================================================================| New in version 0.9.0: Library Changes: * Literals in format metadata may now have complex form (i.e. include a semicolon) when the parameter is purely real. However, a non-zero imaginary part is still an error. * gd_free_entry_strings() now NULLs pointers after freeing them. * gd_entry() now returns entry metadata when they contain scalar field codes which do not exist. In this case the GD_EN_CALC flag in the object will not be set. Previously, on such entries, this function would fail with the error GD_E_BAD_SCALAR, and return nothing. * gd_rename() now by default updates the target of ALIASes pointing to a renamed field to point to the new field instead of leaving them dangle. (But see GD_REN_DANGLE in the API section below). * CARRAYs are no longer truncated to GD_MAX_CARRAY_LENGTH elements. Flushing metadata to disk will now fail if writing a CARRAY would overflow a format file line. (It's platform specific, but format file lines are typically permitted to be at least 2**31 bytes long, so such an error usually indicates something pathological happening.) The GD_MAX_CARRAY_LENGTH symbol has been removed from the GetData header file. * Write support for bzip2-encoded and lzma-encoded data has been added. LZMA write support is only available for .xz files, not the obsolete .lzma format. The write support occurs out-of-place, just like how writing gzip-encoded data works. See the gzip discussion in the 0.8.0 section below for important notes. * A new encoding scheme using the Free Lossless Audio Codec (FLAC) to compress data has been implemented. For some datasets, it provides a good trade-off between speed and compression. Like gzip, bzip2, and lzma, is also uses out-of-place writes (see previous point). * A newly-created dirfile is now always opened in read-write mode, ignor- ing the access mode specified in the call. Previously, specifying both GD_RDONLY and GD_CREAT in open calls would result in an access mode (GD_E_ACCMODE) error if the dirfile didn't already exist. * Many functions which used to silently ignore representation suffixes in field codes passed to them no longer do that. Most of these will report an error (GD_E_BAD_CODE) if passed a representation suffix. The affected functions are: gd_bof, gd_entry, gd_entry_type, gd_eof, gd_flush, gd_linterp_tablename, gd_put_carray, gd_put_carray_slice, gd_putdata, gd_raw_close, gd_raw_filename, gd_seek, gd_spf, gd_sync, gd_tell. * The error code GD_E_BAD_REPR has been merged into GD_E_BAD_CODE. The symbol GD_E_BAD_REPR remains as an alias for GD_E_BAD_CODE, but is deprecated. * Attempts to seek past the end-of-field with gd_seek() now always succeed, although the resultant position is encoding specific. Previously, attempting to seek past the end-of-field on some encodings would return an error. * BUG FIX: The library now properly recovers from an I/O error while trying to open an unencoded datafile. Previously, such an error would poison the library's bookkeeping data, preventing all subsequent attempts to open that file unless the Dirfile was re-opened. Reported by Alexandra Rahlin. * BUG FIX: GetData no longer segfaults when trying to do a large forward seek before a write to a gzipped file. Reported by Joy Didier. * BUG FIX: gd_putdata() no longer ignores I/O errors while seeking to the first sample of a write. * BUG FIX: If the reference field is being written to, gd_nframes() now flushes it first before calculating the size of the dirfile. Previously a short count could result for some encodings in this case. * BUG FIX: Calling gd_putdata() to write gzip data with a non-zero starting offset equal to the field's current I/O position, no longer result in the call hanging. * BUG FIX: In addition to the addition of write support mentioned above, a number of problems with reading LZMA files has been fixed, which should result in fewer segmentation faults. * BUG FIX: The parser no longer silently appends a closing > to scalar field codes that contain an unmatched opening < (e.g. "scalar<3"). This is now interpreted as a simple field code (which may be rejected later due to the presence of the invalid '<' character). * BUG FIX: The parser no longer interprets various numbers as field codes when it shouldn't (e.g. when specifying a PHASE shift as "1." instead of "1"). * BUG FIX: When writing scalar field codes to disk which could be inter- preted as a number (e.g. the field code "1"), the library now forces the interpretation of these field codes as codes rather than numbers by appending a scalar index (making, e.g., "1<0>"), which is harmless. Previously, these were written as-is, resulting in misinterpretation the next time the Dirfile was opened. This only happens with Standards Version 8 or later, see the following for earlier versions. * BUG FIX: If the current Standards Version in effect is 7 or earlier, ambiguous field codes (e.g., "1"), are now rejected by gd_[m]add() and gd_alter_entry() with the error GD_E_BAD_CODE, since they can't be represented in the metadata on disk. For the behaviour with later Versions, and in permissive mode, see the previous. * BUG FIX: If performing a metadata update due to renaming fields (perhaps by passing GD_REN_UPDB to gd_rename()) results in an invalid field code due to affix restrictions, the update now fails (but see GD_REN_FORCE). Previously the invalid field code would be stored, leading to errors when flushing the modified metadata to disk. * BUG FIX: When performing a metadata update due to a renamed field, the field codes containing subfields of the renamed field are now also updated, including field codes specifying meta subfields which do not exist. * BUG FIX: reading a LINTERP table with fewer than two lines no longer results in a segfault on close/discard. * BUG FIX: gd_alter_raw() and similar no longer fail when asked to re- encode the data file of a RAW field which has not been previously accessed. * BUG FIX: A previously-read LINTERP table is now always discarded when changing table paths with gd_alter_linterp() or similar. Previously these obsolete, cached LUTs would sometimes linger, causing incorrect LINTERP computation. * BUG FIX: The I/O position reported by gd_tell and gd_seek for slim, zzip, and zzslim encoded data is now correct. API Changes: * CLARIFICATION: The macro GD_SIZE() declared in getdata.h is indeed part of the public API. It returns the size in bytes of a sample of data of a given type (e.g. GD_SIZE(GD_COMPLEX64) returns 8). It has been around since GetData-0.3.0, but has only been documented since GetData-0.8.3. * The comp_scal member of the gd_entry_t object has been replaced with a flags member, containing a flag (GD_EN_COMPSCAL) with the meaning of the former comp_scal member. There are also flags for hiddenness (GD_EN_HIDDEN) and whether the scalar entry codes in the field defi- nition have been dereferenced (GD_EN_CALC). * gd_[m]add() and gd_alter_entry() can now be used to set or change the hiddenness of a field by setting or clearing the GD_EN_HIDDEN bit in the supplied gd_entry_t object. * Two new rename flags have been added: - GD_REN_DANGLE which indicates the library shouldn't update ALIASes whose target has been renamed (instead it will turn them into dangling aliases) - GD_REN_FORCE which causes the library to skip updating field codes which would be invalid due to affixes instead of failing. * The move_data argument of gd_move() has been replaced with a flags argument which accepts the GD_REN_* flags, which have the same meaning as they do with gd_rename(). * gd_move_alias() and gd_delete_alias() have been deleted: their functions are now performed by gd_move() and gd_delete(), which now operate on the alias itself when given the field code to an alias, rather than the field the alias points to. * A number of different error codes which indicated the same problem (an I/O error returned by the operating system) have been merged into one. The error codes GD_E_OPEN, GD_E_TRUNC, GD_E_RAW_IO, GD_E_OPEN_FRAGMENT, GD_E_FLUSH are replaced by the new error GD_E_IO. The old symbols remain as aliases but are deprecated. The corresponding error strings also now include information from the underlying encoding library, where possible. There is one exception to this merge: attempts to flush metadata lines which are too long are now reported using GD_E_LINE_TOO_LONG. Previously, these errors used GD_E_FLUSH. * The error code GD_E_OPEN_LINFILE has also been removed. It has been split into two parts: - I/O errors resulting from reading the LINTERP table file are now reported using GD_E_IO; - Syntax errors in the table are reported using the new GD_E_LUT error code. GD_E_OPEN_LINFILE remains as a deprecated alias for GD_E_LUT. * gd_encoding_support() has been added to permit run-time determination of supported encodings. * gd_array_len() is the new name for gd_carray_len(). It now also handles STRINGs (which have a length of one). The gd_carray_len() name remains in the library, but has been marked deprecated. * BUG FIX: If the dirfile path provided cannot be resolved (due to, for instance, a symbolic link pointing to a non-existent path), gd_open() and friends now return the correct error code (GD_E_IO). * BUG FIX: gd_naliases() now returns an unsigned int, and zero on error, as documented. * BUG FIX: The API on 32-bit systems, which was broken in 0.8.7 and only partially fixed in 0.8.8, should now work as expected again. Bindings Changes: * PHP bindings have been added. * C++: There is no longer a default value for the "index" argument for Entry methods (including subclasses) which accept it (viz. Input, Scalar, ScalarIndex, Scale, CScale, Offset COffset, Coefficient, CCoefficient). The exception to this is with Entry subclasses for which zero is the only allowed value for the parameter. * F77 and F95: The bindings no longer raise SIGABRT when the dirfile space is exhausted. Instead they simply return a dirfile unit number referencing a static, invalid dirfile. * F77: Functions to add fields with named scalar parameters have been added (GDASBT GDASCL GDASCP GDASCR GDASLC GDASMX GDASPH GDASPN GDASRC GDASRW GDASSB GDASWD), but only for those field types which permit named scalars. Similarly, functions for altering field metadata with named scalars are also present (GDLSBT GDLSCL GDLSCP GDLSCR GDLSLC GDLSMX GDLSPH GDLSPN GDLSRC GDLSRW GDLSSB GDLSWD). These are provided as an alternative to using GDASCA after the fact. * IDL: The entry structure parser has been rewritten. It no longer requires members which it doesn't need, and is also a lot more lax about numerical data types. Notably, it now ignores a supplied COMP_SCAL member. Floating point parameters can be specified in either the base name (M, B, A, DIVIDEND) or else the member prefixed with 'C' (CM, CB, CA, CDIVIDEND), whatever numerical type. The bindings will ingest them appropriately. Also, N_FIELDS and POLY_ORD, may be omitted, and will be calculated from the supplied data. A scalar IN_FIELDS is treated like an single element array. * IDL: GD_REFERENCE is now a function, instead of a procedure, as the documentation has always claimed it was. It returns the current reference field (or the empty string, if there is none). The second parameter, the new reference field, is optional. (Previously the second parameter was required.) * PERL: The entry hash parser has been rewritten. It no longer requires keys which it doesn't need. * PERL: alter_entry() now only updates defined elements in the passed entry hash. * PYTHON: Building the python bindings now requires NumPy. Previously, NumPy support was optional. * PYTHON: for backwards compatibility, exceptions now exist for deprecated error codes (such as OpenError). These deprecated exceptions are simply aliases for the current ones and are never returned by the bindings. * C++ BUG FIX: The Entry methods Input, Scalar, and ScalarIndex (including subclasses) now return zero or NULL when passed an out-of- range index value. Previously they would return, variously, zero, NULL, another value for some other, valid index value, or segfault. * C++ BUG FIX: The flags parameter to Dirfile::Delete() is now unsigned, as it is in the C API. * F95 BUG FIX: fgd_add and fgd_alter_entry no longer ignore named scalar parameters provided in supplied entry structures. * PYTHON BUG FIX: Several memory leaks have been plugged. Patch from Matthew Petroff. Miscellaneous: * The minimum autotools versions have been bumped. Autoconf-2.65 or newer, automake-1.13 or newer, and libtool-2.2.7b or newer are now required to rebuild the configure script and associated build environ- ment. NOTE: In general, most people building GetData from a source release don't need the tools to build GetData; the autotools are only needed if changes need to be made to the configure script or Makefile input files provided in the release or if building from the repository. |=========================================================================| New in version 0.8.9: Library Changes: * BUG FIX: The metadata writer now correctly stores bytes in the range 0x01 through 0x1F to the format files (encoded as hex escape sequences: \x##). * BUG FIX: A number of memory leaks associated with error returns from library functions have been fixed. * BUG FIX: Attempting to create a new field or alias with the name of an existing dangling alias now fails with error GD_E_DUPLICATE, as it should. Reported by Alexandra Rahlin. * BUG FIX: Random-access reads on a RAW field with a frame offset no longer result in mispositioning of the field's I/O pointer, which previously would result in returning data from the wrong part of a field. Reported by S. J. Benton. * BUG FIX: The return value of gd_nframes() is now correct when using a sample-index encoded field as the reference field. * BUG FIX: The sample-index encoding no longer creates sequential records with the same value. Bindings Changes: * F77 and F95 BUG FIX: Passing zero as the field code length (F77) to GDFLSH GDSYNC GDRCLO, or, equivalently, an empty string as the field code to fgd_flush, fgd_sync, fgd_raw_close now causes operation on all fields, as with passing NULL to the corresponding C API functions does. This is what the documentation said these functions did all along. * PYTHON BUG FIX: Accessing the prefix or suffix member of fragment objects no longer leaks memory if the other affix is non-NULL. |=========================================================================| New in version 0.8.8: Library Changes: * BUG FIX: The incorrect handling of the explicit 64-bit API declarations in getdata.h (those associated with the GD_64BIT_API symbol), which prevented GetData-0.8.7 from compiling on 32-bit systems, has been fixed. GetData-0.8.8 is ABI compatible with GetData-0.8.7. |=========================================================================| New in version 0.8.7: Library Changes: * BUG FIX: Opening a Dirfile read-write (GD_RDWR) no longer fails if /INCLUDEd fragments are read-only. Reported by Alexandra Rahlin. * BUG FIX: Several fixes have been made to the sample index encoding (SIE) engine, which should now produce properly encoded data when performing random writes. Bindings Changes: * F95 BUG FIX: Passing the empty string to fgd_reference() now returns the current reference field without modifying it, instead of crashing. * PYTHON BUG FIX: Querying dirfile.reference no longer causes a crash on an empty dirfile. Reported by Alexandra Rahlin. * PYTHON BUG FIX: Memory leaks have been fixed in functions returning lists of field names or other metadata lists. Reported by Alexandra Rahlin. * PYTHON BUG FIX: On error, dirfile.get_string() no longer segfaults or leaks memory. |=========================================================================| New in version 0.8.6: Library Changes: * BUG FIX: Computation of LINCOMs with complex valued input fields now correctly happens in the complex plane. As a side effect, gd_native_type() now also correctly reports such LINCOM fields to be complex valued. * BUG FIX: The gd_[m]add() functions now ignore zero-length scalar strings. Previously they would store these invalid field codes, causing problems later. * BUG FIX: Returning complex-valued CARRAYs as purely real now works. Previously only the first element requested would be returned, the remaining output buffer containing uninitialised data. * BUG FIX: Entry members spf, bitnum, numbits, and period are now com- pletely ignored by gd_[m]add() when corresponding named scalars are specified. Previously, an invalid value in these members would result in the entry being rejected, even though the rest of GetData ignored the invalid, unused value. * BUG FIX: The parsing of the \x and \u escape sequences is now correct. * BUG FIX: A scalar field code specified for the last factor in a POLYNOM entry is no longer ignored by gd_[m]add(). * BUG FIX: gd_[m]add() no longer rejects MPLEX fields with negative count_val. * BUG FIX: DIVIDE fields with complex-valued divisors are now properly computed. * BUG FIX: Complex-valued POLYNOM and RECIP fields are now computed properly when the library is built in ANSI C mode. * BUG FIX: gd_alter_entry() no longer corrupts the DIRFILE when modifying named scalars of MPLEX fields. * BUG FIX: Writing complex-valued MPLEX fields no longer corrupts the stored data. * BUG FIX: gd_bof() now returns the correct number (i.e.: zero) when reporting the beginning of field of derived fields shifted to before the start of the dirfile. Previously, this function incorrectly returned values ranging from zero to one less than the samples-per- frame of the requested field. * BUG FIX: gd_flush(), gd_sync(), gd_raw_close() no longer segfault when operating on a LINCOM with only one input field. * BUG FIX: gd_seek() now works correctly on PHASE fields; previously, the sign of the PHASE shift was flipped. * BUG FIX: gd_seek() now correctly positions the virtual I/O pointer of the INDEX field. * BUG FIX: gd_framenum_subset() now returns the correct value when passed a field_end which is beyond the end of field, and then is required to extrapolate outside of the specified frame range. * BUG FIX: gd_error_string() now produces the correct string when reporting an out-of-range poly_ord encountered by gd_add_polynom() and similar. * BUG FIX: gd_[m]alter_spec() no longer ignore co-efficients specified for POLYNOM entries. * BUG FIX: gd_alter_encoding() now deletes the internal cache of RAW filenames of the affected fragment; previously, these old, cached filenames could lead to I/O errors when reading and writing the re- encoded RAW data files. * BUG FIX: Calling the Legacy API function GetFormat() on a Dirfile with MPLEX or WINDOW fields no longer results in a segmentation fault. * BUG FIX: Attempts to read past the EOF of a gzipped field no longer results in an I/O error, but successfully returns no data. * BUG FIX: The internal recursion counter wasn't being properly reset on certain error conditions, leading to spurious GD_E_RECURSE_LEVEL errors being returned from valid calls. Bindings Changes: * F77 and F95 BUG FIX: Named scalar indices are now indexed from one instead of zero, like all other array indices are in the Fortran GetData bindings. * C++ BUG FIX: Fixed segfault in RawEntry destructor. Reported by S. J. Benton. * C++ BUG FIX: Entry::ComplexScalars() now returns non-zero for RECIP entries when appropriate, instead of always returning zero. (The RecipEntry::ComplexScalars() method always returned the correct value.) * IDL BUG FIX: The /UPDATEDB flag to gd_rename is no longer ignored; also, the /MOVE_DATA flag no longer also acts as if /UPDATEDB had been specified. * PYTHON BUG FIX: The keyword for the "dirfile" parameter in the getdata.fragment constructor is now properly spelled. |=========================================================================| New in version 0.8.5: Library Changes: * BUG FIX: The zzslim encoding framework, which was inadvertently rendered non-compilable in 0.8.4, should once again work. Reported by Matthew Hasselfield. Bindings Changes: * PYTHON BUG FIX: A spurious debugging message accidentally left in the 0.8.4 release has been expunged. |=========================================================================| New in version 0.8.4: Dirfile Changes: * CLARIFICATION: Inconsistent behaviour in the library, coupled with contradictory statements in the Dirfile Standards, has resulted in confusion over the meaning of the final, optional, parameter in a MPLEX definition. To be clear: this parameter is simply the expected period between successive occurrences of the "count" value in the index vector. (It is used by GetData only to figure out a reasonable length for the MPLEX lookback.) The following clarifications and corrections have been made to the definition of the MPLEX field type in dirfile-format(5): - The final, optional parameter has been renamed from "max" to "period". - This parameter indicates not the maximal range of the index vector, as previously stated, but the expected number of samples between successive occurrences of the specified "count" value in the index vector; as before, it should be regarded as a hint, and does not place any actual restriction on the contents of the index vector. - The incorrect requirement that the "count" parameter be non-negative has been deleted. - The incorrect requirement that "count" be less than or equal to "max", if "max" is present and non-zero, has been deleted. - The meaning of an omitted or zero "period", indicating that the spacing of "count" in the index vector is unknown or non-uniform, is now explicitly stated. - Different MPLEX fields using the same index vector may specify different periods. See also the related library bug fixes below. Library Changes: * BUG FIX: The library no longer incorrectly rejects negative count_val MPLEX parameters. Furthermore, a count_val of -1 in gd_alter_entry or gd_[m]alter_mplex calls is not a special value: it just sets count_val to -1. * BUG FIX: The library no longer incorrectly rejects MPLEX fields where count_val is greater than the period (formerly called count_max). * BUG FIX: The default MPLEX period, if none is specified, is 2 * count_val + 1, not simply 2 * count_val, when count_val is greater than 5. * BUG FIX: Trying to position an I/O pointer to before sample zero with gd_seek() now properly fails. Previously it would allow it, causing bizarre things to happen later. * BUG FIX: Using gd_tell to get the I/O pointer position of a derived field with exactly two input fields (MULTIPLY, DIVIDE, MPLEX, WINDOW) now properly reports an error in the "multiposition" case (ie. when the two inputs are ultimately reading from different locations in the same RAW field). * BUG FIX: Trying to read sample zero of a gzipped RAW field the second time now works as expected. Previously, the request to reposition the I/O pointer back to zero would be ignored by the gzip framework. Reported by Alexandra Rahlin. * BUG FIX: The field code of a reference field declared in a subfragment included with affixes is no longer corrupted by spurious application of the affixes when the subfragment's metadata are (re-)written. Reported by Seth. API Changes: * The count_max member of the gd_entry_t object has been renamed to period. The corresponding dummy argument in various function proto- types has been similarly renamed. Bindings Changes: * C++: The Entry and MplexEntry member functions CountMax and SetCountMax have been renamed to Period and SetPeriod for consistency with the changes listed above. CountMax and SetCountMax are still available as aliases, but are marked deprecated. * IDL BUG FIX: GD_ENTRY structures representing MPLEX fields are now properly interpreted. Previously, the underlying gd_entry_t object was being incorrectly initialised. * PYTHON BUG FIX: Numpy arrays returned by dirfile.getdata() are now the correct length. Previously they would always be the length requested (or the length of the dirfile, if no length was explicitly given) even if fewer samples were returned. Extra elements would contain unini- tialised memory. |=========================================================================| New in version 0.8.3: Library Changes: * BUG FIX: Several bugs on big-ended systems have been fixed. Reported by Dinar Valeev. * BUG FIX: Adding an entry via gd_[m]add() with a negative CARRAY scalar index no longer results in an internal error or worse when the added field is later read. Instead, a proper error is returned. * BUG FIX: Changing a CONST field to a real floating point type using gd_alter_const() (or similar) no longer results in the value of the field being corrupted. * BUG FIX: A bug in the parser's tokeniser has been fixed to prevent spurious "unterminated token" syntax errors. Typically this would only be seen when using gd_strtok() to tokenise an (apparently innocuous) user-supplied string, but a carefully crafted format file could also be made to get gd_open() to produce it. Bindings Changes: * Bindings for MATLAB have been added. * The IDL, Perl, and Python bindings no longer require a C99 compiler to be built. |=========================================================================| New in version 0.8.2: Library Changes: * BUG FIX: A trailing symlink (i.e. situations where a symlink is the last element of a path) no longer confuses GetData when the target of the symlink: (1) is absolute, or (2) starts with "../". Reported by S. J. Benton. * BUG FIX: Trying to read data from the first sample of an MPLEX no longer results in an internal error if the first sample of the index doesn't match the target value. |=========================================================================| New in version 0.8.1: Library Changes: * gd_dirfilename() now returns a fully canonicalised version of the dirfile path. * BUG FIX: A segfault when negotiating symbolic links in file paths, typically manifesting in gd_open calls, has been fixed. * BUG FIX: gd_strtok now saves a copy of the string passed to it, as the documentation suggests it should. Previously it cached the pointer itself. * BUG FIX: A number of minor memory leaks, mostly occurring when the library encounters an error, have been fixed. * BUG FIX: /HIDDEN directives weren't being written for aliases. * BUG FIX: The parser now fails properly on /INCLUDEs which specify paths which don't point to regular files. * BUG FIX: Fixed a bug which would result in the parser getting confused over the current Standards Version, potentially leading to rejected valid dirfiles. Reported by Daniel Flanigan. * BUG FIX: gd_alter_entry() and gd_madd_alias() weren't clearing the Dirfile error before operation, resulting in them failing erroneously in certain situations. * BUG FIX: When including an existing fragment which itself has subfrag- ments, gd_include() no longer returns the wrong fragment index. * WIN32 BUG FIX: On Windows, the parser can now properly handle hexade- cimal floating point. API Changes: * How the API deals with field code affixes has changed. Input vector and scalar fields are reported by gd_entry(), &c. now include their prefix and suffix, if any, which should remove the need to do manual affix bookkeeping when reading metadata. The other side to this change is that when modifying metadata (gd_add(), gd_alter_entry(), &c.), supplied field codes must also contain the appropriate affixes. * A new function, gd_linterp_tablename() has been added which returns a fully canonicalised version of the look-up table pathname for a LINTERP. Bindings Changes: * F77 BUG FIX: A memory leak has been fixed in GDALLC. |=========================================================================| New in version 0.8.0: Dirfile Changes: * Dirfile Standards Version 9 has been released. It adds two new field types: MPLEX, which allows multiplexing multiple low-rate channels in one high-rate one, and WINDOW, which extracts a portion of a field based on the value of a second field. More changes are listed below. * Alternate field names may be defined using the /ALIAS directive. Aliases can't be used as a parent when defining metafields, but are otherwise indistinguishable from a field's original ("canonical") name. * The /VERSION directive now has fragment scope, which should alleviate some of the problems associated with "version leakage" found in previous versions. Previous versions are unchanged, but a "/VERSION 9" directive will not propagate back up into a parent fragment, nor will any /VERSION directive be propagated back up into a Version 9 fragment. * Fields may be "hidden" using the /HIDDEN directive. The Standards don't say what a "hidden" field does, but GetData will, by default, exclude them from the count and list functions. * Literal integers in the format may now be specified in octal (using 0####) or hexadecimal (using 0x##### or 0X#####) in addition to decimal. C99-standard hexadecimal floating point literals (0x##.##p##, &c.) are also accepted. * The /INCLUDE directive can take two additional, optional parameters which specify a prefix and/or suffix used to modify the entry names of the fields defined in the included fragment. (The intent of this is to permit the inclusion of multiple subdirfiles which define the same set of fields, by modifying their field names.) * Three new encoding schemes are available. The first is the Sample- Index Encoding (SIE), similar to run-length encoding, useful for compressing data which changes very rarely. Like the ASCII encoding, GetData implements it internally, so it's always available. The other two schemes (zzip and zzslim) are based around the ZZip library, an access library for PKWARE ZIP files. These are unusual in that they store all raw data in the same ZIP archive. * The /ENCODING directive takes an optional second token. The zzip and zzslim encoding schemes optionally use this token to specify the name of the combined ZIP archive. * The data type names FLOAT (an alias for FLOAT32) and DOUBLE (an alias for FLOAT64) have been deprecated. Library Changes: * On POSIX systems, GetData now caches the dirfile directory (and any other necessary subdirectories). This guards against third-parties renaming the directory while GetData is interested in it. * GetData is, in general, more careful about pathname management. At least on POSIX systems, it will now try to canonicalise all paths it comes across, including resolving symlinks. As a result, GetData no longer becomes befuddled by a caller who opens a Dirfile with a relative pathname and then calls chdir(2). Functions like gd_fragmentname() and gd_raw_filename() now return canonicalised paths. * The library no longer forces a sync of all raw data on close. The sync causes a flush of filesystem buffers, which is usually unnecessary overhead, and degredatious to efficient disk I/O. Use an explicit gd_sync() or gd_flush() before closing to recover the old behaviour, if desired. * The error messages output by GetData with GD_VERBOSE turned on now has "libgetdata: " prefixed to it. A further prefix may be specified by the caller. * When truncating a dirfile, the primary format file is no longer deleted and then recreated. Instead, it is simply truncated in-place. This change means that a Dirfile which is undergoing truncation always appears (to GetData, at least) to be a valid Dirfile, which can be useful if it is being concurrently read. * Write support for gzip-encoded data has been added. Writes occur out of place, which means sufficient space on the filesystem is needed for a second copy of every field being written. Writing to a gzip compressed field requires uncompressing the portion of the field before the write of the existing file, recompressing it to the temporary file, then writing the new data to the temporary file. A subsequent write at a later position, will continue to write to the compressed file, but a subsequent write to an earlier position requires coping all remaining data from the old file to the temporary file, finalising it, moving it over top of the old file, and then starting afresh with a new temporary file. As a result, it's not very fast, and likely should be avoided for non-sequential writes. * Raw data files are now only opened for writing if writing is going to occur. In particular, this means that opening a dirfile with GD_RDWR will still work even if you don't have write access to it, so long as you don't try writing to it. * GetData now tracks a "current location" for all RAW files, and allows reads and writes to occur at the "current location" (via the special symbol GD_HERE). For some derived fields the idea of a "current location" doesn't make sense, and this won't work. * BUG FIX: In previous versions, renaming a field with gd_rename() wouldn't update or invalidate the cache of fields which depended on the old name, leading to surprising results. It will now either invalidate these fields' metadata caches or else change the name in those definitions as well. * BUG FIX: The invalid syntax "/META parent/child grandchild ..." is now properly rejected by the parser. Previously it half-worked, resulting in a segfault in gd_close(). * BUG FIX: Several syntax errors encountered in PHASE and RECIP field specifications weren't being reported by the parser, leading to corrupted metadata on open. * BUG FIX: GetData no longer omits the input field to a RECIP when writing metadata to disk. Nor does it forget to put newlines after RECIP or DIVIDE field specifications. * BUG FIX: GetData no longer assumes all LINCOM input fields have the same number of samples per frame. * BUG FIX: Writing an empty fragment with GD_PRETTY_PRINT turned on no longer results in a floating-point exception. * BUG FIX: Renaming a field now properly flags the fragment which contains it as dirty, which will ensure the rename is written to disk when the metadata are next flushed. * BUG FIX: Whitespace within tokens is now properly escaped when metadata is flushed to disk. * BUG FIX: The parser no longer gets tripped up by two character escape sequences in a row, nor by an escape sequence at the start of a token. * BUG FIX: "/PROTECT data" directives are now being written when they should be. Previously, they were dropped completely. Reported by Alexandra Rahlin. * BUG FIX: CARRAY indices provided to gd_add() in the entry.scalar_ind member are no longer ignored. Reported by S. J. Benton. * BUG FIX: When adding a metafield with gd_(m)add(), GetData no longer checks the protection of the fragment indexed by entry->fragment_index. Instead it ignores entry->fragment_index completely (as the documentation indicates it should) and checks the protection of the fragment containing the parent field. * BUG FIX: calling gd_putdata() with num_frames and num_samples both zero no longer confuses GetData: instead it simply does nothing. API Changes: * Functions which add fields (gd_add(), gd_add_(), &c.) can now be used to add metafields by using their full "parent/name" field code. The old, explicit metafield adding routines (gd_madd(), &c.) are still available. * As with their corresponding names in the Standards, the gd_type_t symbols GD_FLOAT and GD_DOUBLE are now deprecated. Use GD_FLOAT32 and GD_FLOAT64 as appropriate. * A new open flag, GD_TRUNCSUB, will cause GD_TRUNC to descend into subdirectories of a directory it is truncating. If not specified along with GD_TRUNC, it does nothing. * New public functions gd_nentries() and gd_entry_list() provide a generic interface to the field counting and list functionality. The other functions (gd_nfields(), gd_nfields_by_type(), &c. and gd_field_list(), gd_field_list_by_type(), &c.) are now simply special cases of these two functions. * The "hiddenness" of a field name may be queried with gd_hidden(), set with gd_hide() and cleared with gd_unhide(). * /INCLUDE affixes may be queried with gd_fragment_affixes() and added, modified or deleted with gd_alter_affixes(). A new fragment can be added to the dirfile with affixes using gd_include_affix(). * New aliases may be added with gd_add_alias() or gd_madd_alias. The target of an alias is returned by gd_alias_target(); a list of aliases of a field, or their number, is returned by gd_aliases() and gd_naliases(). * Because GetData considers aliases and canonical names to be indistinguishable, passing an alias name to most functions will result in the function operating on target field. To manipulate aliases themselves, the API has the new interfaces gd_move_alias() and gd_delete_alias(). * In addition to the special symbol GD_HERE which may be passed to gd_getdata() and gd_putdata(), the "current location" of fields may be queried with gd_tell() and moved with gd_seek(). * The typedefs gd_bit_t and gd_spf_t have been deprecated. They are replaced with 'int' and 'unsigned int' respectively. For backwards compatibility, the typedefs (with their new types) are still declared in getdata.h, but the API no longer makes use of them. * GetData's tokeniser has been exposed to the public API via the gd_strtok() function, which works similar to strtok(3), and returns successive tokens of a string given it. * A new function gd_error_count() returns the number of GetData errors encountered on a Dirfile since it was last called (or since the Dirfile was opened). This is useful for programs that care whether *something* failed, but don't really care what it was. * Reading MPLEX fields can result in inefficiencies due to GetData having to look backwards for the first value of the part of the derived field requested. A new function, gd_mplex_lookback(), can be used to alter how much lookback is done, if any. * The "move_data" flag in gd_rename() has been replaced with a generic "flags" parameter. Two rename flags are defined: GD_REN_DATA, which recovers the old behaviour of move_data, and GD_REN_UBDB which will cause references to the renamed field to be renamed also in derived fields which refer to it. Without GD_REN_UBDB, these definitions are left unchanged. * Flags which affect the long-term operation of GetData may be modified after open using gd_flags(). Currently these flags are just GD_VERBOSE and GD_PRETTY_PRINT. * The gd_flush() function, which both syncs and closes RAW files, has been broken up into its two parts with two new functions: gd_sync(), which just syncs and gd_raw_close(), which just closes. gd_flush() still does both. * Using the new gd_desync() function will cause GetData to determine whether the Dirfile metadata on disk has changed since the Dirfile was opened. * A prefix to the automatic error messages printed by GetData when using the GD_VERBOSE flag may be specified with gd_verbose_prefix(). * When requesting the exclusive creation of a new dirfile with (GD_CREAT | GD_EXCL) GetData will now return the new error GD_E_EXISTS if the dirfile already exists. Previously it would return GD_E_CREAT in this situation. * The error codes GD_E_BAD_ENDIANNESS, GD_E_BAD_PROTECTION, and GD_E_BAD_VERSION have been combined into a more generic error code: GD_E_ARGUMENT. The old symbols are retained for backwards compatibility but have been marked deprecated. * CONST and CARRAY handling has been combined somewhat. When passed to {get,put}_carray[_slice]() or gd_carray_len(), CONSTs behave like length-one CARRAYs. Similarly, {get,put}_constant() will operate on the first element of a CARRAY, as if it were a CONST field. * C89 API: Passing NULL to gd_alter_crecip() for cdividend is now treated as if it were passed zero (ie. it indicates no change for cdividend). * BUG FIX: gd_spf() now returns GD_E_DIMENSION when passed a scalar field code, as the documentation says it should. Previously it returned GD_E_BAD_FIELD_TYPE. Legacy API Changes: * The never-used n_mplex/mplexEntries part of the FormatType now has a use: both MPLEX and WINDOW entries are reported there. Bindings Changes: * Bindings for Perl5 have been added. Perl is not well suited to numerical analysis, and the bindings may be inefficient. * F77: Due to small namespace issues, the encoding parameters (corresponding to the GD_...ENCODED symbols in the C API) have all been renamed. They are now of the form GDE_xx. * C++: The bindings now make use of the explicitly-64-bit offset type gd_off64_t defined in getdata.h. Previously, the bindings tried to force off_t to be 64-bit through preprocessor shenanigans, which led, in certain cases, to linking problems down the road when used by third parties. * Python: The return_type, num_fields, and num_samples parameters to dirfile.getdata() are now optional. The return_type defaults to pygetdata.FLOAT, and if neither num_fields nor num_samples are given, all frames (ie. the value of dirfile.nframes) are returned. * IDL: A number of undocumented, unnecessary function aliases have been removed. They should be replaced with their canonical names, without change in use: Alias Replacement ---------------------- ---------------- GD_ADD_CLINCOM GD_ADD_LINCOM GD_ADD_CPOLYNOM GD_ADD_POLYNOM GD_ADD_CRECIP GD_ADD_RECIP GD_ALTER_CLINCOM GD_ALTER_LINCOM GD_ALTER_CPOLYNOM GD_ALTER_POLYNOM GD_ALTER_CRECIP GD_ALTER_RECIP GD_FIELD_LIST_BY_TYPE GD_ENTRY_LIST GD_FRAMENUM_SUBSET GD_FRAMENUM GD_MADD GD_ADD GD_MADD_BIT GD_ADD_BIT GD_MADD_CARRAY GD_ADD_CARRAY GD_MADD_CLINCOM GD_ADD_LINCOM GD_MADD_CPOLYNOM GD_ADD_POLYNOM GD_MADD_CRECIP GD_ADD_RECIP GD_MADD_DIVIDE GD_ADD_DIVIDE GD_MADD_LINCOM GD_ADD_LINCOM GD_MADD_LINTERP GD_ADD_LINTERP GD_MADD_MULTIPLY GD_ADD_MULTIPLY GD_MADD_PHASE GD_ADD_PHASE GD_MADD_POLYNOM GD_ADD_POLYNOM GD_MADD_RECIP GD_ADD_RECIP GD_MADD_SBIT GD_ADD_SBIT GD_MADD_STRING GD_ADD_STRING GD_MALTER_SPEC GD_ALTER_SPEC GD_MCONSTANTS GD_CONSTANTS GD_MFIELD_LIST GD_ENTRY_LIST GD_MFIELD_LIST_BY_TYPE GD_ENTRY_LIST GD_MSTRINGS GD_STRINGS GD_MVECTOR_LIST GD_VECTOR_LIST GD_NFIELDS_BY_TYPE GD_NENTRIES GD_NMFIELDS GD_NENTRIES GD_NMFIELDS_BY_TYPE GD_NENTRIES GD_NMVECTORS GD_NVECTORS * BUG FIX: All bindings now provide a named constant corresponding to the C API error GD_E_UNKNOWN_ENCODING, which was overlooked in previous releases. * IDL BUG FIX: The missing /IGNORE_REFS and /PRETTY_PRINT are now available in gd_open. |=========================================================================| New in version 0.7.3: Library Changes * BUG FIX: Passing a zero dividend to alter_crecip() and alter_recip() now results in the dividend not changing, per documentation. * BUG FIX: GetData wasn't properly computing the module directory, causing encodings using external modules to fail with GD_E_UNSUPPORTED. * BUG FIX: Metafield records are now properly stored in the format file for Standards Version 8. * BUG FIX: gd_raw_filename() wasn't clearing the Dirfile error before operation, resulting in it failing erroneously in certain situations. Utilities Changes: * BUG FIX: A potential segmentation fault has been patched in dirfile2ascii. |=========================================================================| New in version 0.7.2: Library Changes * If built with modules, the plugin shared objects will now be installed in a separate directory (by default "${libdir}/getdata", but see the --with-module-dir option in ./configure). * BUG FIX: When using modules, a missing plugin no longer results in a lock up the second time an attempt is made to find it. * BUG FIX: GetData now properly processes /INCLUDE directives with absolute paths, as the documentation insists it should. * BUG FIX: gd_add()ing the first raw field in a dirfile to a subfragment no longer results in memory corruption when calling gd_close(). API Changes * If called with GD_CREAT but no encoding specified, gd_include() now will duplicate the encoding of the parent fragment (if any). Miscellaneous * The python module install dir can now be changed by passing --with-python-module-dir to configure. |=========================================================================| New in version 0.7.1: Library Changes * BUG FIX: When the parser fails in a gd_include() call, the internal list of conformant Standards Versions for the loaded dirfile is no longer corrupted. * BUG FIX Performing two consecutive reads of the same field in an ASCII encoded datasources no longer confuses GetData. * BUG FIX: Numerous memory leaks have been patched (mostly pin-holes, although there are a few, rare but egregious ones). API Changes: * The value(s) of a CONST or CARRAY field added with gd_[m]add() is now guaranteed to be zero; previously, it was unspecified. * BUG FIX: Specifying both GD_VERBOSE and GD_IGNORE_DUPS no longer results in spurious "Field code already defined" messages when dupli- cate fields are encountered. * BUG FIX: Calling gd_delete on a metafield no longer results in a segfault. Bindings Changes: * F77 BUG FIX: A memory leak in GDASCA has been fixed. * Python BUG FIX: The first element of a python list (instead of a NumPy array) is no longer dropped when passed to pygetdata on 64-bit systems. Utilities Changes: * dirfile2ascii now pads reads past the end-of-field. The value used to pad may be specified with '-z', and defaults to 0/NaN. Previously, the value of uninitialized memory was output. * BUG FIX: dirfile2ascii now does what the user intends when presented with cart-before-horse arguments, ie. "dirfile2ascii -e field DIRFILE". Previously this resulted in confusion. * BUG FIX: dirfile2ascii now accepts field conversion '-u', which was documented but overlooked in the argument parser. |=========================================================================| New in version 0.7.0: Dirfile Changes * Dirfile Standards Version 8 has been released. It adds three new field types: DIVIDE, which computes X/Y for vectors X and Y, RECIP, which computes A/X for vector X and scalar A, and CARRAY, which is an array of CONST scalars. A few other changes are listed below. * Like CONST fields, CARRAY elements can be used as parameters in the definition of other fields. Syntax is 'field', where 'field' is the field name and 'n' the element index in the CARRAY (and the angle brackets are literal angle brackets). If the '' is omitted, the first element (numbered 0) is assumed. * Slashes on reserved words are now mandatory, allowing fields with reserved words as names. * The long-deprecated single character data type codes have been removed from Standards Version 8. (They are still accepted by GetData, which handles all previous Standards Versions as well.) * The /ENDIAN directive now takes an optional second parameter. The only allowed value for this second token is "arm", which indicates that double precision floating point data (including double precision complex data) are stored in the ARM middle-endian format. Without this, floating point data are assumed to have the same byte sex as integer data. Library Changes * GetData, including its bindings, is now supported under MacOS X, Cygwin, and Win32 using MinGW. In the case of MinGW, this includes linking against the Microsoft C Runtime, which isn't POSIX compliant. Users should exercise caution when writing to dirfiles in this case. * GetData can now be used on ARM processors. As a side-effect, GetData can now read and write the middle-ended floating point format used by older ARM processors (on any processor, via endianness conversion). * Some changes have been made to the parser. It now operates in one of two modes: pedantic mode, which adheres strictly to one particular Dirfile Standards Version, and permissive mode (the default) which is more lenient, as the old parser was. Encountering a /VERSION directive will switch the parser from permissive to pedantic mode unless told not to (see GD_PERMISSIVE below), and will change the particular Standards Version to which pedantic mode is adhering. * As a result of the previous change, the Standards compliance of the parser in strict mode has been greatly increased. This affects allowed characters in field names, and field name lengths. The existence or lack thereof of the implicit FILEFRAM field (an alias for INDEX) is also affected. * Since Standards Versions 4 and earlier were never codified, we've made some assumptions as to their behaviour in certain undocumented situta- tions. Most noticeably, they still prohibit ASCII control codes (bytes 0x01 through 0x1F) and '/' in field names. We also assume these dirfiles have native byte sex, even though some early GetData behaviour implied only little-ended data was acceptable. * Every Dirfile object now records its current Standards Version, which affects functions such as gd_add_spec(). The Version will start out as the value of the last /VERSION directive encountered by the parser, or else the latest compliant Standards Version. It can be changed (see below). * When writing metadata to disk, fragments will be written according to the current Standards Version of the dirfile object, rather than always using the latest version, as before. * The C library now sports an "ANSI C" mode, which allows compilation of GetData without a C99-compliant compiler. Some functionality may be lost, most prominently, but unsurprisingly, the default (C99) API. * The artificial maximum line length of 4096 bytes for format metadata has been removed. The maximum line length is now 2**31 bytes on 32-bit systems, and more on 64-bit systems. This also has the side-effect of removing the same length limit which had been applied to field names. * LINTERP tables are no longer required to be sorted. The library will sort them, if needed. As a side-effect of this, tables are no longer read twice when loading, leading to potential speed-up. * BUG FIX: The library wasn't properly checking that the second and higher input fields in LINCOM and MULTIPLY fields were, in fact, vectors, leading to possible segfaults. * BUG FIX: A memory leak associated with modifying LINCOM metadata has been fixed. * BUG FIX: Using a complex valued vector as the second input field in a MULTIPLY wouldn't result in the field being internally flagged as complex, leading to loss of the imaginary part in certain situations. * BUG FIX: Trying to write to a LINTERP via a table which isn't monotonic now results in error. Previously, this was allowed, with arbitrary data stored to disk. * BUG FIX: On encountering a line longer than it was prepared to deal with, the parser used to get very confused, potentially resulting in bizarre behaviour. On the off chance such a line is encountered, GetData will now raise GD_E_LINE_TOO_LONG in these cases. (And, since the maximum line length has been increased to at least 2**31 charac- ters, this should only occur in pathological situations.) * BUG FIX: Deleting a metafield was not being communicated to its parent, resulting in differing opinions within the library as to the deleted field's existence. API Changes * Due to namespace conflicts with the MacOS X System Library (which contains the C Standard Library), a wholesale renaming of public symbols has been performed. All public functions, and data types now begin with 'gd_'. All public preprocessor macros now begin with 'GD_'. Symbol renaming has occurred in the following manner: - functions which started with 'put' simply prefix 'gd_' to their name. So put_constant() becomes gd_put_constant(). - functions which started with 'get' replace 'get' with 'gd', except those functions with a corresponding 'put' function; these simply prefix 'gd_'. So, get_constants() becomes gd_constants(), but get_constant() becomes gd_get_constant(). - functions which started with 'dirfile' replace 'dirfile' with 'gd', except for dirfilename(), which becomes gd_dirfilename(). So, dirfile_alter_encoding() becomes gd_alter_encoding(). - macros which started with 'GETDATA_' now start with 'GD_'. Those that started with 'NO_GETDATA_' now start with 'GD_NO_'. So GETDATA_LEGACY_API becomes GD_LEGACY_API and NO_GETDATA_LEGACY_API becomes GD_NO_LEGACY_API. * The above renaming rules result in one gd_ namespace clash. As a result, get_reference() has been removed completely. Calls will have to be changed to use gd_reference(), formerly dirfile_reference(). * Furthermore, the function dirfile_protect() is now gd_alter_protection() to match the naming of other fragment metadata altering functions. * To ease transition to the new namespace, defining GD_FUNCTION_ALIASES before including getdata.h will create a bunch of preprocessor macros which map the old names to the new. * The meaning of the GD_PEDANTIC flag to gd_[cb]open() has changed. It now forces the parser to start off in pedantic mode (see above), following the latest Standards Version, rather than the default permissive mode. This is a subtle change, and in most cases the effects of GD_PEDANTIC will be the same as they were before. * A new dirfile flag, GD_PERMISSIVE, has been added. Passing this to gd_[cb]open() will prohibit the parser from switching into pedantic mode when it encounters a /VERSION directive. This flag is needed to read dirfiles which claim to adhere to the Standards via a /VERSION directive, but contain invalid syntax nonetheless. * The current Standards Version of an open dirfile can be set or queried by calling the new function gd_dirfile_standards(). NB: this only affects the open dirfile, not the metadata stored on disk. * A new function, gd_rewrite_fragment() will force GetData to write the specified fragment to disk, even if no metadata in the fragment has changed. * Two new functions, gd_bof() and gd_eof() can be used to find the sample number of the beginning- or end-of-field for a given field. Notably, these can be used to calculate the amount of raw data on disk for a given field. * The new gd_invalid_dirfile() function will return a newly created, invalid dirfile. Primarily useful to callers as a "null dirfile" placeholder. * GD_E_OPEN_INCLUDE has been renamed to GD_E_OPEN_FRAGMENT, but the former symbol remains as an alias. * Two new bitflags to gd_cbopen and gd_open: GD_ARM_ENDIAN and GD_NOT_ARM_ENDIAN allow specifying the default ordering of double pre- cision data. These flags may also be used with gd_alter_endianness() and will be returned by gd_endianness(). On all platforms except for middle-endian ARMs, GD_NOT_ARM_ENDIAN equals zero and may be omitted. * Where gd_flush() and gd_metaflush() in the past raised GD_E_OPEN_INCLUDE on I/O error, they now raise the new GD_E_FLUSH. * All functions now raise GD_E_DIMENSION when encountering a scalar field when expecting a vector. In the past, the error value returned was different for different functions. * getdata.h now wraps itself in an extern "C" block if included by a C++ compiler. * If passed a NULL pointer, gd_error_string() will return a newly malloc'd buffer long enough to hold the entire string. It should be free'd by the caller. * The parser data passed to the callback function now includes the length of the buffer. The parser also permits the callback to assign a new pointer to the 'line' element of the parser data if the buffer passed in is too small. * gd_metaflush() now fails and raises GD_E_ACCMODE if called on a read-only dirfile. (Previously it would successfully do nothing.) Legacy API Changes * In the legacy format struct, DIVIDE fields are listed as MULTIPLYs and RECIP fields are listed as LINCOMs, with the dividend stored in m[0]. * BUG FIX: In previous versions, attempting to access an invalid dirfile (anything that would cause gd_open to fail) resulted in leaked memory. This leak has been plugged. * BUG FIX: A segfault involving the reporting of POLYNOM fields has been fixed in GetFormat(). Bindings * Bindings for the functionality introduced into 0.7.0 have been added. * F77: GDPROT has been renamed GDAPRT to mirror the renaming of gd_alter_protection(). * F77: GDGTCO and GDPTCO no longer return an indicator of success. * F77: The signatures of GDGSCA and GDASCA have changed to return/take the scalar index as well. * F95: For consistency sake, the Fortran 95 bindings have undergone the same renaming so they have the same name as their C counterparts. All functions are now preceded by 'fgd_'. * F95: The fgd_get_constant_* and fgd_put_constant_* procedures are now subroutines, not functions. * C++: Const member functions are now flagged as such. * C++: The len parameter to Dirfile::ErrorString is now ignored. * C++: dirfiles are now opened read-only by default. * IDL: The IDL bindings have also followed suit in the great renaming. The exception to this is GETDATA_CONSTANTS(), which is not renamed. * IDL: Various constants defined in getdata.h but not used by the IDL bindings have been removed from the structure returned by GETDATA_CONSTANTS(). * Python: dirfile.getdata() now raises ValueError if neither num_frames nor num_samples are specified. In the past, it would happily return nothing in this case. * Python: dirfiles are now opened read-only by default. * C++ BUG FIX: Entry::Scalar() no longer rejects valid values of index. * IDL BUG FIX: A memory impropriety involving keyword arguments. This bug manifested by rejecting valid keyword arguments on 64-bit systems, but may have had other, subtle effects on 32-bit systems as well. * Python BUG FIX: A non-literal phase shift is now properly reported. * Python BUG FIX: Calling Dirfile.discard() or Dirfile.close() no longer results in a segfault. Miscellaneous * The --disable-checkdirfile option has been removed from ./configure. It is now always installed. * A utility, dirfile2ascii, is now installed alongside checkdirfile. This utility converts all or part of a dirfile into ASCII text. dirfile2ascii was written by Matthew Truch. * The thoroughness of the checks done by checkdirfile have increased. It now reports format syntax errors, Dirfile Standards compliance, problems with field definitions (such as missing input fields), and the size of the dirfile. |=========================================================================| New in version 0.6.3: Bindings * A memory leak has been plugged in the Python bindings manifesting in calls to dirfile.getdata() returning a NumPy array. In previous versions, the memory used by the returned array to hold the data was not reclaimed by GC. |=========================================================================| New in version 0.6.2: Library Changes * Attempting to get the number of frames from an ASCII file no longer fails the second time. |=========================================================================| New in version 0.6.1: API Changes * The getdata.h header no longer includes complex.h explicitly. Legacy API Changes * A potential segfault relating to error reporting in the legacy API has been fixed. Bindings * C++ BUG FIX: Several bugs preventing compilation of the C++ test-suite under Fedora 13 have been fixed. * IDL BUG FIX: Several segfaults in the IDL bindings have been eradicated. Miscellaneous * Libtool has been upgraded to 2.2.6b, which provides proper support for Fortran-9x. The outdated, internal libltdl has been removed. Building modules now requires an installed libltdl. |=========================================================================| New in version 0.6.0: License * The GetData library and its bindings are now distributed under the GNU Lesser Public License, version 2.1 or later. Dirfile Changes * GetData now support Dirfile Standards Version 7. Standards Version 7 adds complex data types, a signed bitfield (SBIT), a polynomial derived field (POLYNOM), and other features detailed below. * Metafields can now be specified using a normal format specification line and their full (slashed) field code. * Field codes can now contain a "representation suffix", which allows specifying a complex norm used to convert complex valued data into purely real data. * The n_fields parameter in a LINCOM specification is now optional. * A new encoding scheme which supports LZMA compressed data using the .xz container format has been added. * Look-up tables may contain a complex y-value. This prohibits writing to the associated LINTERP field. Library Changes * The field parameters spf (RAW), bitnum and numbits (BIT and SBIT) are now restricted to 16-bit numbers, which should be more than enough. The public API uses gd_spf_t and gd_bit_t for these quantities. * The shift field parameter is now a 64-bit integer type, called gd_shift_t. * The parser callback has received a new, API breaking signature, which should hopefully reduce the amount of API breakage in the future. It is also now possible for the caller to pass data through GetData to the callback, if desired. * Computation of derived fields can now handle complex valued inputs, and computation will occur in the complex plane when required. * The computation of LINCOM fields has been re-written and is now ~20% faster for LINCOMs with 2 or 3 input fields. (No change for LINCOMs of 1 field.) * A new open flag, GD_PRETTY_PRINT, tells the library to attempt to write a nicer looking format file. What this specifically means, isn't part of the public API and shouldn't be relied on in portable code. * BUG FIX: dirfile_madd_spec() and dirfile_add_spec() didn't properly flag the fragments they modified as dirty, leading to possible loss of metadata. * BUG FIX: calling get_nframes on a bzip2 encoded dirfile would report the wrong dirfile length. * BUG FIX: LINTERP tables are now closed after loading, resolving a file descriptor leak of great age (ie. pre-0.3.0). * BUG FIX: When re-writing a format file fragment containing fields with CONST scalar field parameters, GetData no longer replaces the CONST field codes in the field specification lines with the value of the CONST field. * BUG FIX: when moving a field with dirfile_move(), the fragment index in the entry objects of metafields defined for the field were not updated. (The metafields were placed in the correct fragment when the metadata was flushed to disk, despite this.) * BUG FIX: Calling getdata() or putdata() on 64-bit architectures in certain situations involving a non-zero frameoffset no longer results in bizarrity. * BUG FIX: A memory leak associated with RAW field metadata has been fixed. * BUG FIX: 64-bit integer literals in the format file are no longer truncated to double precision when read. API Changes * This release breaks ABI compatibility and, to a lesser extent, API compatibility. The SOVERSION of the library has been incremented accordingly. * The dirfile flags, including encoding and endianness symbols are now explicitly long ints. Previously these quantities mixed long int with int arbitrarily. This affects the public API. * A C-89 API will be declared by getdata.h if GETDATA_C89_API is defined before inclusion. If this symbol is not defined, the C API defined will include C-99 conforming complex types, and getdata.h will include complex.h if `complex' has not been defined before inclusion. * Due to the new, simple way of specifying META fields in Standards Version 7, dirfile_add_spec() and dirfile_alter_spec() can be used to modify metafields. dirfile_madd_spec() and dirfile_malter_spec() remain as well. * An array containing the CONST scalar field codes used when non-literal parameters are used in the specification of various different fields has been added to the gd_entry_t object. As a side-effect of this, dirfile_add(), dirfile_alter_entry(), and dirfile_madd() can be used to manipulate non-literal field parameters. * A function, get_native_type(), now exists which returns the underlying data type of a specified field code. * A function, get_framenum(), is now available to perform a "reverse look-up" on a field: given a data value, it reports the frame number where the field has that value. The field must be monotonic. * A function, dirfile_validate(), now exists which can be used to check the validity of a field code before use. * Two new error codes have been added: GD_E_DOMAIN, potentially returned by get_framenum(), and GD_E_BAD_REPR, indicating an unrecognised representation suffix, or an attempt to write to a representation. * A new syntax error suberror code has been added: GD_E_FORMAT_LITERAL, which indicates a syntax error in a complex literal number. * BUG FIX: On success, dirfilename(), dirfile_reference(), and get_reference() didn't change the error code to GD_E_OK, resulting in spurious error codes from these functions. * BUG FIX: dirfile_madd_string() and dirfile_madd_const() now don't fail erroneously by reporting GD_E_BAD_CODE. * BUG FIX: error strings relating to GD_E_DELETE error codes now report the proper error information. * BUG FIX: the list returned by get_mstrings() is now properly NULL terminated. * BUG FIX: the entry->table member now reports the proper path to the LUT. (That is, the path stored in the format file.) Previously it incorrectly reported a path relative to the base dirfile directory. Legacy API Changes * Because they would otherwise be unreported, in the structure returned by GetFormat(), POLYNOM entries are reported as LINCOM entries, by discarding higher order terms. Similarly, SBIT entries are reported as BIT entries. Bindings * Bindings for Python and the Interactive Data Language (IDL) have been added. * Bindings for the functionality introduced into 0.6.0 have been added. * C++: The missing bindings for dirfilename(), get_nvectors(), and get_fragment_index() have been added. * C++: Empty entry class constructors have been added which allow delayed specification of field metadata. * C++: The dirfile.h header now includes all other C++ headers, and is now the only header required to be included in applications. (The other headers can still be included, with no ill effects.) * All bindings have a much expanded test suite, which has resulted in the discovery of many bugs (subsequently fixed). * F77: The callback function is now dirfile-specific, as it is in the C API. * C++ BUG FIX: Entry class constructors now properly set the field name for metafields. * C++ BUG FIX: Fragment::SetEncoding() now actually does what it adver- tises, instead of doing nothing. * C++ BUG FIX: The recode argument to various functions is now optional and defaults to zero, as the documentation already indicated. * C++ BUG FIX: dirfile_close() was not being called by the Dirifle destructor. * F77 BUG FIX: a NULL C string returned from the C API now returns an empty string, rather than causing a segfault. * F77 BUG FIX: attempting to call another F77 function before calling GDCOPN or GDOPEN no longer results in a segfault. * F77 BUG FIX: passing an out-of-range dirfile unit no longer results in undefined behaviour. * F77 BUG FIX: the last character of a string returned by the bindings is no longer deleted. * F77 BUG FIX: string lengths now report the correct length. * F77 BUG FIX: passing dirfile unit 0 to GDCLOS or GDDSCD no longer corrupts the bindings. * F77 BUG FIX: GDFLDN no longer accepts indices less than one. * F77 BUG FIX: GDESTR now space pads its output string, rather than filling the remainder of the string with garbage. * F77 BUG FIX: The get_entry() bindings, GDGExx no longer segfault on dirfile error. * F77 BUG FIX: GDGTST and GDPTST now report the FORTRAN string length, which is one less than the C API would report, due to the lack of a terminating null. * F95 BUG FIX: Attempting to retrieve a PHASE entry with fget_entry() now works. * F95 BUG FIX: Several routines which were named fdrifile_ are now properly spelled. Miscellaneous * All bindings can now be disabled by passing --disable-bindings to ./configure. |=========================================================================| New in version 0.5.0: Dirfile Changes * Support for two new encoding schemes has been added which handle bzip2 and gzip compression. Like the slim encoding, the bzip and gzip encod- ing schemes currently do not support writing, but do allow reading dirfile data compressed with the standard gzip and bzip2 utilities. Reading compressed data is unsurprisingly slow. Library Changes * Encoding schemes relying on external libraries (slim, gzip, bzip2) may now be built into stand-alone library modules which will be loaded, as needed, at runtime. GetData will fail gracefully if modules are missing. See the README for full details. * BUG FIX: Writing metadata to disc now preserves the permissions of the format file fragments which are changed. * BUG FIX: Format files without a line feed character at the end of the file no longer cause the parser to segfault. * BUG FIX: putdata() now reports the correct number of samples written for ASCII encoded files. * BUG FIX: GetData no longer inserts unnecessary "/./" elements into the paths it reports. API Changes * The caller can now register a callback function with a DIRFILE by opening the dirfile with dirfile_cbopen instead of dirfile_open. This callback function will be called by GetData whenever the format file parser encounters a syntax error. The callback function can be used by the caller to either correct the syntax error, or else tell GetData whether to ignore the line or not. The callback function may be later modified by calling dirfile_parser_callback. * A fragment may be removed from a dirfile by calling dirfile_uninclude. * The pathname of the dirfile may be retrieved by calling dirfilename. * The reference field for the dirfile may be set or retrieved by calling dirfile_reference. * Fragment metadata may be queried or modified by calling: - get_encoding, dirfile_alter_encoding - get_endianness, dirfile_alter_endianness - get_frameoffset, dirfile_alter_frameoffset - get_protection, dirfile_protect - get_parent_fragment * Various functions now exist to modify field metadata: - dirfile_alter_entry, dirfile_alter_, dirfile_alter_spec, dirfile_malter_spec * A field may be moved to a different format file fragment using dirfile_move. * A field may be renamed by using dirfile_rename. * A field may be removed from the dirfile by calling dirfile_delete. * The type or fragment index of a field may be retrieved by calling get_entry_type or get_fragment_index, respectively. * The pathname of a raw field may be obtained from a call to get_raw_filename. * A DIRFILE may be deallocated without saving modified metadata by calling dirfile_discard. * A new flag, GD_IGNORE_DUPS, may be passed to dirfile_open to tell the parser to ignore duplicate field specifications in the format files. This is really only useful identically duplicate specifications, since there is no indication of which of the duplicates is honoured. Explicit control can be obtained by handling this inside a caller supplied callback function. Bindings: * All functions in the Fortran 77 bindings have been completely renamed from GDFxxx to GDxxxx to provide a larger namespace for our use. * Fortran 77, Fortran 95, and C++ bindings have been updated for the latest API changes. * BUG FIX: Fortran 77 functions which return character strings no longer corrupt memory as a side-effect of operation. This also affected the Fortran 95 bindings. Miscellaneous: * BUG FIX: The dirfile_madd_bit(3) manual page has been corrected to show the correct order or parameters for all the dirfile_madd_ functions. |=========================================================================| New in version 0.4.2: Library Changes * BUG FIX: The default protection level of format file fragments is now properly set to "none". Reported by Brendan Crill. * BUG FIX: getdata() now reports the correct number of samples read for slim and ASCII encoded files. Reported by Adam Hincks. * BUG FIX: A segmentation fault only encountered when reading slim encoded data has been fixed in dirfile_flush(). Legacy API Changes * Several fixes have been made to better re-create the legacy API: - the return value of GetFormat is no longer marked const (although it should be considered so). - a symbolic link getdata_struct.h -> getdata.h is created when the headers are installed, but only if the legacy API is present in the library. - when included with a C++ compiler, the legacy function prototypes are wrapped in an extern "C" block. (The new API functions are not.) |=========================================================================| New in version 0.4.1: Library Changes * BUG FIX: Adding an ASCII encoded RAW field no longer creates an empty, unencoded binary file. * BUG FIX: A few minor memory leaks have been patched. Most of these were the result of insufficient clean up in dirfile_close(). * As an exception to the Standards, the full stop character '.' is once again permitted in field names. This is required by BLAST. Instead, field names are prohibited from ending with the known encoding exten- sions. The Standards still prohibit the full stop character in field names, and the GD_PEDANTIC flag will disable this exception. Further- more, fields may not be added via GetData which contain a full stop character. * Some minor speed improvements have been made to the format file parser resulting in a ~30% decrease in parsing time. |=========================================================================| New in version 0.4.0: Dirfile Changes * GetData now supports Dirfile Standards Version 6. Standards Version 6 adds numerical (CONST) and string (STRING) scalar fields to the dirfile, as well as other features outlined below. * CONST names can now appear as parameters in field specifications in certain places in lieu of a literal number. * The tokeniser has been re-written. Any sequence of non-NUL bytes can now form a valid token. Field codes may not have ASCII control charac- ters or the "reserved characters" &, /, ;, <, >, |, . but may contain any other characters (including whitespace). * Subordinate fields may now be attached to fields. These subfields are defined with the META directive, and referenced in most cases as "/". * Binary files in dirfiles may now be "encoded". In addition to raw (unencoded) files, two encodings are currently supported: - ASCII encoding (a simple proof-of-functionality encoding), and - Slimlib encoding (a compression library used by ACT). Encoding schemes are fragment-local, although they are inherited from parent fragments. * Endianness is now also fragment-local, meaning different endiannesses can co-exist in the same dirfile (so long as the fields are defined in different format file fragments), and GetData will do the Right Thing. * "FILEFRAM" is no-longer supported as an alias for "INDEX". This affects GetData's implementation of all Standards Versions. Code that made use of FILEFRAM will have to be updated to use INDEX instead. * The reference field, previously the first RAW field in the format file, may now be specified using the REFERENCE directive. The reference field is the field that is looked at when get_nframes() is called. This is important to programs like kst which support streaming dirfiles. * Portions (or all) of a dirfile's data and metadata can be protected from change by specifying the PROTECT directive. This protection is "advisory", ie. while GetData will respect it in all cases, don't count on the PROTECT directive to ensure data integrity. Library Changes * INDEX is now a normal field. It appears in the count/list functions. Attempts to write to INDEX will fail. * The GD_E_EMPTY error code has been removed: it is no longer an error to query a dirfile containing no RAW fields. * The presence of two fields in the dirfile with the same name is now detected, and will cause a syntax error when parsing the format file. Previously, such things were allowed, but only one of the synonym fields could ever be queried. (Which field was returned was arbitrary.) * Field code look-ups for input fields are now cached, which should result in slightly better performance from the library. API Changes * Fields may now be added to dirfiles in a number of different ways: - by passing an gd_entry_t to dirfile_add() - by passing a field specification line to dirfile_add_spec() - by passing field parameters to one of the dirfile_add_() functions. * Whole other format file fragments may also be added to the dirfile by calling dirfile_include(). * A function is now present (dirfile_metaflush()) to flush metadata changes to disk (by re-writing format file fragments). dirfile_flush() and dirfile_close() will also flush metadata, if needed. * STRING and CONST values can be retrieved/set by calling (get,put)_string() and (get,put)_constant(). (get,put)data() are only for vector type fields. * META fields can be queried/set like normal fields using the get/put functions and their full (slashed) field code. * In addition to get_nfields() and get_field_list(), there are now corresponding functions that provide lists/counts of vectors (get_vector_list()/get_nvectors()), particular field types (get_field_list_by_type()/get_nfields_by_type()), as well as functions that provide lists of string values (get_string_values()) and constant values (get_const_values()). * Analogous functions for the add, list, and counting functions exist for META fields, as well. * Extra flags have been added to dirfile_open() to permit indicating the encoding type, if not specified in the dirfile itself. * DIRFILE struct members and gd_entry_t private members are now com- pletely hidden from the public interface. Where previously callers could query dirfile->error to check for an error, they must now call get_error(dirfile). This change was made to reduce unintentional ABI breakage when modifying internal library properties. * GD_VERBOSE has been added to the list of available dirfile_open() flags. If this flag is specified, GetData will write errors to stderr as encountered. Legacy API Changes * Error codes which cannot be returned by the legacy API might not have a corresponding string in GD_ERROR_CODES[]. Instead, these entries will simply be the NULL pointer. * No facilities exist in the legacy API to set or query CONST or STRING fields. However, META fields of vector type can be queried/set using GetData() and PutData(), as in the new API. |=========================================================================| New in version 0.3.1: Legacy API Changes * BUG FIX: Dirfiles are now opened in read-only mode, unless instantiated via PutData(), allowing GetData() calls on read-only dirfiles. If PutData() is called on a dirfile previously opened read-only, it will be re-opened in read-write mode. Bindings * The C++ bindings, formerly called libdirfile, are now called libgetdata++ to be more explicit about what this library is. Miscellaneous * The package now includes pkg-config support for libgetdata. |=========================================================================| New in version 0.3.0: Dirfile Changes * GetData now supports Dirfile Standards Version 5 which includes support for signed 8-bit, and signed and unsigned 64-bit integer types. * As part of Standards Version 5, the restriction on field name length has been removed. (The filesystem will impose an effective limit on RAW fields of a few hundred characters, and format file lines are limited to several thousand, limiting derived field names.) * The library can now convert between big and little endiannesses. An optional directive, ENDIAN, added in Standards Version 5, is available to specify the byte-sex of a dirfile. API Changes * There is a new interface which fixes issues with thread safety and largefile support in the old interface. The old interface (referred to as the "legacy API") is still supported, but doesn't fully implement Dirfile Standards Version 5. See the README for full details. * putdata now respects FRAMEOFFSET. * putdata can now write to PHASE fields, and multi-bit BIT fields. * Some error codes have been renamed, and others removed or added, in order to regularise error codes between getdata and putdata in the sundry versions. Furthermore, the underlying values of some of these codes have changed (notably excluding GD_E_OK, which is guaranteed to be zero). Changes include: - GD_E_OPEN_FORMAT is now called GD_E_OPEN - GD_E_BAD_RETURN_TYPE is now called GD_E_BAD_TYPE - GD_E_NO_RAW_FIELDS is now called GD_E_EMPTY - PD_E_MULT_LINCOM is now called GD_E_BAD_PUT_FIELD - GD_E_OPEN_RAWFIELD and PD_E_OPEN RAWFIELD are now both represented by GD_E_RAW_IO - PD_E_BAD_CODE is now handled by GD_E_BAD_CODE - GD_E_FIELD, GD_E_SIZE_MISMATCH, ENDIAN_ERROR, CLOSE_ERROR are no longer applicable and have been removed - PD_E_CLOSE_RDONLY, PD_E_WRITE_LOCK, PD_E_FLOCK_ALLOC, which were were defined in the header but never used, have been removed - GD_E_TRUNC, GD_E_CREAT, GD_E_BAD_DIRFILE, GD_E_RANGE and GD_E_ACCMODE are new This affects the legacy API. Legacy API Changes * The legacy API has been marked deprecated. * Error codes have changed per the description above. * Only the public members of FormatType are now initialised by GetFormat. Bindings * Added bindings for C++ (libdirfile), Fortran 77 (libfgetdata), and Fortran 95 (libf95getdata) Miscellaneous * A rudimentary, but thorough, test-suite has been made.