v3.4.11 (2020-09-08) -------------------- * Updated arangosync to 0.7.11. * Added missing state rollback for failed attempt-based write locking of spin locks. * Correct some log entries. * Allows restoring collections from v3.3.0 with their all-numeric collection GUID values, by creating a new, unambiguous collection GUID for them. v3.3.0 had a bug because it created all-numeric GUID values, which can be confused with numeric collection ids in lookups. v3.3.1 already changed the GUID routine to produce something non-numeric already, but collections created with v3.3.0 can still have an ambiguous GUID. This fix adjusts the restore routine to drop such GUID values, so it only changes something if v3.3.0 collections are dumped, dropped on the server and then restored with the flawed GUIDs. * Fixed velocypack validator for proper check of keys. * Fixed issue ES-598. Web UI now shows correct permissions in case wildcard database level and wildcard collection level permissions are both set. * Pass through AQL query errors to UI graph viewer without turning them into "Internal Server Error"s. * Fixed the creation of edges in the web UI, which did not work due to a JavaScript error. * Updated ArangoDB Starter to 0.14.15. * Optimized Web UI styling for ArangoSearch Views when in read-only mode. * Fixed issue #11104 case with complex condition on second branch of OR operator. * Update OpenLDAP to 2.4.50. * Fixed issue #11525: Address security vulnerability by updating Swagger-UI dependency (upgraded Swagger UI to 3.25.1). * Fixed issue ES-605: arangodump broken when using encryption. * Do not always loadPlan from agency when collections of a database are listed. This is particularly important for arangosync. v3.4.10 (2020-04-24) -------------------- * Don't invoke collection key ranges compaction after performing a truncate operation on a RocksDB storage engine collection via the REST API's truncate endpoint. The compaction was previously done to make RocksDB physically free data of key ranges that were supposed to be empty after the truncate, but it seems that the used version of RocksDB is struggling to achieve that goal. Although it does compact the ranges, it seems to read a lot of extra data from unrelated ranges as well, which blows up the overall time required to do the compactions. Most recent versions of RocksDB do not seem to have this problem anymore. But without upgrading RocksDB to the latest version the most sensible fix is to turn off the manual compaction and rely on RocksDB eventually freeing the key ranges that are empty after a truncate. * Fixed issue #11104: `ANALYZER` function doesn't correctly process `!= OR` queries. * Honor `cacheEnabled` attribute when updating the properties of a collection in case of the cluster. Previously, this attribute was ignored in the cluster, but it shouldn't have been. * Updated arangosync to 0.7.3. * Updated ArangoDB Starter to 0.14.14. * Fix issue #10897, using COLLECT in a subquery could lead to unexpected and confusing error messages. * Internal statistics API now uses integer timestamps instead of doubles. The old behaviour sometimes leads to failed requests because of parse errors which occurred in the internally used JavaScript joi library. * Make sure heartbeats are actually sent out and received once per second. * Made active failover integration tests more stable. * Fix spurious "substr index out of bounds" VelocyPack errors when accessing a full substring. v3.4.9 (2020-01-23) ------------------- * Fix Foxxmaster failover retry loop to not spin forever and give up after a while. * Fix string comparison bug that lead to traversal queries accepting prefixes of "edges" and "vertices" to be used as object accessors for the path object. * MoveShard to check, if target is in sync follower before promotion to leader. * Fix installation of arangoinspect and libraries in Windows client installer. * Fixed issue #10867: arangod based binaries lack product version info and icon on Windows. * Now clearing an internal map inside the traverser engine correctly. * Fix Windows client package JS installation paths. * Updated arangosync to 0.7.2. * Fixed internal issue #656: While executing large amount of concurrent insert/removes iresearch seems to leak open file handles. This results in error 32 in cleanup (observed only on Windows as Linux doesn`t have file sharing locks). * Fixed issue #10725: Wrong document count shown inside an empty collection in the web UI. * Fixed internal issue #4748: Editing a single edgeDefinition using the graph API failed if it was not shared between all available graphs. * Fixed a bug in smart graph bfs traversals that might violate path uniqueness requirements in rare cases. * Bugfix: An AQL ternary expression with the condition being true at query compile time would not execute its false branch. * Fixed GET _api/gharial to also include the name property in every returned graph. This is a consistency fix within the API as all other APIs include the name. As a work around the returned _key can be used, which is identical to the name. * Fixed permissions for dump/restore. * Allow the optimizer to use indexes when a collection attribute is compared to an expansion followed by an attribute name, e.g. `doc.value IN something[*].name`. * Fixed issue #10440: Incorrect sorting with sort criteria partially covered by index. * Fixed UPSERT matching. Empty objects in the `UPSERT { ... }` expression will not be omitted anymore: db._collection("col").insert({ "find" : "me" }); db._query(` UPSERT { "find" : "me", "foo" : {} } UPDATE { "foo" : "not gonna happen" } INSERT { "find" : "me", "foo" : {} } INTO col `) This will now correctly insert a document instead of updating the existing, that only partially machtes the upsert-expression. * In the RocksDB engine, automatically flush more exotic column families 10 minutes after server startup, plus every 12 hours while the server is running. This should prevent data from more exotic column families to stay only in WAL and memtables, which may block WAL journals collection. * The _users collection is now properly restored when using arangorestore. * Fixed issue #10470: The WebUI now shows potential errors and details which occured using _api/import (e.g. unique constraint violated). * Make the timeouts for replication requests (for active failover and master-slave replication configurable via startup options: --replication.connect-timeout --replication.request-timeout * Fixed ArangoSearch index removes being discarded on commiting consolidation results with pending removes after some segments under consolidation were already committed. * Disallow the usage of subqueries inside AQL traversal PRUNE conditions. Using subqueries inside PRUNE conditions causes undefined behavior, so such queries will now be aborted early on with a parse error instead of running into undefined behavior. * Fixed search not working in document view while in code mode. * Fixed issue #10090: fix repeatable seek to the same document in SEARCH operations for ArangoSearch views. * Fix config directory handling, so we don't trap into UNC path lookups on Windows. * Fixed issue #10193: Arangoexport does not handle line feeds when exporting as csv. * Properly report parse errors for extraneous unterminated string literals at the end of AQL query strings. For example, in the query `RETURN 1 "abc`, the `RETURN 1` part was parsed fully, and the `"abc` part at the end was parsed until the EOF and then forgotten. But as the fully parsed tokens `RETURN 1` already form a proper query, the unterminated string literal at the end was not reported as a parse error. This is now fixed for unterminated string literals in double and single quotes as well as unterminated multi-line comments at the end of the query string. * Fixed issue #10078: FULLTEXT with sort on same field not working. * Fixed issue #10062: AQL: Could not extract custom attribute. * Made the mechanism in the Web UI of replacing and upgrading a foxx app more clear. * Fixed a bug in the edge cache's internal memory accounting, which led to the edge cache underreporting its current memory usage. * Fix strange shutdown hanger which came from the fact that currently libgcc/libmusl wrongly detect multi-threadedness in statically linked executables. * Obtain new unique IDs via a background thread. * Fixed a shutdown hanger because of a read/write lock race. * Fixed "ArangoDB is not running in cluster mode" errors in active failover setups. This affected at least /_admin/cluster/health. * Harden database creation against spurious "duplicate name" errors that were caused by other parallel operations lazily creating required system collections in the same database. * Fixed the removal (including a collection drop) of an orphanCollection from a graph definition when using the arango shell. The boolean flag whether to drop the collection or not was not transferred properly. * Fixed internal issue #633: made ArangoSearch functions BOOST, ANALYZER, MIN_MATCH callable with constant arguments. This will allow running queries where all arguments for these functions are deterministic and do not depend on loop variables. * Added UI support to create documents in a collection using smartGraphAttribute and/or smartJoinAttribute. * Add --server.statistics-history flag to allow disabling of only the historical statistics. Also added rocksdbengine.write.amplification.x100 statistics for measurement of compaction option impact. Enabled non-historical statistics for agents. * Fixed output of `_listObjects()` function of "general-graph" and "smart-graph" modules, which did not print the `_id` values of the return graphs correctly. * Fixed undefined behavior in single server mode when trying to create a smart graph. * Add count of objects to latency reporting in arangoimport. * Fixed adding an orphan collections as the first collection in a SmartGraph. * The graph viewer of the web interface now tries to find a vertex document of all available vertex collections before it aborts. * Bugfix: Save distinct WAL ticks for multiple replication clients from the same server. Also, when a follower is added for synchronous replication, the WAL tick held by the client is freed immediately, rather than waiting for a timeout. The corresponding APIs get a new parameter `syncerId`, which, if given, supersedes `serverId`. This affects the routes /_api/wal/tail, /_api/replication/batch, /_api/replication/logger-follow and the internal route /_api/replication/addFollower. The new field `syncerId` is also added to the response of /_api/replication/logger-state. v3.4.8 (2019-09-09) ------------------- * Drop collection action to timeout more quickly to stay on fast lane. * Fixed non-deterministic occurrences of "document not found" errors in sharded collections with custom shard keys (i.e. non-`_key`) and multi-document lookups. * Fixed issue #9798: limit with offset results in an infinite query that can't be cancelled. * Agents to remove callback entries when responded to with code 404. * Make minimum timeout for synchronous replication configurable via parameter (--cluster.synchronous-replication-timeout-minimum) and increase default value to 30.0 seconds to prevent dropping followers unnecessarily. Should be changed only with intent and great care. * Fixed issue #9795: fix AQL `NOT IN` clause in SEARCH operations for ArangoSearch views. * Don't retry persisting follower information for collections/shards already dropped. The previous implementation retried (unsuccessfully in this case) for up to 2 hours, occupying one scheduler thread. * Show query string length and cacheability information in query explain output. * Fixed internal issue #4378: fix bug in MoveShard::abort which causes a duplicate entry in the follower list. * Fixed issue #9654: honor value of `--rocksdb.max-write-buffer-number` if it is set to at least 9 (which is the recommended value). Ignore it if it is set to a lower value than 9, and warn the end user about it. Previous versions of ArangoDB always silently ignored the value of this setting and effectively hard-coded the value to 9. * Fixed cut'n'pasting code from the documentation into arangosh. * Refactor maintenance to use a TakeoverShardLeadership job. This fixes a bug that a shard follower could have set the wrong leader for the shard locally. * Fixed issue #9679: improved error message for FULLTEXT function invocation failures. * Added resign leadership job to supervision. * Fixed issue #9612: Concurrent Writes leads to data loss in ArangoSearch/Views. * Keep followers in sync if the old leader resigned and stopped writes. * Abort a FailedLeader job when its _to_ server fails. * Re-added warning for kernel setting "overcommit_memory=2" if jemalloc is also enabled. * Significantly reduced "random" 1 second delays in some arangosearch DDL operations. * Decreased unnecessary wait times for agency callbacks in case they were called earlier than expected by main thread. * Upgraded arangodb starter version to 0.14.12 * Fixed internal issue #3680 by adding two new options to make the server more secure: - `--foxx.api`: set to false disables Foxx management API - `--foxx.store`: set to false disables Foxx UI * updated arangosync to 0.6.5 * Fixed issue #9459: Optimization rule remove-collect-variables does not KEEP all necessary data. * Fixed issue #9558: RTRIM not working as expected. * Added gzip and encryption options to arangoimport and arangoexport. * Added startup error for bad temporary directory setting. If the temporary directory (--temp.path) setting is identical to the database directory (`--database.directory`) this can eventually lead to data loss, as temporary files may be created inside the temporary directory, causing overwrites of existing database files/directories with the same names. Additionally the temporary directory may be cleaned at some point, and this would lead to an unintended cleanup of the database files/directories as well. Now, if the database directory and temporary directory are set to the same path, there will be a startup warning about potential data loss (though in ArangoDB 3.4 allowing to continue the startup - in 3.5 and higher we will abort the startup). * Fixed a query abort error in smart joins when both collections were restricted to a single shard. * Fixed potential thread handle leak on thread shutdown on Windows. * Fixed recovery of arangosearch remove operations with the RocksDB storage engine. * Fixed some races in cluster collection creation, which allowed collections with the same name to be created in parallel under some rare conditions (Internal Issue #4296). * arangoimport would not stop, much less report, communications errors. Add CSV reporting of line numbers that are impacted during such errors. * Fixed a bug which could lead to some unnecessary HTTP requests during an AQL query in a cluster. Only occurs with views in the query. * Prevent rare cases of duplicate DDL actions being executed by Maintenance. * Coordinator code was reporting rocksdb error codes, but not the associated detail message. Corrected. * Fixed some error reporting and logging in Maintenance. * Fixed an error condition in which an ex-leader for a short still believed to be the leader and wrongly reported to Current. * Fixed an agency bug found in Windows tests. v3.4.7 (2019-07-02) ------------------- * updated arangosync to 0.6.4 * bug-fix for a very rare race condition on cluster collection creation process. It can only occur in the following situation: 1) DBServer sucessfully creates all shards for collections, and reports it back 2) DBServer dies 3) Agency recognized death and reorganizes shards 4) Coordinator receives (1) but not (3) and decided everything was good 5) => Coordinator reverts (3) only on this collection 6) Coordinator receives (3) now The bugfix disallows (5) if (3) is run in Agency. * fixed internal issue #4040: gharial api is now checking existence of `_from` and `_to` vertices during edge replacements and edge updates * fix timeout-response in case of simultaneous index create/drop in cluster * Fixed editing a user within the web UI if the user added a gravatar profile picture * fixed internal issue #4172: on agent servers the RocksDB WAL files in the archive directory were retained due to an error. Now they are removed, eventually * fix error reporting in arangoexport. previously, when the server returned an HTTP 404 error, arangoexport unconditionally translated this into a "collection not found" error, which may have masked an actually different server-side error. Instead, make arangoexport return the server's actual error message as is. * allow pregel to select the shard key via `shardKeyAttribute` in pregel start parameters * Speed up collection creation process in cluster, if not all agency callbacks are delivered successfully. * Fixed parsing of ArangoDB config files with inlined comments. Previous versions didn't handle line comments properly if they were appended to an otherwise valid option value. For example, the comment in the line max-total-wal-size = 1024000 # 1M was not ignored and made part of the value. In the end, the value was interpreted as if max-total-wal-size = 10240001000000 was specified. This version fixes the handling of comments in the config files so that they behave as intended. * Fixed AQL query tracing for traversals, this lead to wrong "calls" count in query profiles * Pregel algorithms can be run with the option "useMemoryMaps: true" to be able to run algorithms on data that is bigger than the available RAM. * Bugfix for smart graph traversals with uniqueVertices: path, which could sometimes lead to erroneous traversal results * The system-collection '_jobs' will from now on use non-unique, non-sparse indexes. * fixed internal issue #3919: The web UI is now using precompiled ejs templates. * properly create VERSION files for any newly created databases again, as it has been the case in 3.3 and before. If no VERSION files are created, this may cause problems on upgrade to 3.5. The 3.5 branch was also modified to handle this problem gracefully. * fixed "collection not found" exception during setup of 3-way smart join queries in the cluster * fixed an edge case of handling `null` values in the AQL function `MIN` for input sequences that started with a `null` value. In this case, `null` was always returned as the minimum value even though other non-null values may have followed, and `MIN` was supposed to return `null` only if there are no input values or all input values are `null`. * fix client id lookup table in agency * Fix agency's handling of object assignments with TTL * reduce resident memory usage when creating non-unique indexes with the RocksDB storage engine * Fix agency election bug. v3.4.6.1 (2019-06-05) --------------------- * upgraded ArangoDB Starter version to 0.14.5 * Speed up collection creation process in cluster, if not all agency callbacks are delivered successfully. * Fix agency's handling of object assignments with TTL v3.4.6 (2019-05-21) ------------------- * updated ArangoDB Starter to 0.14.4 * fixed internal issue #3912: improved the performance of graph creation with multiple relations. They do now create multiple collections within a single round-trip in the agency. * fixed a crash when posting an async request to the server using the "x-arango-async" request header and the server's request queue was full * added error code 1240 "incomplete read" for RocksDB-based reads which cannot retrieve documents due to the RocksDB block cache being size-restricted (with size limit enforced) and uncompressed data blocks not fitting into the block cache The error can only occur for collection or index scans with the RocksDB storage engine when the RocksDB block cache is used and set to a very small size, plus its maximum size is enforced by setting the `--rocksdb.enforce-block-cache-size-limit` option to `true`. Previously these incomplete reads could have been ignored silently, making collection or index scans return less documents than there were actually present. * fixed internal issue #3918: added optional second parameter "withId" to AQL function PREGEL_RESULT this parameter defaults to `false`. When set to `true` the results of the Pregel computation run will also contain the `_id` attribute for each vertex and not just `_key`. This allows distinguishing vertices from different vertex collections. * fixed issue #7011: description when replacing a foxx application was misleading * fixed issue #8841: Graph Viewer dropped ability to edit an edge after rerendering. * fix removal of sort order for sorted results that were passed into a RETURN DISTINCT in an AQL query running on a collection with more than a single shard. In this case, the results from the individual shards were sorted, but the overall result was not sorted according to the sort criterion. * removed bug during start up with a single agent, that leads to dbserver crash. * fix the creation of debug packages (in the web interface) for queries that involve smart graphs and/or multiple edge collections from a traversal * add --compress-output flag to arangodump. Activates gzip compression for collection data. Metadata files, such as .structure.json and .view.json, do not get compressed. No option is needed for arangorestore to restore .data.json.gz files. * fixed internal issue #3893: correct wrong option name `--database.ignore-logfile-errors` in error message and correct it to `--database.ignore-datafile-errors` * fixed internal issue #2946: create graph autocomplete was not working under certain circumstances. * fixed a memory leak in PRUNE operation for traversals. The leak occurred only when PRUNE was actively used in queries. * fixed a crash (SIGSEGV) when opening a RocksDB database directory that contained an empty (0 bytes filesize) journal file and encryption was in use. * improve test performance by doing the 0.1 delay between phase 1 and 2 only if phase 1 is slow * port back timestamp replication in agency from devel * better logging when scheduler queue is half-full and full * fixed internal issue #536: ArangoSearch query may crash during internal lookup in some cases due to invalid index structure for exact input data v3.4.5 (2019-03-27) ------------------- * fixed a shutdown issue when the server was shut down while there were active Pregel jobs executing * fixed internal issue #3815: fixed the removal of connected edges when removing a vertex graph node in a smart graph environment. * added AQL functions CRC32 and FNV64 for hashing data * internal issue #2276: fixed the sorting of the databases in the database selection dropdown in the web ui. The sort order differed based on whether authentication was enabled or disabled. * fixed internal issue #3789: restricted the allowed query names for user- defined custom queries within the web ui. * fixed internal issue #3546: improved the shards view in the web ui if there is only one shard to display. * fixed a display issues when editing a graph within the web ui * fixed internal-issue #3787: Let user know about conflicting attribute in AQL queries if information is available. * fixed issue #8294: wrong equals behavior on arrays with ArangoSearch * fixed internal issue #528: ArangoSearch range query sometimes doesn't work correctly with numeric values * fixed internal issue #3757: when restarting a follower in active failover mode, try an incremental sync instead of a full resync. Fixed also the case where a double resync was made * don't check for the presence of ArangoDB upgrades available when firing up an arangosh enterprise edition build * added startup option `--rocksdb.allow-fallocate` When set to true, allows RocksDB to use the fallocate call. If false, fallocate calls are bypassed and no preallocation is done. Preallocation is turned on by default, but can be turned off for operating system versions that are known to have issues with it. This option only has an effect on operating systems that support fallocate. * added startup option `--rocksdb.limit-open-files-at-startup` If set to true, this will limit the amount of .sst files RocksDB will inspect at startup, which can reduce the number of IO operations performed at start. * don't run compact() on a collection after a truncate() was done in the same transaction running compact() in the same transaction will only increase the data size on disk due to RocksDB not being able to remove any documents physically due to the snapshot that is taken at transaction start. This change also exposes db..compact() in the arangosh, in order to manually run a compaction on the data range of a collection should it be needed for maintenance. * don't attempt to remove non-existing WAL files, because such attempt will trigger unnecessary error log messages in the RocksDB library * updated arangosync to 0.6.3 * added --log.file-mode to specify a file mode of newly created log files * added --log.file-group to specify the group of newly created log files * fixed some escaping issues within the web ui. * fixed issue #8359: How to update a document with unique index (constraints)? * when restarting a follower in active failover mode, try an incremental sync instead of a full resync * add "PRUNE " to AQL Traversals (internal issue #3068). This allows to early abort searching of unnecessary branches within a traversal. PRUNE is only allowed in the Traversal statement and only between the graphdefinition and the options of the traversal. e.g.: FOR v, e, p IN 1..3 OUTBOUND @source GRAPH "myGraph" PRUNE v.value == "bar" OPTIONS {} /* These options remain optional */ RETURN v for more details refer to the documentation chapter. * added option `--console.history` to arangosh for controlling whether the command-line history should be loaded from and persisted in a file. The default value for this option is `true`. Setting it to `false` will make arangosh not load any command-line history from the history file, and not store the current session's history when the shell is exited. The command-line history will then only be available in the current shell session. * display the server role when connecting arangosh against a server (e.g. SINGLE, COORDINATOR) * added replication applier state figures `totalDocuments` and `totalRemovals` to access the number of document insert/replace operations and the number of document removals operations separately. Also added figures `totalApplyTime` and `totalFetchTime` for determining the total time the replication spent for applying changes or fetching new data from the master. Also added are the figures `averageApplyTime` and `averageFetchTime`, which show the average time spent for applying a batch or for fetching data from the master, resp. * fixed race condition in which the value of the informational replication applier figure `ticksBehind` could underflow and thus show a very huge number of ticks. * always clear all ongoing replication transactions on the slave if the slave discovers the data it has asked for is not present anymore on the master and the `requireFromPresent` value for the applier is set to `false`. In this case aborting the ongoing transactions on the slave is necessary because they may have held exclusive locks on collections, which may otherwise not be released. * added option `--rocksdb.wal-archive-size-limit` for controlling the maximum total size (in bytes) of archived WAL files. The default is 0 (meaning: unlimited). When setting the value to a size bigger than 0, the RocksDB storage engine will force a removal of archived WAL files if the total size of the archive exceeds the configured size. The option can be used to get rid of archived WAL files in a disk size-constrained environment. Note that archived WAL files are normally deleted automatically after a short while when there is no follower attached that may read from the archive. However, in case when there are followers attached that may read from the archive, WAL files normally remain in the archive until their contents have been streamed to the followers. In case there are slow followers that cannot catch up this will cause a growth of the WAL files archive over time. The option `--rocksdb.wal-archive-size-limit` can now be used to force a deletion of WAL files from the archive even if there are followers attached that may want to read the archive. In case the option is set and a leader deletes files from the archive that followers want to read, this will abort the replication on the followers. Followers can however restart the replication doing a resync. * agents need to be able to overwrite a compacted state with same _key * in case of resigned leader, set isReady=false in clusterInventory * abort RemoveFollower job if not enough in-sync followers or leader failure * fix shrinkCluster for satelliteCollections * fix crash in agency supervision when leadership is lost * speed up supervision in agency for large numbers of jobs * fix log spamming after leader resignation in agency * make AddFollower less aggressive * fix cases, where invalid json could be generated in agents' store dumps * coordinator route for full agency dumps contains compactions and time stamps * lots of agency performance improvements, mostly avoiding copying * priority queue for maintenance jobs * do not wait for replication after each job execution in Supervision * fix a blockage in MoveShard if a failover happens during the operation * check health of servers in Current before scheduling removeFollower jobs * wait for statistics collections to be created before running resilience tests * fix ttl values in agency when key overwritten with no ttl v3.4.4 (2019-03-12) ------------------- * added missing test for success in failed leader: this could lead to a crash * follow up to fix JWT authentication in arangosh (#7530): also fix reconnect * now also syncing _jobs and _queues collections in active failover mode * fixed overflow in Windows NowNanos in RocksDB * fixed issue #8165: AQL optimizer does not pick up multiple geo index * when creating a new database with an initial user, set the database permission for this user as specified in the documentation * Supervision fix: abort MoveShard job does not leave a lock behind, * Supervision fix: abort MoveShard (leader) job moves forwards when point of no return has been reached, * Supervision fix: abort CleanOutServer job does not leave server in ToBeCleanedServers, * Supervision fix: move shard with data stopped to early due to wrong usage of compare function * Supervision fix: AddFollower only counts good followers, fixing a situation after a FailedLeader job could not find a new working follower * Supervision fix: FailedLeader now also considers temporarily BAD servers as replacement followers and does not block servers which currently receive a new shard * Supervision fix: Servers in ToBeCleanedServers are no longer considered as replacement servers * Maintenance fix: added precondition of unchanged Plan in phase2 * Allow MoveShard from leader to a follower, thus swapping the two * Supervision fix: Satellite collections, various fixes * Add coordinator route for agency dump * speed up replication of transactions containing updates of existing documents. The replication protocol does not provide any information on whether a document was inserted on the master or updated/replaced. Therefore the slave will always try an insert first, and move to a replace if the insert fails with "unique constraint violation". This case is however very costly in a bigger transaction, as the rollback of the insert will force the underlying RocksDB write batch to be entirely rewritten. To circumvent rewriting entire write batches, we now do a quick check if the target document already exists, and then branch to either insert or replace internally. v3.4.3 (2019-02-19) ------------------- * fixed JS AQL query objects with empty query strings not being recognized as AQL queries * fixed issue #8137: NULL input field generates U_ILLEGAL_ARGUMENT_ERROR * fixed issue #8108: AQL variable - not working query since upgrade to 3.4 release * fixed possible segfault when using COLLECT with a LIMIT and an offset * fixed COLLECT forgetting top-level variables after 1000 rows * fix undefined behavior when calling user-defined AQL functions from an AQL query via a streaming cursor * fix broken validation of tick range in arangodump * updated bundled curl library to version 7.63.0 * added "peakMemoryUsage" in query results figures, showing the peak memory usage of the executed query. In a cluster, the value contains the peak memory usage across all shards, but it is not summed up across shards. * data masking: better documentation, fixed default phone number, changed default range to -100 and 100 for integer masking function * fix supervision's failed server handling to transactionally create all failed leader/followers along v3.4.2.1 (2019-02-01) --------------------- * upgrade to new velocypack version v3.4.2 (2019-01-24) ------------------- * added configurable masking of dumped data via `arangodump` tool to obfuscate exported sensible data * upgraded to OpenSSL 1.1.0j * fixed an issue with AQL query IN index lookup conditions being converted into empty arrays when they were shared between multiple nodes of a lookup condition that used an IN array lookup in an OR that was multiplied due to DNF transformations This issue affected queries such as the following FILTER (... && ...) || doc.indexAttribute IN non-empty-array * upgraded arangodb starter version to 0.14.0 * upgraded arangosync version to 0.6.2 * fixed an issue where a crashed coordinator can lead to some Foxx queue jobs erroneously either left hanging or being restarted * fix issue #7903: Regression on ISO8601 string compatibility in AQL millisecond parts of AQL date values were limited to up to 3 digits. Now the length of the millisecond part is unrestricted, but the millisecond precision is still limited to up to 3 digits. * fix issue #7900: Bind values of `null` are not replaced by empty string anymore, when toggling between json and table view in the web-ui. * Use base64url to encode and decode JWT parts. * added AQL function `CHECK_DOCUMENT` for document validity checks * when detecting parse errors in the JSON input sent to the restore API, now abort with a proper error containing the problem description instead of aborting but hiding there was a problem. * do not respond with an internal error in case of JSON parse errors detected in incoming HTTP requests * added arangorestore option `--cleanup-duplicate-attributes` to clean up input documents with redundant attribute names Importing such documents without the option set will make arangorestore fail with an error, and setting the option will make the restore process clean up the input by using just the first specified value for each redundant attribute. * the arangorestore options `--default-number-of-shards` and `--default-replication-factor` are now deprecated in favor of the much more powerful options `--number-of-shards` and `--replication-factor` The new options `--number-of-shards` and `--replication-factor` allow specifying default values for the number of shards and the replication factor, resp. for all restored collections. If specified, these default values will be used regardless of whether the number of shards or the replication factor values are already present in the metadata of the dumped collections. It is also possible to override the values on a per-collection level by specifying the options multiple times, e.g. --number-of-shards 2 --number-of-shards mycollection=3 --number-of-shards test=4 The above will create all collections with 2 shards, except the collection "mycollection" (3 shards) and "test" (4 shards). By omitting the default value, it is also possible to use the number of shards/replication factor values from the dump for all collections but the explicitly specified ones, e.g. --number-of-shards mycollection=3 --number-of-shards test=4 This will use the number of shards as specified in the dump, except for the collections "mycollection" and "test". The `--replication-factor` option works similarly. * validate uniqueness of attribute names in AQL in cases in which it was not done before. When constructing AQL objects via object literals, there was no validation about object attribute names being unique. For example, it was possible to create objects with duplicate attribute names as follows: INSERT { a: 1, a: 2 } INTO collection This resulted in a document having two "a" attributes, which is obviously undesired. Now, when an attribute value is used multiple times, only the first assigned value will be used for that attribute in AQL. It is not possible to specify the same attribute multiple times and overwrite the attribute's value with by that. That means in the above example, the value of "a" will be 1, and not 2. This changes the behavior for overriding attribute values in AQL compared to previous versions of ArangoDB, as previous versions in some cases allowed duplicate attribute names in objects/documents (which is undesired) and in other cases used the _last_ value assigned to an attribute instead of the _first_ value. In order to explicitly override a value in an existing object, use the AQL MERGE function. To avoid all these issues, users are encouraged to use unambiguous attribute names in objects/documents in AQL. Outside of AQL, specifying the same attribute multiple times may even result in a parse error, e.g. when sending such data to ArangoDB's HTTP REST API. * fixed issue #7834: AQL Query crashes instance * Added --server.jwt-secret-keyfile option. * Improve single threaded performance by scheduler optimization. * Releveling logging in maintenance v3.4.1 (2018-12-19) ------------------- * fixed issue #7757: Using multiple filters on nested objects produces wrong results * fixed issue #7763: Collect after update does not execute updates * fixed issue #7586: a running query within the user interface was not shown if the active view was `Running Queries` or `Slow Query History`. * fixed issue #7749: AQL Query result changed for COLLECT used on empty data/array * fixed a rare thread local dead lock situation in replication: If a follower tries to get in sync in the last steps it requires a lock on the leader. If the follower cancels the lock before the leader has succeeded with locking we can end up with one thread being deadlocked. * fix thread shutdown in _WIN32 builds Previous versions used a wrong comparison logic to determine the current thread id when shutting down a thread, leading to threads hanging in their destructors on thread shutdown * reverted accidental change to error handling in geo index In previous versions, if non-valid geo coordinates were contained in the indexed field of a document, the document was simply ignored an not indexed. In 3.4.0, this was accidentally changed to generate an error, which caused the upgrade procedure to break in some cases. * fixed TypeError being thrown instead of validation errors when Foxx manifest validation fails * make AQL REMOVE operations use less memory with the RocksDB storage engine the previous implementation of batch removals read everything to remove into memory first before carrying out the first remove operation. The new version will only read in about 1000 documents each time and then remove these. Queries such as FOR doc IN collection FILTER ... REMOVE doc IN collection will benefit from this change in terms of memory usage. * make `--help-all` now also show all hidden program options Previously hidden program options were only returned when invoking arangod or a client tool with the cryptic `--help-.` option. Now `--help-all` simply retuns them as well. The program options JSON description returned by `--dump-options` was also improved as follows: - the new boolean attribute "dynamic" indicates whether the option has a dynamic default value, i.e. a value that depends on the target host capabilities or configuration - the new boolean attribute "requiresValue" indicates whether a boolean option requires a value of "true" or "false" when specified. If "requiresValue" is false, then the option can be specified without a boolean value following it, and the option will still be set to true, e.g. `--server.authentication` is identical to `--server.authentication true`. - the new "category" attribute will contain a value of "command" for command-like options, such as `--version`, `--dump-options`, `--dump-dependencies` etc., and "option" for all others. * Fixed a bug in synchroneous replication intialisation for, where a shard's db server is rebooted during that period v3.4.0 (2018-12-06) ------------------- * Add license key checking to enterprise version in Docker containers. v3.4.0-rc.5 (2018-11-29) ------------------------ * Persist and check default language (locale) selection. Previously we would not check if the language (`--default-language`) had changed when the server was restarted. This could cause issues with indexes over text fields, as it will resulted in undefined behavior within RocksDB (potentially missing entries, corruption, etc.). Now if the language is changed, ArangoDB will print out an error message on startup and abort. * fixed issue #7522: FILTER logic totally broke for my query in 3.4-rc4 * export version and storage engine in `_admin/cluster/health` for Coordinators and DBServers. * restrict the total amount of data to build up in all in-memory RocksDB write buffers by default to a certain fraction of the available physical RAM. This helps restricting memory usage for the arangod process, but may have an effect on the RocksDB storage engine's write performance. In ArangoDB 3.3 the governing configuration option `--rocksdb.total-write-buffer-size` had a default value of `0`, which meant that the memory usage was not limited. ArangoDB 3.4 now changes the default value to about 50% of available physical RAM, and 512MiB for setups with less than 4GiB of RAM. * lower default value for `--cache.size` startup option from about 30% of physical RAM to about 25% percent of physical RAM. * fix internal issue #2786: improved confirmation dialog when clicking the truncate button in the web UI * Updated joi library (web UI), improved Foxx mount path validation * disable startup warning for Linux kernel variable `vm.overcommit_memory` settings values of 0 or 1. Effectively `overcommit_memory` settings value of 0 or 1 fix two memory-allocation related issues with the default memory allocator used in ArangoDB release builds on 64bit Linux. The issues will remain when running with an `overcommit_memory` settings value of 2, so this is now discouraged. Setting `overcommit_memory` to 0 or 1 (0 is the Linux kernel's default) fixes issues with increasing numbers of memory mappings for the arangod process (which may lead to an out-of-memory situation if the kernel's maximum number of mappings threshold is hit) and an increasing amount of memory that the kernel counts as "committed". With an `overcommit_memory` setting of 0 or 1, an arangod process may either be killed by the kernel's OOM killer or will die with a segfault when accessing memory it has allocated before but the kernel could not provide later on. This is still more acceptable than the kernel not providing any more memory to the process when there is still physical memory left, which may have occurred with an `overcommit_memory` setting of 2 after the arangod process had done lots of allocations. In summary, the recommendation for the `overcommit_memory` setting is now to set it to 0 or 1 (0 is kernel default) and not use 2. * fixed Foxx complaining about valid `$schema` value in manifest.json * fix for supervision, which started failing servers using old transient store * fixed a bug where indexes are used in the cluster while still being built on the db servers * fix move leader shard: wait until all but the old leader are in sync. This fixes some unstable tests. * cluster health features more elaborate agent records * agency's supervision edited for advertised endpoints v3.4.0-rc.4 (2018-11-04) ------------------------ * fixed Foxx queues not retrying jobs with infinite `maxFailures` * increase AQL query string parsing performance for queries with many (100K+) string values contained in the query string * increase timeouts for inter-node communication in the cluster * fixed undefined behavior in `/_api/import` when importing a single document went wrong * replication bugfixes * stop printing `connection class corrupted` in arangosh when just starting the arangosh without a connection to a server and running code such as `require("internal")`, the shell always printed "connection class corrupted", which was somewhat misleading. * add separate option `--query.slow-streaming-threshold` for tracking slow streaming queries with a different timeout value * increase maximum number of collections/shards in an AQL query from 256 to 2048 * don't rely on `_modules` collection being present and usable for arangod startup * force connection timeout to be 7 seconds to allow libcurl time to retry lost DNS queries. * fixes a routing issue within the web ui after the use of views * fixes some graph data parsing issues in the ui, e.g. cleaning up duplicate edges inside the graph viewer. * in a cluster environment, the arangod process now exits if wrong credentials are used during the startup process. * added option `--rocksdb.total-write-buffer-size` to limit total memory usage across all RocksDB in-memory write buffers * suppress warnings from statistics background threads such as `WARNING caught exception during statistics processing: Expecting Object` during version upgrade v3.4.0-rc.3 (2018-10-23) ------------------------ * fixed handling of broken Foxx services Installation now also fails when the service encounters an error when executed. Upgrading or replacing with a broken service will still result in the broken services being installed. * restored error pages for broken Foxx services Services that could not be executed will now show an error page (with helpful information if development mode is enabled) instead of a generic 404 response. Requests to the service that do not prefer HTML (i.e. not a browser window) will receive a JSON formatted 503 error response instead. * added support for `force` flag when upgrading Foxx services Using the `force` flag when upgrading or replacing a service falls back to installing the service if it does not already exist. * The order of JSON object attribute keys in JSON return values will now be "random" in more cases. In JSON, there is no defined order for object attribute keys anyway, so ArangoDB is taking the freedom to return the attribute keys in a non-deterministic, seemingly unordered way. * Fixed an AQL bug where the `optimize-traversals` rule was falsely applied to extensions with inline expressions and thereby ignoring them * fix side-effects of sorting larger arrays (>= 16 members) of constant literal values in AQL, when the array was used not only for IN-value filtering but also later in the query. The array values were sorted so the IN-value lookup could use a binary search instead of a linear search, but this did not take into account that the array could have been used elsewhere in the query, e.g. as a return value. The fix will create a copy of the array and sort the copy, leaving the original array untouched. * disallow empty LDAP password * fixes validation of allowed or not allowed foxx service mount paths within the Web UI * The single database or single coordinator statistics in a cluster environment within the Web UI sometimes got called way too often. This caused artifacts in the graphs, which is now fixed. * An aardvark statistics route could not collect and sum up the statistics of all coordinators if one of them was ahead and had more results than the others * Web UI now checks if server statistics are enabled before it sends its first request to the statistics API * fix internal issue #486: immediate deletion (right after creation) of a view with a link to one collection and indexed data reports failure but removes the link * fix internal issue #480: link to a collection is not added to a view if it was already added to other view * fix internal issues #407, #445: limit ArangoSearch memory consumption so that it won't cause OOM while indexing large collections * upgraded arangodb starter version to 0.13.5 * removed undocumented `db..toArray()` function from ArangoShell * prevent creation of collections and views with the same in cluster setups * fixed issue #6770: document update: ignoreRevs parameter ignored * added AQL query optimizer rules `simplify-conditions` and `fuse-filters` * improve inter-server communication performance: - move all response processing off Communicator's socket management thread - create multiple Communicator objects with ClusterComm, route via round robin - adjust Scheduler threads to always be active, and have designated priorities. * fix internal issue #2770: the Query Profiling modal dialog in the Web UI was slightly malformed. * fix internal issue #2035: the Web UI now updates its indices view to check whether new indices exist or not. * fix internal issue #6808: newly created databases within the Web UI did not appear when used Internet Explorer 11 as a browser. * fix internal issue #2957: the Web UI was not able to display more than 1000 documents, even when it was set to a higher amount. * fix internal issue #2688: the Web UI's graph viewer created malformed node labels if a node was expanded multiple times. * fix internal issue #2785: web ui's sort dialog sometimes got rendered, even if it should not. * fix internal issue #2764: the waitForSync property of a satellite collection could not be changed via the Web UI * dynamically manage libcurl's number of open connections to increase performance by reducing the number of socket close and then reopen cycles * recover short server id from agency after a restart of a cluster node this fixes problems with short server ids being set to 0 after a node restart, which then prevented cursor result load-forwarding between multiple coordinators to work properly this should fix arangojs#573 * increased default timeouts in replication this decreases the chances of followers not getting in sync with leaders because of replication operations timing out * include forward-ported diagnostic options for debugging LDAP connections * fixed internal issue #3065: fix variable replacements by the AQL query optimizer in arangosearch view search conditions The consequence of the missing replacements was that some queries using view search conditions could have failed with error messages such as "missing variable #3 (a) for node #7 (EnumerateViewNode) while planning registers" * fixed internal issue #1983: the Web UI was showing a deletion confirmation multiple times. * Restricted usage of views in AQL, they will throw an error now (e.g. "FOR v, e, p IN 1 OUTBOUND @start edgeCollection, view") instead of failing the server. * Allow VIEWs within the AQL "WITH" statement in cluster environment. This will now prepare the query for all collections linked within a view. (e.g. "WITH view FOR v, e, p IN OUTBOUND 'collectionInView/123' edgeCollection" will now be executed properly and not fail with unregistered collection any more) * Properly check permissions for all collections linked to a view when instantiating an AQL query in cluster environment * support installation of ArangoDB on Windows into directories with multibyte character filenames on Windows platforms that used a non-UTF8-codepage This was supported on other platforms before, but never worked for ArangoDB's Windows version * display shard synchronization progress for collections outside of the `_system` database * change memory protection settings for memory given back to by the bundled JEMalloc memory allocator. This avoids splitting of existing memory mappings due to changes of the protection settings * added missing implementation for `DeleteRangeCF` in RocksDB WAL tailing handler * fixed agents busy looping gossip * handle missing `_frontend` collections gracefully the `_frontend` system collection is not required for normal ArangoDB operations, so if it is missing for whatever reason, ensure that normal operations can go on. v3.4.0-rc.2 (2018-09-30) ------------------------ * upgraded arangosync version to 0.6.0 * upgraded arangodb starter version to 0.13.3 * fixed issue #6611: Properly display JSON properties of user defined foxx services configuration within the web UI * improved shards display in web UI: included arrows to better visualize that collection name sections can be expanded and collapsed * added nesting support for `aql` template strings * added support for `undefined` and AQL literals to `aql.literal` * added `aql.join` function * fixed issue #6583: Agency node segfaults if sent an authenticated HTTP request is sent to its port * fixed issue #6601: Context cancelled (never ending query) * added more AQL query results cache inspection and control functionality * fixed undefined behavior in AQL query result cache * the query editor within the web UI is now catching HTTP 501 responses properly * added AQL VERSION function to return the server version as a string * added startup parameter `--cluster.advertised-endpoints` * AQL query optimizer now makes better choices regarding indexes to use in a query when there are multiple competing indexes and some of them are prefixes of others In this case, the optimizer could have preferred indexes that covered less attributes, but it should rather pick the indexes that covered more attributes. For example, if there was an index on ["a"] and another index on ["a", "b"], then previously the optimizer may have picked the index on just ["a"] instead the index on ["a", "b"] for queries that used all index attributes but did range queries on them (e.g. `FILTER doc.a == @val1 && doc.b >= @val2`). * Added compression for the AQL intermediate results transfer in the cluster, leading to less data being transferred between coordinator and database servers in many cases * forward-ported a bugfix from RocksDB (https://github.com/facebook/rocksdb/pull/4386) that fixes range deletions (used internally in ArangoDB when dropping or truncating collections) The non-working range deletes could have triggered errors such as `deletion check in index drop failed - not all documents in the index have been deleted.` when dropping or truncating collections * improve error messages in Windows installer * allow retrying installation in Windows installer in case an existing database is still running and needs to be manually shut down before continuing with the installation * fix database backup functionality in Windows installer * fixed memory leak in `/_api/batch` REST handler * `db._profileQuery()` now also tracks operations triggered when using `LIMIT` clauses in a query * added proper error messages when using views as an argument to AQL functions (doing so triggered an `internal error` before) * fixed return value encoding for collection ids ("cid" attribute") in REST API `/_api/replication/logger-follow` * fixed dumping and restoring of views with arangodump and arangorestore * fix replication from 3.3 to 3.4 * fixed some TLS errors that occurred when combining HTTPS/TLS transport with the VelocyStream protocol (VST) That combination could have led to spurious errors such as "TLS padding error" or "Tag mismatch" and connections being closed * make synchronous replication detect more error cases when followers cannot apply the changes from the leader * fixed issue #6379: RocksDB arangorestore time degeneration on dead documents * fixed issue #6495: Document not found when removing records * fixed undefined behavior in cluster plan-loading procedure that may have unintentionally modified a shared structure * reduce overhead of function initialization in AQL COLLECT aggregate functions, for functions COUNT/LENGTH, SUM and AVG this optimization will only be noticable when the COLLECT produces many groups and the "hash" COLLECT variant is used * fixed potential out-of-bounds access in admin log REST handler `/_admin/log`, which could have led to the server returning an HTTP 500 error * catch more exceptions in replication and handle them appropriately * agency endpoint updates now go through RAFT * fixed a cleanup issue in Current when a follower was removed from Plan * catch exceptions in MaintenanceWorker thread * fixed a bug in cleanOutServer which could lead to a cleaned out server still being a follower for some shard v3.4.0-rc.1 (2018-09-06) ------------------------ * Release Candidate for 3.4.0, please check the `ReleaseNotes/KnownIssues34.md` file for a list of known issues. * upgraded bundled RocksDB version to 5.16.0 * upgraded bundled Snappy compression library to 1.1.7 * fixed issue #5941: if using breadth first search in traversals uniqueness checks on path (vertices and edges) have not been applied. In SmartGraphs the checks have been executed properly. * added more detailed progress output to arangorestore, showing the percentage of how much data is restored for bigger collections plus a set of overview statistics after each processed collection * added option `--rocksdb.use-file-logging` to enable writing of RocksDB's own informational LOG files into RocksDB's database directory. This option is turned off by default, but can be enabled for debugging RocksDB internals and performance. * improved error messages when managing Foxx services Install/replace/upgrade will now provide additional information when an error is encountered during setup. Errors encountered during a `require` call will also include information about the underlying cause in the error message. * fixed some Foxx script names being displayed incorrectly in web UI and Foxx CLI * major revision of the maintenance feature * added `uuidv4` and `genRandomBytes` methods to crypto module * added `hexSlice` methods `hexWrite` to JS Buffer type * added `Buffer.from`, `Buffer.of`, `Buffer.alloc` and `Buffer.allocUnsafe` for improved compatibility with Node.js * Foxx HTTP API errors now log stacktraces * fixed issue #5831: custom queries in the ui could not be loaded if the user only has read access to the _system database. * fixed issue #6128: ArangoDb Cluster: Task moved from DBS to Coordinator * fixed some web ui action events related to Running Queries view and Slow Queries History view * fixed internal issue #2566: corrected web UI alignment of the nodes table * fixed issue #5736: Foxx HTTP API responds with 500 error when request body is too short * fixed issue #6106: Arithmetic operator type casting documentation incorrect * The arangosh now supports the velocystream transport protocol via the schemas "vst+tcp://", "vst+ssl://", "vst+unix://" schemes. * The server will no longer lowercase the input in --server.endpoint. This means Unix domain socket paths will now be treated as specified, previously they were lowercased * fixed logging of requests. A wrong log level was used * fixed issue #5943: misplaced database ui icon and wrong cursor type were used * fixed issue #5354: updated the web UI JSON editor, improved usability * fixed issue #5648: fixed error message when saving unsupported document types * fixed internal issue #2812: Cluster fails to create many indexes in parallel * Added C++ implementation, load balancer support, and user restriction to Pregel API. If an execution is accessed on a different coordinator than where it was created, the request(s) will be forwarded to the correct coordinator. If an execution is accessed by a different user than the one who created it, the request will be denied. * the AQL editor in the web UI now supports detailed AQL query profiling * fixed issue #5884: Subquery nodes are no longer created on DBServers * intermediate commits in the RocksDB engine are now only enabled in standalone AQL queries (not within a JS transaction), standalone truncate as well as for the "import" API * the AQL editor in the web UI now supports GeoJSON types and is able to render them. * fixed issue #5035: fixed a vulnerability issue within the web ui's index view * PR #5552: add "--latency true" option to arangoimport. Lists microsecond latency * added `"pbkdf2"` method to `@arangodb/foxx/auth` module * the `@arangodb/foxx/auth` module now uses a different method to generate salts, so salts are no longer guaranteed to be alphanumeric * fixed internal issue #2567: the Web UI was showing the possibility to move a shard from a follower to the current leader * Renamed RocksDB engine-specific statistics figure `rocksdb.block-cache-used` to `rocksdb.block-cache-usage` in output of `db._engineStats()` The new figure name is in line with the statistics that the RocksDB library provides in its new versions. * Added RocksDB engine-specific statistics figures `rocksdb.block-cache-capacity`, `rocksdb.block-cache-pinned-usage` as well as level-specific figures `rocksdb.num-files-at-level` and `rocksdb.compression-ratio-at-level` in output of `db._engineStats()` * Added RocksDB-engine configuration option `--rocksdb.block-align-data-blocks` If set to true, data blocks are aligned on lesser of page size and block size, which may waste some memory but may reduce the number of cross-page I/Os operations. * Usage RocksDB format version 3 for new block-based tables * Bugfix: The AQL syntax variants `UPDATE/REPLACE k WITH d` now correctly take _rev from k instead of d (when ignoreRevs is false) and ignore d._rev. * Added C++ implementation, load balancer support, and user restriction to tasks API If a task is accessed on a different coordinator than where it was created, the request(s) will be forwarded to the correct coordinator. If a task is accessed by a different user than the one who created it, the request will be denied. * Added load balancer support and user-restriction to async jobs API. If an async job is accessed on a different coordinator than where it was created, the request(s) will be forwarded to the correct coordinator. If a job is accessed by a different user than the one who created it, the request will be denied. * switch default storage engine from MMFiles to RocksDB In ArangoDB 3.4, the default storage engine for new installations is the RocksDB engine. This differs to previous versions (3.2 and 3.3), in which the default storage engine was the MMFiles engine. The MMFiles engine can still be explicitly selected as the storage engine for all new installations. It's only that the "auto" setting for selecting the storage engine will now use the RocksDB engine instead of MMFiles engine. In the following scenarios, the effectively selected storage engine for new installations will be RocksDB: * `--server.storage-engine rocksdb` * `--server.storage-engine auto` * `--server.storage-engine` option not specified The MMFiles storage engine will be selected for new installations only when explicitly selected: * `--server.storage-engine mmfiles` On upgrade, any existing ArangoDB installation will keep its previously selected storage engine. The change of the default storage engine is thus only relevant for new ArangoDB installations and/or existing cluster setups for which new server nodes get added later. All server nodes in a cluster setup should use the same storage engine to work reliably. Using different storage engines in a cluster is unsupported. * added collection.indexes() as an alias for collection.getIndexes() * disable V8 engine and JavaScript APIs for agency nodes * renamed MMFiles engine compactor thread from "Compactor" to "MMFilesCompactor". This change will be visible only on systems which allow assigning names to threads. * added configuration option `--rocksdb.sync-interval` This option specifies interval (in milliseconds) that ArangoDB will use to automatically synchronize data in RocksDB's write-ahead log (WAL) files to disk. Automatic syncs will only be performed for not-yet synchronized data, and only for operations that have been executed without the *waitForSync* attribute. Automatic synchronization is performed by a background thread. The default sync interval is 100 milliseconds. Note: this option is not supported on Windows platforms. Setting the sync interval to a value greater 0 will produce a startup warning. * added AQL functions `TO_BASE64`, `TO_HEX`, `ENCODE_URI_COMPONENT` and `SOUNDEX` * PR #5857: RocksDB engine would frequently request a new DelayToken. This caused excessive write delay on the next Put() call. Alternate approach taken. * changed the thread handling in the scheduler. `--server.maximal-threads` will be the maximum number of threads for the scheduler. * The option `--server.threads` is now obsolete. * use sparse indexes in more cases now, when it is clear that the index attribute value cannot be null * introduce SingleRemoteOperationNode via "optimize-cluster-single-document-operations" optimizer rule, which triggers single document operations directly from the coordinator instead of using a full-featured AQL setup. This saves cluster roundtrips. Queries directly referencing the document key benefit from this: UPDATE {_key: '1'} WITH {foo: 'bar'} IN collection RETURN OLD * Added load balancer support and user-restriction to cursor API. If a cursor is accessed on a different coordinator than where it was created, the requests will be forwarded to the correct coordinator. If a cursor is accessed by a different user than the one who created it, the request will be denied. * if authentication is turned on requests to databases by users with insufficient rights will be answered with the HTTP forbidden (401) response. * upgraded bundled RocksDB library version to 5.15 * added key generators `uuid` and `padded` The `uuid` key generator generates universally unique 128 bit keys, which are stored in hexadecimal human-readable format. The `padded` key generator generates keys of a fixed length (16 bytes) in ascending lexicographical sort order. * The REST API of `/_admin/status` added: "operationMode" filed with same meaning as the "mode" field and field "readOnly" that has the inverted meaning of the field "writeOpsEnabled". The old field names will be deprecated in upcoming versions. * added `COUNT_DISTINCT` AQL function * make AQL optimizer rule `collect-in-cluster` optimize aggregation functions `AVERAGE`, `VARIANCE`, `STDDEV`, `UNIQUE`, `SORTED_UNIQUE` and `COUNT_DISTINCT` in a cluster by pushing parts of the aggregation onto the DB servers and only doing the total aggregation on the coordinator * replace JavaScript functions FULLTEXT, NEAR, WITHIN and WITHIN_RECTANGLE with regular AQL subqueries via a new optimizer rule "replace-function-with-index". * the existing "fulltext-index-optimizer" optimizer rule has been removed because its duty is now handled by the "replace-function-with-index" rule. * added option "--latency true" option to arangoimport. Lists microsecond latency statistics on 10 second intervals. * fixed internal issue #2256: ui, document id not showing up when deleting a document * fixed internal issue #2163: wrong labels within foxx validation of service input parameters * fixed internal issue #2160: fixed misplaced tooltips in indices view * Added exclusive option for rocksdb collections. Modifying AQL queries can now set the exclusive option as well as it can be set on JavaScript transactions. * added optimizer rule "optimize-subqueries", which makes qualifying subqueries return less data The rule fires in the following situations: * in case only a few results are used from a non-modifying subquery, the rule will add a LIMIT statement into the subquery. For example LET docs = ( FOR doc IN collection FILTER ... RETURN doc ) RETURN docs[0] will be turned into LET docs = ( FOR doc IN collection FILTER ... LIMIT 1 RETURN doc ) RETURN docs[0] Another optimization performed by this rule is to modify the result value of subqueries in case only the number of results is checked later. For example RETURN LENGTH( FOR doc IN collection FILTER ... RETURN doc ) will be turned into RETURN LENGTH( FOR doc IN collection FILTER ... RETURN true ) This saves copying the document data from the subquery to the outer scope and may enable follow-up optimizations. * fixed Foxx queues bug when queues are created in a request handler with an ArangoDB authentication header * abort startup when using SSLv2 for a server endpoint, or when connecting with a client tool via an SSLv2 connection. SSLv2 has been disabled in the OpenSSL library by default in recent versions because of security vulnerabilities inherent in this protocol. As it is not safe at all to use this protocol, the support for it has also been stopped in ArangoDB. End users that use SSLv2 for connecting to ArangoDB should change the protocol from SSLv2 to TLSv12 if possible, by adjusting the value of the `--ssl.protocol` startup option. * added `overwrite` option to document insert operations to allow for easier syncing. This implements almost the much inquired UPSERT. In reality it is a REPSERT (replace/insert) because only replacement and not modification of documents is possible. The option does not work in cluster collections with custom sharding. * added startup option `--log.escape` This option toggles the escaping of log output. If set to `true` (which is the default value), then the logging will work as before, and the following characters in the log output are escaped: * the carriage return character (hex 0d) * the newline character (hex 0a) * the tabstop character (hex 09) * any other characters with an ordinal value less than hex 20 If the option is set to `false`, no characters are escaped. Characters with an ordinal value less than hex 20 will not be printed in this mode but will be replaced with a space character (hex 20). A side effect of turning off the escaping is that it will reduce the CPU overhead for the logging. However, this will only be noticable when logging is set to a very verbose level (e.g. debug or trace). * increased the default values for the startup options `--javascript.gc-interval` from every 1000 to every 2000 requests, and for `--javascript.gc-frequency` from 30 to 60 seconds This will make the V8 garbage collection run less often by default than in previous versions, reducing CPU load a bit and leaving more contexts available on average. * added `/_admin/repair/distributeShardsLike` that repairs collections with distributeShardsLike where the shards aren't actually distributed like in the prototype collection, as could happen due to internal issue #1770 * Fixed issue #4271: Change the behavior of the `fullCount` option for AQL query cursors so that it will only take into account `LIMIT` statements on the top level of the query. `LIMIT` statements in subqueries will not have any effect on the `fullCount` results any more. * We added a new geo-spatial index implementation. On the RocksDB storage engine all installations will need to be upgraded with `--database.auto-upgrade true`. New geo indexes will now only report with the type `geo` instead of `geo1` or `geo2`. The index types `geo1` and `geo2` are now deprecated. Additionally we removed the deprecated flags `constraint` and `ignoreNull` from geo index definitions, these fields were initially deprecated in ArangoDB 2.5 * Add revision id to RocksDB values in primary indexes to speed up replication (~10x). * PR #5238: Create a default pacing algorithm for arangoimport to avoid TimeoutErrors on VMs with limited disk throughput * Starting a cluster with coordinators and DB servers using different storage engines is unsupported. Doing it anyway will now produce a warning on startup * fixed issue #4919: C++ implementation of LIKE function now matches the old and correct behavior of the javascript implementation. * added `--json` option to arangovpack, allowing to treat its input as plain JSON data make arangovpack work without any configuration file * added experimental arangodb startup option `--javascript.enabled` to enable/disable the initialization of the V8 JavaScript engine. Only expected to work on single-servers and agency deployments * pull request #5201: eliminate race scenario where handlePlanChange could run infinite times after an execution exceeded 7.4 second time span * UI: fixed an unreasonable event bug within the modal view engine * pull request #5114: detect shutdown more quickly on heartbeat thread of coordinator and DB servers * fixed issue #3811: gharial api is now checking existence of `_from` and `_to` vertices during edge creation * There is a new method `_profileQuery` on the database object to execute a query and print an explain with annotated runtime information. * Query cursors can now be created with option `profile`, with a value of 0, 1 or 2. This will cause queries to include more statistics in their results and will allow tracing of queries. * fixed internal issue #2147: fixed database filter in UI * fixed internal issue #2149: number of documents in the UI is not adjusted after moving them * fixed internal issue #2150: UI - loading a saved query does not update the list of bind parameters * removed option `--cluster.my-local-info` in favor of persisted server UUIDs The option `--cluster.my-local-info` was deprecated since ArangoDB 3.3. * added new collection property `cacheEnabled` which enables in-memory caching for documents and primary index entries. Available only when using RocksDB * arangodump now supports `--threads` option to dump collections in parallel * arangorestore now supports `--threads` option to restore collections in parallel * Improvement: The AQL query planner in cluster is now a bit more clever and can prepare AQL queries with less network overhead. This should speed up simple queries in cluster mode, on complex queries it will most likely not show any performance effect. It will especially show effects on collections with a very high amount of Shards. * removed remainders of dysfunctional `/_admin/cluster-test` and `/_admin/clusterCheckPort` API endpoints and removed them from documentation * added new query option `stream` to enable streaming query execution via the `POST /_api/cursor` rest interface. * fixed issue #4698: databases within the UI are now displayed in a sorted order. * Behavior of permissions for databases and collections changed: The new fallback rule for databases for which an access level is not explicitly specified: Choose the higher access level of: * A wildcard database grant * A database grant on the `_system` database The new fallback rule for collections for which an access level is not explicitly specified: Choose the higher access level of: * Any wildcard access grant in the same database, or on "*/*" * The access level for the current database * The access level for the `_system` database * fixed issue #4583: add AQL ASSERT and AQL WARN * renamed startup option `--replication.automatic-failover` to `--replication.active-failover` using the old option name will still work in ArangoDB 3.4, but the old option will be removed afterwards * index selectivity estimates for RocksDB engine are now eventually consistent This change addresses a previous issue where some index updates could be "lost" from the view of the internal selectivity estimate, leading to inaccurate estimates. The issue is solved now, but there can be up to a second or so delay before updates are reflected in the estimates. * support `returnOld` and `returnNew` attributes for in the following HTTP REST APIs: * /_api/gharial//vertex/ * /_api/gharial//edge/ The exception from this is that the HTTP DELETE verb for these APIs does not support `returnOld` because that would make the existing API incompatible * fixed internal issue #478: remove unused and undocumented REST API endpoints _admin/statistics/short and _admin/statistics/long These APIs were available in ArangoDB's REST API, but have not been called by ArangoDB itself nor have they been part of the documented API. They have been superseded by other REST APIs and were partially dysfunctional. Therefore these two endpoints have been removed entirely. * fixed issue #1532: reload users on restore * fixed internal issue #1475: when restoring a cluster dump to a single server ignore indexes of type primary and edge since we mustn't create them here. * fixed internal issue #1439: improve performance of any-iterator for RocksDB * issue #1190: added option `--create-database` for arangoimport * UI: updated dygraph js library to version 2.1.0 * renamed arangoimp to arangoimport for consistency Release packages will still install arangoimp as a symlink so user scripts invoking arangoimp do not need to be changed * UI: Shard distribution view now has an accordion view instead of displaying all shards of all collections at once. * fixed issue #4393: broken handling of unix domain sockets in JS_Download * added AQL function `IS_KEY` this function checks if the value passed to it can be used as a document key, i.e. as the value of the `_key` attribute * added AQL functions `SORTED` and `SORTED_UNIQUE` `SORTED` will return a sorted version of the input array using AQL's internal comparison order `SORTED_UNIQUE` will do the same, but additionally removes duplicates. * added C++ implementation for AQL functions `DATE_NOW`, `DATE_ISO8601`, `DATE_TIMESTAMP`, `IS_DATESTRING`, `DATE_DAYOFWEEK`, `DATE_YEAR`, `DATE_MONTH`, `DATE_DAY`, `DATE_HOUR`, `DATE_MINUTE`, `DATE_SECOND`, `DATE_MILLISECOND`, `DATE_DAYOFYEAR`, `DATE_ISOWEEK`, `DATE_LEAPYEAR`, `DATE_QUARTER`, `DATE_DAYS_IN_MONTH`, `DATE_ADD`, `DATE_SUBTRACT`, `DATE_DIFF`, `DATE_COMPARE`, `TRANSLATE` and `SHA512` * fixed a bug where clusterinfo missed changes to plan after agency callback is registred for create collection * Foxx manifest.json files can now contain a $schema key with the value of "http://json.schemastore.org/foxx-manifest" to improve tooling support. * fixed agency restart from compaction without data * fixed agency's log compaction for internal issue #2249 * only load Plan and Current from agency when actually needed v3.3.18 (XXXX-XX-XX) -------------------- * improved logging in case of replication errors * recover short server id from agency after a restart of a cluster node this fixes problems with short server ids being set to 0 after a node restart, which then prevented cursor result load-forwarding between multiple coordinators to work properly this should fix arangojs#573 * increased default timeouts in replication this decreases the chances of followers not getting in sync with leaders because of replication operations timing out * fixed internal issue #1983: the Web UI was showing a deletion confirmation multiple times. * handle missing `_frontend` collections gracefully the `_frontend` system collection is not required for normal ArangoDB operations, so if it is missing for whatever reason, ensure that normal operations can go on. v3.3.17 (2018-10-04) -------------------- * upgraded arangosync version to 0.6.0 * added several advanced options for configuring and debugging LDAP connections. Please note that some of the following options are platform-specific and may not work on all platforms or with all LDAP servers reliably: - `--ldap.serialized`: whether or not calls into the underlying LDAP library should be serialized. This option can be used to work around thread-unsafe LDAP library functionality. - `--ldap.serialize-timeout`: sets the timeout value that is used when waiting to enter the LDAP library call serialization lock. This is only meaningful when `--ldap.serialized` has been set to `true`. - `--ldap.retries`: number of tries to attempt a connection. Setting this to values greater than one will make ArangoDB retry to contact the LDAP server in case no connection can be made initially. - `--ldap.restart`: whether or not the LDAP library should implicitly restart connections - `--ldap.referrals`: whether or not the LDAP library should implicitly chase referrals - `--ldap.debug`: turn on internal OpenLDAP library output (warning: will print to stdout). - `--ldap.timeout`: timeout value (in seconds) for synchronous LDAP API calls (a value of 0 means default timeout). - `--ldap.network-timeout`: timeout value (in seconds) after which network operations following the initial connection return in case of no activity (a value of 0 means default timeout). - `--ldap.async-connect`: whether or not the connection to the LDAP library will be done asynchronously. * fixed a shutdown race in ArangoDB's logger, which could have led to some buffered log messages being discarded on shutdown * display shard synchronization progress for collections outside of the `_system` database * fixed issue #6611: Properly display JSON properties of user defined foxx services configuration within the web UI * fixed issue #6583: Agency node segfaults if sent an authenticated HTTP request is sent to its port * when cleaning out a leader it could happen that it became follower instead of being removed completely * make synchronous replication detect more error cases when followers cannot apply the changes from the leader * fix some TLS errors that occurred when combining HTTPS/TLS transport with the VelocyStream protocol (VST) That combination could have led to spurious errors such as "TLS padding error" or "Tag mismatch" and connections being closed * agency endpoint updates now go through RAFT v3.3.16 (2018-09-19) -------------------- * fix undefined behavior in AQL query result cache * the query editor within the web ui is now catching http 501 responses properly * fixed issue #6495 (Document not found when removing records) * fixed undefined behavior in cluster plan-loading procedure that may have unintentionally modified a shared structure * reduce overhead of function initialization in AQL COLLECT aggregate functions, for functions COUNT/LENGTH, SUM and AVG this optimization will only be noticable when the COLLECT produces many groups and the "hash" COLLECT variant is used * fixed potential out-of-bounds access in admin log REST handler /_admin/log, which could have led to the server returning an HTTP 500 error * catch more exceptions in replication and handle them appropriately v3.3.15 (2018-09-10) -------------------- * fixed an issue in the "sorted" AQL COLLECT variant, that may have led to producing an incorrect number of results * upgraded arangodb starter version to 0.13.3 * fixed issue #5941 if using breadth-first search in traversals uniqueness checks on path (vertices and edges) have not been applied. In SmartGraphs the checks have been executed properly. * added more detailed progress output to arangorestore, showing the percentage of how much data is restored for bigger collections plus a set of overview statistics after each processed collection * added option `--rocksdb.use-file-logging` to enable writing of RocksDB's own informational LOG files into RocksDB's database directory. This option is turned off by default, but can be enabled for debugging RocksDB internals and performance. * improved error messages when managing Foxx services Install/replace/upgrade will now provide additional information when an error is encountered during setup. Errors encountered during a `require` call will also include information about the underlying cause in the error message. * fixed some Foxx script names being displayed incorrectly in web UI and Foxx CLI * added startup option `--query.optimizer-max-plans value` This option allows limiting the number of query execution plans created by the AQL optimizer for any incoming queries. The default value is `128`. By adjusting this value it can be controlled how many different query execution plans the AQL query optimizer will generate at most for any given AQL query. Normally the AQL query optimizer will generate a single execution plan per AQL query, but there are some cases in which it creates multiple competing plans. More plans can lead to better optimized queries, however, plan creation has its costs. The more plans are created and shipped through the optimization pipeline, the more time will be spent in the optimizer. Lowering this option's value will make the optimizer stop creating additional plans when it has already created enough plans. Note that this setting controls the default maximum number of plans to create. The value can still be adjusted on a per-query basis by setting the *maxNumberOfPlans* attribute when running a query. This change also lowers the default maximum number of query plans from 192 to 128. * bug fix: facilitate faster shutdown of coordinators and db servers * cluster nodes should retry registering in agency until successful * fixed some web ui action events related to Running Queries view and Slow Queries History view * Create a default pacing algorithm for arangoimport to avoid TimeoutErrors on VMs with limited disk throughput * backport PR 6150: establish unique function to indicate when application is terminating and therefore network retries should not occur * backport PR #5201: eliminate race scenario where handlePlanChange could run infinite times after an execution exceeded 7.4 second time span v3.3.14 (2018-08-15) -------------------- * upgraded arangodb starter version to 0.13.1 * Foxx HTTP API errors now log stacktraces * fixed issue #5736: Foxx HTTP API responds with 500 error when request body is too short * fixed issue #5831: custom queries in the ui could not be loaded if the user only has read access to the _system database. * fixed internal issue #2566: corrected web UI alignment of the nodes table * fixed internal issue #2869: when attaching a follower with global applier to an authenticated leader already existing users have not been replicated, all users created/modified later are replicated. * fixed internal issue #2865: dumping from an authenticated arangodb the users have not been included * fixed issue #5943: misplaced database ui icon and wrong cursor type were used * fixed issue #5354: updated the web UI JSON editor, improved usability * fixed issue #5648: fixed error message when saving unsupported document types * fixed issue #6076: Segmentation fault after AQL query This also fixes issues #6131 and #6174 * fixed issue #5884: Subquery nodes are no longer created on DBServers * fixed issue #6031: Broken LIMIT in nested list iterations * fixed internal issue #2812: Cluster fails to create many indexes in parallel * intermediate commits in the RocksDB engine are now only enabled in standalone AQL queries (not within a JS transaction), standalone truncate as well as for the "import" API * Bug fix: race condition could request data from Agency registry that did not exist yet. This caused a throw that would end the Supervision thread. All registry query APIs no longer throw exceptions. v3.3.13 (2018-07-26) -------------------- * fixed internal issue #2567: the Web UI was showing the possibility to move a shard from a follower to the current leader * fixed issue #5977: Unexpected execution plan when subquery contains COLLECT * Bugfix: The AQL syntax variants `UPDATE/REPLACE k WITH d` now correctly take _rev from k instead of d (when ignoreRevs is false) and ignore d._rev. * put an upper bound on the number of documents to be scanned when using `db..any()` in the RocksDB storage engine previous versions of ArangoDB did a scan of a random amount of documents in the collection, up to the total number of documents available. this produced a random selection with a good quality, but needed to scan half the number of documents in the collection on average. The new version will only scan up to 500 documents, so it produces a less random result, but will be a lot faster especially for large collections. The implementation of `any()` for the MMFiles engine remains unchanged. The MMFiles engine will pick a random document from the entire range of the in-memory primary index without performing scans. * return an empty result set instead of an "out of memory" exception when querying the geo index with invalid (out of range) coordinates * added load balancer support and user-restriction to cursor API. If a cursor is accessed on a different coordinator than where it was created, the requests will be forwarded to the correct coordinator. If a cursor is accessed by a different user than the one who created it, the request will be denied. * keep failed follower in followers list in Plan. This increases the changes of a failed follower getting back into sync if the follower comes back after a short time. In this case the follower can try to get in sync again, which normally takes less time than seeding a completely new follower. * fix assertion failure and undefined behavior in Unix domain socket connections, introduced by 3.3.12 * added configuration option `--rocksdb.sync-interval` This option specifies interval (in milliseconds) that ArangoDB will use to automatically synchronize data in RocksDB's write-ahead log (WAL) files to disk. Automatic syncs will only be performed for not-yet synchronized data, and only for operations that have been executed without the *waitForSync* attribute. Automatic synchronization is performed by a background thread. The default sync interval is 0, meaning the automatic background syncing is turned off. Background syncing in 3.3 is opt-in, whereas in ArangoDB 3.4 the default sync interval will be 100 milliseconds. Note: this option is not supported on Windows platforms. Setting the sync interval to a value greater 0 will produce a startup warning. * fixed graph creation sometimes failing with 'edge collection already used in edge def' when the edge definition contained multiple vertex collections, despite the edge definitions being identical * inception could get caught in a trap, where agent configuration version and timeout multiplier lead to incapacitated agency * fixed issue #5827: Batch request handling incompatible with .NET's default ContentType format * fixed agency's log compaction for internal issue #2249 * inspector collects additionally disk data size and storage engine statistics v3.3.12 (2018-07-12) -------------------- * issue #5854: RocksDB engine would frequently request a new DelayToken. This caused excessive write delay on the next Put() call. Alternate approach taken. * fixed graph creation under some circumstances failing with 'edge collection already used in edge def' despite the edge definitions being identical * fixed issue #5727: Edge document with user provided key is inserted as many times as the number of shards, violating the primary index * fixed internal issue #2658: AQL modification queries did not allow `_rev` checking. There is now a new option `ignoreRevs` which can be set to `false` in order to force AQL modification queries to match revision ids before doing any modifications * fixed issue #5679: Replication applier restrictions will crash synchronisation after initial sync * fixed potential issue in RETURN DISTINCT CollectBlock implementation that led to the block producing an empty result * changed communication tasks to use boost strands instead of locks, this fixes a race condition with parallel VST communication over SSL * fixed agency restart from compaction without data * fixed for agent coming back to agency with changed endpoint and total data loss * more patient agency tests to allow for ASAN tests to successfully finish v3.3.11 (2018-06-26) -------------------- * upgraded arangosync version to 0.5.3 * upgraded arangodb starter version to 0.12.0 * fixed internal issue #2559: "unexpected document key" error when custom shard keys are used and the "allowUserKeys" key generator option is set to false * fixed AQL DOCUMENT lookup function for documents for sharded collections with more than a single shard and using a custom shard key (i.e. some shard key attribute other than `_key`). The previous implementation of DOCUMENT restricted to lookup to a single shard in all cases, though this restriction was invalid. That lead to `DOCUMENT` not finding documents in cases the wrong shard was contacted. The fixed implementation in 3.3.11 will reach out to all shards to find the document, meaning it will produce the correct result, but will cause more cluster-internal traffic. This increase in traffic may be high if the number of shards is also high, because each invocation of `DOCUMENT` will have to contact all shards. There will be no performance difference for non-sharded collections or collections that are sharded by `_key` or that only have a single shard. * reimplemented replication view in web UI * fixed internal issue #2256: ui, document id not showing up when deleting a document * fixed internal issue #2163: wrong labels within foxx validation of service input parameters * fixed internal issue #2160: fixed misplaced tooltips in indices view * added new arangoinspect client tool, to help users and customers easily collect information of any ArangoDB server setup, and facilitate troubleshooting for the ArangoDB Support Team v3.3.10 (2018-06-04) -------------------- * make optimizer rule "remove-filter-covered-by-index" not stop after removing a sub-condition from a FILTER statement, but pass the optimized FILTER statement again into the optimizer rule for further optimizations. This allows optimizing away some more FILTER conditions than before. * allow accessing /_admin/status URL on followers too in active failover setup * fix cluster COLLECT optimization for attributes that were in "sorted" variant of COLLECT and that were provided by a sorted index on the collected attribute * apply fulltext index optimization rule for multiple fulltext searches in the same query this fixes https://stackoverflow.com/questions/50496274/two-fulltext-searches-on-arangodb-cluster-v8-is-involved * validate `_from` and `_to` values of edges on updates consistently * fixed issue #5400: Unexpected AQL Result * fixed issue #5429: Frequent 'updated local foxx repository' messages * fixed issue #5252: Empty result if FULLTEXT() is used together with LIMIT offset * fixed issue #5035: fixed a vulnerability issue within the web ui's index view * inception was ignoring leader's configuration v3.3.9 (2018-05-17) ------------------- * added `/_admin/repair/distributeShardsLike` that repairs collections with distributeShardsLike where the shards aren't actually distributed like in the prototype collection, as could happen due to internal issue #1770 * fixed Foxx queues bug when queues are created in a request handler with an ArangoDB authentication header * upgraded arangosync version to 0.5.1 * upgraded arangodb starter version to 0.11.3 * fix cluster upgrading issue introduced in 3.3.8 the issue made arangod crash when starting a DB server with option `--database.auto-upgrade true` * fix C++ implementation of AQL ZIP function to return each distinct attribute name only once. The previous implementation added non-unique attribute names multiple times, which led to follow-up issues. Now if an attribute name occurs multiple times in the input list of attribute names, it will only be incorporated once into the result object, with the value that corresponds to the first occurrence. This fix also changes the V8 implementation of the ZIP function, which now will always return the first value for non-unique attribute names and not the last occurring value. * self heal during a Foxx service install, upgrade or replace no longer breaks the respective operation * make /_api/index, /_api/database and /_api/user REST handlers use the scheduler's internal queue, so they do not run in an I/O handling thread * fixed issue #4919: C++ implementation of LIKE function now matches the old and correct behavior of the JavaScript implementation. * added REST API endpoint /_admin/server/availability for monitoring purposes * UI: fixed an unreasonable event bug within the modal view engine * fixed issue #3811: gharial api is now checking existence of _from and _to vertices during edge creation * fixed internal issue #2149: number of documents in the UI is not adjusted after moving them * fixed internal issue #2150: UI - loading a saved query does not update the list of bind parameters * fixed internal issue #2147 - fixed database filter in UI * fixed issue #4934: Wrong used GeoIndex depending on FILTER order * added `query` and `aql.literal` helpers to `@arangodb` module. * remove post-sort from GatherNode in cluster AQL queries that do use indexes for filtering but that do not require a sorted result This optimization can speed up gathering data from multiple shards, because it allows to remove a merge sort of the individual shards' results. * extend the already existing "reduce-extraction-to-projection" AQL optimizer rule for RocksDB to provide projections of up to 5 document attributes. The previous implementation only supported a projection for a single document attribute. The new implementation will extract up to 5 document attributes from a document while scanning a collection via an EnumerateCollectionNode. Additionally the new version of the optimizer rule can also produce projections when scanning an index via an IndexNode. The optimization is benefial especially for huge documents because it will copy out only the projected attributes from the document instead of copying the entire document data from the storage engine. When applied, the explainer will show the projected attributes in a `projections` remark for an EnumerateCollectionNode or IndexNode. The optimization is limited to the RocksDB storage engine. * added index-only optimization for AQL queries that can satisfy the retrieval of all required document attributes directly from an index. This optimization will be triggered for the RocksDB engine if an index is used that covers all required attributes of the document used later on in the query. If applied, it will save retrieving the actual document data (which would require an extra lookup in RocksDB), but will instead build the document data solely from the index values found. It will only be applied when using up to 5 attributes from the document, and only if the rest of the document data is not used later on in the query. The optimization is currently available for the RocksDB engine for the index types primary, edge, hash, skiplist and persistent. If the optimization is applied, it will show up as "index only" in an AQL query's execution plan for an IndexNode. * added scan-only optimization for AQL queries that iterate over collections or indexes and that do not need to return the actual document values. Not fetching the document values from the storage engine will provide a considerable speedup when using the RocksDB engine, but may also help a bit in case of the MMFiles engine. The optimization will only be applied when full-scanning or index-scanning a collection without refering to any of its documents later on, and, for an IndexNode, if all filter conditions for the documents of the collection are covered by the index. If the optimization is applied, it will show up as "scan only" in an AQL query's execution plan for an EnumerateCollectionNode or an IndexNode. * extend existing "collect-in-cluster" optimizer rule to run grouping, counting and deduplication on the DB servers in several cases, so that the coordinator will only need to sum up the potentially smaller results from the individual shards. The following types of COLLECT queries are covered now: - RETURN DISTINCT expr - COLLECT WITH COUNT INTO ... - COLLECT var1 = expr1, ..., varn = exprn (WITH COUNT INTO ...), without INTO or KEEP - COLLECT var1 = expr1, ..., varn = exprn AGGREGATE ..., without INTO or KEEP, for aggregate functions COUNT/LENGTH, SUM, MIN and MAX. * honor specified COLLECT method in AQL COLLECT options for example, when the user explicitly asks for the COLLECT method to be `sorted`, the optimizer will now not produce an alternative version of the plan using the hash method. additionally, if the user explcitly asks for the COLLECT method to be `hash`, the optimizer will now change the existing plan to use the hash method if possible instead of just creating an alternative plan. `COLLECT ... OPTIONS { method: 'sorted' }` => always use sorted method `COLLECT ... OPTIONS { method: 'hash' }` => use hash if this is technically possible `COLLECT ...` (no options) => create a plan using sorted, and another plan using hash method * added bulk document lookups for MMFiles engine, which will improve the performance of document lookups from an inside an index in case the index lookup produces many documents v3.3.8 (2018-04-24) ------------------- * included version of ArangoDB Starter (`arangodb` binary) updated to v0.10.11, see [Starter changelog](https://github.com/arangodb-helper/arangodb/blob/master/CHANGELOG.md) * added arangod startup option `--dump-options` to print all configuration parameters as a JSON object * fixed: (Enterprise only) If you restore a SmartGraph where the collections are still existing and are supposed to be dropped on restore we ended up in duplicate name error. This is now gone and the SmartGraph is correctly restored. * fix lookups by `_id` in smart graph edge collections * improve startup resilience in case there are datafile errors (MMFiles) also allow repairing broken VERSION files automatically on startup by specifying the option `--database.ignore-datafile-errors true` * fix issue #4582: UI query editor now supports usage of empty string as bind parameter value * fixed internal issue #2148: Number of documents found by filter is misleading in web UI * added startup option `--database.required-directory-state` using this option it is possible to require the database directory to be in a specific state on startup. the options for this value are: - non-existing: database directory must not exist - existing: database directory must exist - empty: database directory must exist but be empty - populated: database directory must exist and contain specific files already - any: any state allowed * field "$schema" in Foxx manifest.json files no longer produce warnings * added `@arangodb/locals` module to expose the Foxx service context as an alternative to using `module.context` directly. * `db._executeTransaction` now accepts collection objects as collections. * supervision can be put into maintenance mode v3.3.7 (2018-04-11) ------------------- * added hidden option `--query.registry-ttl` to control the lifetime of cluster AQL query parts * fixed internal issue #2237: AQL queries on collections with replicationFactor: "satellite" crashed arangod in single server mode * fixed restore of satellite collections: replicationFactor was set to 1 during restore * fixed dump and restore of smart graphs: a) The dump will not include the hidden shadow collections anymore, they were dumped accidentially and only contain duplicated data. b) Restore will now ignore hidden shadow collections as all data is contained in the smart-edge collection. You can manually include these collections from an old dump (3.3.5 or earlier) by using `--force`. c) Restore of a smart-graph will now create smart collections properly instead of getting into `TIMEOUT_IN_CLUSTER_OPERATION` * fixed issue in AQL query optimizer rule "restrict-to-single-shard", which may have sent documents to a wrong shard in AQL INSERT queries that specified the value for `_key` using an expression (and not a constant value) Important: if you were affected by this bug in v3.3.5 it is required that you recreate your dataset in v3.3.6 (i.e. dumping and restoring) instead of doing a simple binary upgrade * added /_admin/status HTTP API for debugging purposes * added ArangoShell helper function for packaging all information about an AQL query so it can be run and analyzed elsewhere: query = "FOR doc IN mycollection FILTER doc.value > 42 RETURN doc"; require("@arangodb/aql/explainer").debugDump("/tmp/query-debug-info", query); Entitled users can send the generated file to the ArangoDB support to facilitate reproduction and debugging. * added hidden option `--server.ask-jwt-secret`. This is an internal option for debugging and should not be exposed to end-users. * fix for internal issue #2215. supervision will now wait for agent to fully prepare before adding 10 second grace period after leadership change * fixed internal issue #2215's FailedLeader timeout bug v3.3.5 (2018-03-28) ------------------- * fixed issue #4934: Wrong used GeoIndex depending on FILTER order * make build id appear in startup log message alongside with other version info * make AQL data modification operations that are sent to all shards and that are supposed to return values (i.e. `RETURN OLD` or `RETURN NEW`) not return fake empty result rows if the document to be updated/replaced/removed was not present on the target shard * added AQL optimizer rule `restrict-to-single-shard` This rule will kick in if a collection operation (index lookup or data modification operation) will only affect a single shard, and the operation can be restricted to the single shard and is not applied for all shards. This optimization can be applied for queries that access a collection only once in the query, and that do not use traversals, shortest path queries and that do not access collection data dynamically using the `DOCUMENT`, `FULLTEXT`, `NEAR` or `WITHIN` AQL functions. Additionally, the optimizer will only pull off this optimization if can safely determine the values of all the collection's shard keys from the query, and when the shard keys are covered by a single index (this is always true if the shard key is the default `_key`) * display missing attributes of GatherNodes in AQL explain output * make AQL optimizer rule `undistribute-remove-after-enum-coll` fire in a few more cases in which it is possible * slightly improve index selection for the RocksDB engine when there are multiple competing indexes with the same attribute prefixes, but different amount of attributes covered. In this case, the more specialized index will be preferred now * fix issue #4924: removeFollower now prefers to remove the last follower(s) * added "collect-in-cluster" optimizer rule to have COLLECT WITH COUNT queries without grouping being executed on the DB servers and the coordinator only summing up the counts from the individual shards * fixed issue #4900: Nested FOR query uses index but ignores other filters * properly exit v8::Context in one place where it was missing before * added hidden option `--cluster.index-create-timeout` for controlling the default value of the index creation timeout in cluster under normal circumstances, this option does not need to be adjusted * increase default timeout for index creation in cluster to 3600s * fixed issue #4843: Query-Result has more Docs than the Collection itself * fixed the behavior of ClusterInfo when waiting for current to catch up with plan in create collection. * fixed issue #4827: COLLECT on edge _to field doesn't group distinct values as expected (MMFiles) v3.3.4 (2018-03-01) ------------------- * fix AQL `fullCount` result value in some cluster cases when it was off a bit * fix issue #4651: Simple query taking forever until a request timeout error * fix issue #4657: fixed incomplete content type header * Vastly improved the Foxx Store UI * fix issue #4677: AQL WITH with bind parameters results in "access after data-modification" for two independent UPSERTs * remove unused startup option `--ldap.permissions-attribute-name` * fix issue #4457: create /var/tmp/arangod with correct user in supervisor mode * remove long disfunctional admin/long_echo handler * fixed Foxx API: * PUT /_api/foxx/service: Respect force flag * PATCH /_api/foxx/service: Check whether a service under given mount exists * internal issue #1726: supervision failed to remove multiple servers from health monitoring at once. * more information from inception, why agent is activated * fixed a bug where supervision tried to deal with shards of virtual collections * fix internal issue #1770: collection creation using distributeShardsLike yields errors and did not distribute shards correctly in the following cases: 1. If numberOfShards * replicationFactor % nrDBServers != 0 (shards * replication is not divisible by DBServers). 2. If there was failover / move shard case on the leading collection and creating the follower collection afterwards. * fix timeout issues in replication client expiration * added missing edge filter to neighbors-only traversals in case a filter condition was moved into the traverser and the traversal was executed in breadth-first mode and was returning each visited vertex exactly once, and there was a filter on the edges of the path and the resulting vertices and edges were not used later, the edge filter was not applied * fixed issue #4160: Run arangod with "--database.auto-upgrade" option always crash silently without error log * fix internal issue #1848: AQL optimizer was trying to resolve attribute accesses to attributes of constant object values at query compile time, but only did so far the very first attribute in each object this fixes https://stackoverflow.com/questions/48648737/beginner-bug-in-for-loops-from-objects * fix inconvenience: If we want to start server with a non-existing --javascript.app-path it will now be created (if possible) * fixed: REST API `POST _api/foxx` now returns HTTP code 201 on success, as documented. returned 200 before. * fixed: REST API `PATCH _api/foxx/dependencies` now updates the existing dependencies instead of replacing them. * fixed: Foxx upload of single javascript file. You now can upload via http-url pointing to a javascript file. * fixed issue #4395: If your foxx app includes an `APP` folder it got accidently removed by selfhealing this is not the case anymore. * fixed internal issue #1969 - command apt-get purge/remove arangodb3e was failing v3.3.3 (2018-01-16) ------------------- * fix issue #4272: VERSION file keeps disappearing * fix internal issue #81: quotation marks disappeared when switching table/json editor in the query editor ui * added option `--rocksdb.throttle` to control whether write-throttling is enabled Write-throttling is turned on by default, to reduce chances of compactions getting too far behind and blocking incoming writes. * fixed issue #4308: Crash when getter for error.name throws an error (on Windows) * UI: fixed a query editor caching and parsing issue * Fixed internal issue #1683: fixes an UI issue where a collection name gets wrongly cached within the documents overview of a collection. * Fixed an issue with the index estimates in RocksDB in the case a transaction is aborted. Former the index estimates were modified if the transaction commited or not. Now they will only be modified if the transaction commited successfully. * UI: optimized login view for very small screen sizes * Truncate in RocksDB will now do intermediate commits every 10.000 documents if truncate fails or the server crashes during this operation all deletes that have been commited so far are persisted. * make the default value of `--rocksdb.block-cache-shard-bits` use the RocksDB default value. This will mostly mean the default number block cache shard bits is lower than before, allowing each shard to store more data and cause less evictions from block cache * issue #4222: Permission error preventing AQL query import / export on webui * UI: optimized error messages for invalid query bind parameter * UI: upgraded swagger ui to version 3.9.0 * issue #3504: added option `--force-same-database` for arangorestore with this option set to true, it is possible to make any arangorestore attempt fail if the specified target database does not match the database name specified in the source dump's "dump.json" file. it can thus be used to prevent restoring data into the "wrong" database The option is set to `false` by default to ensure backwards-compatibility * make the default value of `--rocksdb.block-cache-shard-bits` use the RocksDB default value. This will mostly mean the default number block cache shard bits is lower than before, allowing each shard to store more data and cause less evictions from block cache * fixed issue #4255: AQL SORT consuming too much memory * fixed incorrect persistence of RAFT vote and term v3.3.2 (2018-01-04) ------------------- * fixed issue #4199: Internal failure: JavaScript exception in file 'arangosh.js' at 98,7: ArangoError 4: Expecting type String * fixed issue in agency supervision with a good server being left in failedServers * distinguish isReady and allInSync in clusterInventory * fixed issue #4197: AQL statement not working in 3.3.1 when upgraded from 3.2.10 * do not reuse collection ids when restoring collections from a dump, but assign new collection ids, this should prevent collection id conflicts v3.3.1 (2017-12-28) ------------------- * UI: displayed wrong wfs property for a collection when using RocksDB as storage engine * added `--ignore-missing` option to arangoimp this option allows importing lines with less fields than specified in the CSV header line * changed misleading error message from "no leader" to "not a leader" * optimize usage of AQL FULLTEXT index function to a FOR loop with index usage in some cases When the optimization is applied, this especially speeds up fulltext index queries in the cluster * UI: improved the behavior during collection creation in a cluster environment * Agency lockup fixes for very small machines. * Agency performance improvement by finer grained locking. * Use steady_clock in agency whereever possible. * Agency prevent Supervision thread crash. * Fix agency integer overflow in timeout calculation. v3.3.0 (2017-12-14) ------------------- * release version * added a missing try/catch block in the supervision thread v3.3.rc8 (2017-12-12) --------------------- * UI: fixed broken Foxx configuration keys. Some valid configuration values could not be edited via the ui. * UI: pressing the return key inside a select2 box no longer triggers the modal's success function * UI: coordinators and db servers are now in sorted order (ascending) v3.3.rc7 (2017-12-07) --------------------- * fixed issue #3741: fix terminal color output in Windows * UI: fixed issue #3822: disabled name input field for system collections * fixed issue #3640: limit in subquery * fixed issue #3745: Invalid result when using OLD object with array attribute in UPSERT statement * UI: edge collections were wrongly added to from and to vertices select box during graph creation * UI: added not found views for documents and collections * UI: using default user database api during database creation now * UI: the graph viewer backend now picks one random start vertex of the first 1000 documents instead of calling any(). The implementation of "any" is known to scale bad on huge collections with RocksDB. * UI: fixed disappearing of the navigation label in some case special case * UI: the graph viewer now displays updated label values correctly. Additionally the included node/edge editor now closes automatically after a successful node/edge update. * fixed issue #3917: traversals with high maximal depth take extremely long in planning phase. v3.3.rc4 (2017-11-28) --------------------- * minor bug-fixes v3.3.rc3 (2017-11-24) --------------------- * bug-fixes v3.3.rc2 (2017-11-22) --------------------- * UI: document/edge editor now remembering their modes (e.g. code or tree) * UI: optimized error messages for invalid graph definitions. Also fixed a graph renderer cleanup error. * UI: added a delay within the graph viewer while changing the colors of the graph. Necessary due different browser behavior. * added options `--encryption.keyfile` and `--encryption.key-generator` to arangodump and arangorestore * UI: the graph viewer now displays updated label values correctly. Additionally the included node/edge editor now closes automatically after a successful node/edge update. * removed `--recycle-ids` option for arangorestore using that option could have led to problems on the restore, with potential id conflicts between the originating server (the source dump server) and the target server (the restore server) v3.3.rc1 (2017-11-17) --------------------- * add readonly mode REST API * allow compilation of ArangoDB source code with g++ 7 * upgrade minimum required g++ compiler version to g++ 5.4 That means ArangoDB source code will not compile with g++ 4.x or g++ < 5.4 anymore. * AQL: during a traversal if a vertex is not found. It will not print an ERROR to the log and continue with a NULL value, but will register a warning at the query and continue with a NULL value. The situation is not desired as an ERROR as ArangoDB can store edges pointing to non-existing vertex which is perfectly valid, but it may be a n issue on the data model, so users can directly see it on the query now and do not "by accident" have to check the LOG output. * introduce `enforceReplicationFactor` attribute for creating collections: this optional parameter controls if the coordinator should bail out during collection creation if there are not enough DBServers available for the desired `replicationFactor`. * fixed issue #3516: Show execution time in arangosh this change adds more dynamic prompt components for arangosh The following components are now available for dynamic prompts, settable via the `--console.prompt` option in arangosh: - '%t': current time as timestamp - '%a': elpased time since ArangoShell start in seconds - '%p': duration of last command in seconds - '%d': name of current database - '%e': current endpoint - '%E': current endpoint without protocol - '%u': current user The time a command takes can be displayed easily by starting arangosh with `--console.prompt "%p> "`. * make the ArangoShell refill its collection cache when a yet-unknown collection is first accessed. This fixes the following problem: arangosh1> db._collections(); // shell1 lists all collections arangosh2> db._create("test"); // shell2 now creates a new collection 'test' arangosh1> db.test.insert({}); // shell1 is not aware of the collection created // in shell2, so the insert will fail * make AQL `DISTINCT` not change the order of the results it is applied on * incremental transfer of initial collection data now can handle partial responses for a chunk, allowing the leader/master to send smaller chunks (in terms of HTTP response size) and limit memory usage this optimization is only active if client applications send the "offset" parameter in their requests to PUT `/_api/replication/keys/?type=docs` * initial creation of shards for cluster collections is now faster with `replicationFactor` values bigger than 1. this is achieved by an optimization for the case when the collection on the leader is still empty * potential fix for issue #3517: several "filesystem full" errors in logs while there's a lot of disk space * added C++ implementations for AQL function `SUBSTRING()`, `LEFT()`, `RIGHT()` and `TRIM()` * show C++ function name of call site in ArangoDB log output this requires option `--log.line-number` to be set to *true* * UI: added word wrapping to query editor * UI: fixed wrong user attribute name validation, issue #3228 * make AQL return a proper error message in case of a unique key constraint violation. previously it only returned the generic "unique constraint violated" error message but omitted the details about which index caused the problem. This addresses https://stackoverflow.com/questions/46427126/arangodb-3-2-unique-constraint-violation-id-or-key * added option `--server.local-authentication` * UI: added user roles * added config option `--log.color` to toggle colorful logging to terminal * added config option `--log.thread-name` to additionally log thread names * usernames must not start with `:role:`, added new options: --server.authentication-timeout --ldap.roles-attribute-name --ldap.roles-transformation --ldap.roles-search --ldap.superuser-role --ldap.roles-include --ldap.roles-exclude * performance improvements for full collection scans and a few other operations in MMFiles engine * added `--rocksdb.encryption-key-generator` for enterprise * removed `--compat28` parameter from arangodump and replication API older ArangoDB versions will no longer be supported by these tools. * increase the recommended value for `/proc/sys/vm/max_map_count` to a value eight times as high as the previous recommended value. Increasing the values helps to prevent an ArangoDB server from running out of memory mappings. The raised minimum recommended value may lead to ArangoDB showing some startup warnings as follows: WARNING {memory} maximum number of memory mappings per process is 65530, which seems too low. it is recommended to set it to at least 512000 WARNING {memory} execute 'sudo sysctl -w "vm.max_map_count=512000"' * Foxx now warns about malformed configuration/dependency names and aliases in the manifest. v3.2.17 (XXXX-XX-XX) -------------------- * added missing virtual destructor for MMFiles transaction data context object * make synchronous replication detect more error cases when followers cannot apply the changes from the leader * fixed undefined behavior in cluster plan-loading procedure that may have unintentionally modified a shared structure * cluster nodes should retry registering in agency until successful * fixed issue #5354: updated the ui json editor, improved usability * fixed issue #5648: fixed error message when saving unsupported document types * fixed issue #5943: misplaced database ui icon and wrong cursor type were used v3.2.16 (2018-07-12) -------------------- * upgraded arangodb starter version to 0.12.0 * make edge cache initialization and invalidation more portable by avoiding memset on non-POD types * fixed internal issue #2256: ui, document id not showing up when deleting a document * fixed issue #5400: Unexpected AQL Result * Fixed issue #5035: fixed a vulnerability issue within the web ui's index view * issue one HTTP call less per cluster AQL query * self heal during a Foxx service install, upgrade or replace no longer breaks the respective operation * inception was ignoring leader's configuration * inception could get caught in a trap, where agent configuration version and timeout multiplier lead to incapacitated agency * more patient agency tests to allow for ASAN tests to successfully finish * fixed for agent coming back to agency with changed endpoint and total data loss * fixed agency restart from compaction without data v3.2.15 (2018-05-13) -------------------- * upgraded arangodb starter version to 0.11.2 * make /_api/index and /_api/database REST handlers use the scheduler's internal queue, so they do not run in an I/O handling thread * fixed issue #3811: gharial api is now checking existence of _from and _to vertices during edge creation v3.2.14 (2018-04-20) -------------------- * field "$schema" in Foxx manifest.json files no longer produce warnings * added `@arangodb/locals` module to expose the Foxx service context as an alternative to using `module.context` directly. * the internal implementation of REST API `/_api/simple/by-example` now uses C++ instead of JavaScript * supervision can be switched to maintenance mode f.e. for rolling upgrades v3.2.13 (2018-04-13) -------------------- * improve startup resilience in case there are datafile errors (MMFiles) also allow repairing broken VERSION files automatically on startup by specifying the option `--database.ignore-datafile-errors true` * fix issue #4582: UI query editor now supports usage of empty string as bind parameter value * fix issue #4924: removeFollower now prefers to remove the last follower(s) * fixed issue #4934: Wrong used GeoIndex depending on FILTER order * fixed the behavior of clusterinfo when waiting for current to catch up with plan in create collection. * fix for internal issue #2215. supervision will now wait for agent to fully prepare before adding 10 second grace period after leadership change * fixed interal issue #2215 FailedLeader timeout bug v3.2.12 (2018-02-27) -------------------- * remove long disfunctional admin/long_echo handler * fixed Foxx API: * PUT /_api/foxx/service: Respect force flag * PATCH /_api/foxx/service: Check whether a service under given mount exists * fix issue #4457: create /var/tmp/arangod with correct user in supervisor mode * fix internal issue #1848 AQL optimizer was trying to resolve attribute accesses to attributes of constant object values at query compile time, but only did so far the very first attribute in each object this fixes https://stackoverflow.com/questions/48648737/beginner-bug-in-for-loops-from-objects * fix inconvenience: If we want to start server with a non-existing --javascript.app-path it will now be created (if possible) * fixed: REST API `POST _api/foxx` now returns HTTP code 201 on success, as documented. returned 200 before. * fixed: REST API `PATCH _api/foxx/dependencies` now updates the existing dependencies instead of replacing them. * fixed: Foxx upload of single javascript file. You now can upload via http-url pointing to a javascript file. * fixed issue #4395: If your foxx app includes an `APP` folder it got accidently removed by selfhealing this is not the case anymore. * fix internal issue 1770: collection creation using distributeShardsLike yields errors and did not distribute shards correctly in the following cases: 1. If numberOfShards * replicationFactor % nrDBServers != 0 (shards * replication is not divisible by DBServers). 2. If there was failover / move shard case on the leading collection and creating the follower collection afterwards. * fix timeout issues in replication client expiration + fix some inconsistencies in replication for RocksDB engine that could have led to some operations not being shipped from master to slave servers * fix issue #4272: VERSION file keeps disappearing * fix internal issue #81: quotation marks disappeared when switching table/json editor in the query editor ui * make the default value of `--rocksdb.block-cache-shard-bits` use the RocksDB default value. This will mostly mean the default number block cache shard bits is lower than before, allowing each shard to store more data and cause less evictions from block cache * fix issue #4393: broken handling of unix domain sockets in JS_Download * fix internal bug #1726: supervision failed to remove multiple removed servers from health UI * fixed internal issue #1969 - command apt-get purge/remove arangodb3e was failing * fixed a bug where supervision tried to deal with shards of virtual collections v3.2.11 (2018-01-17) -------------------- * Fixed an issue with the index estimates in RocksDB in the case a transaction is aborted. Former the index estimates were modified if the transaction commited or not. Now they will only be modified if the transaction commited successfully. * Truncate in RocksDB will now do intermediate commits every 10.000 documents if truncate fails or the server crashes during this operation all deletes that have been commited so far are persisted. * fixed issue #4308: Crash when getter for error.name throws an error (on Windows) * UI: fixed a query editor caching and parsing issue for arrays and objects * Fixed internal issue #1684: Web UI: saving arrays/objects as bind parameters faulty * Fixed internal issue #1683: fixes an UI issue where a collection name gets wrongly cached within the documents overview of a collection. * issue #4222: Permission error preventing AQL query import / export on webui * UI: optimized login view for very small screen sizes * UI: Shard distribution view now has an accordion view instead of displaying all shards of all collections at once. * UI: optimized error messages for invalid query bind parameter * fixed missing transaction events in RocksDB asynchronous replication * fixed issue #4255: AQL SORT consuming too much memory * fixed issue #4199: Internal failure: JavaScript exception in file 'arangosh.js' at 98,7: ArangoError 4: Expecting type String * fixed issue #3818: Foxx configuration keys cannot contain spaces (will not save) * UI: displayed wrong "waitForSync" property for a collection when using RocksDB as storage engine * prevent binding to the same combination of IP and port on Windows * fixed incorrect persistence of RAFT vote and term v3.2.10 (2017-12-22) -------------------- * replication: more robust initial sync * fixed a bug in the RocksDB engine that would prevent recalculated collection counts to be actually stored * fixed issue #4095: Inconsistent query execution plan * fixed issue #4056: Executing empty query causes crash * fixed issue #4045: Out of memory in `arangorestore` when no access rights to dump files * fixed issue #3031: New Graph: Edge definitions with edges in fromCollections and toCollections * fixed issue #2668: UI: when following wrong link from edge to vertex in nonexisting collection misleading error is printed * UI: improved the behavior during collection creation in a cluster environment * UI: the graph viewer backend now picks one random start vertex of the first 1000 documents instead of calling any(). The implementation of any is known to scale bad on huge collections with rocksdb. * fixed snapshots becoming potentially invalid after intermediate commits in the RocksDB engine * backport agency inquire API changes * fixed issue #3822: Field validation error in ArangoDB UI - Minor * UI: fixed disappearing of the navigation label in some cases * UI: fixed broken foxx configuration keys. Some valid configuration values could not be edited via the ui. * fixed issue #3640: limit in subquery * UI: edge collections were wrongly added to from and to vertices select box during graph creation * fixed issue #3741: fix terminal color output in Windows * fixed issue #3917: traversals with high maximal depth take extremely long in planning phase. * fix equality comparison for MMFiles documents in AQL functions UNIQUE and UNION_DISTINCT v3.2.9 (2017-12-04) ------------------- * under certain conditions, replication could stop. Now fixed by adding an equality check for requireFromPresent tick value * fixed locking for replication context info in RocksDB engine this fixes undefined behavior when parallel requests are made to the same replication context * UI: added not found views for documents and collections * fixed issue #3858: Foxx queues stuck in 'progress' status * allow compilation of ArangoDB source code with g++ 7 * fixed issue #3224: Issue in the Foxx microservices examples * fixed a deadlock in user privilege/permission change routine * fixed a deadlock on server shutdown * fixed some collection locking issues in MMFiles engine * properly report commit errors in AQL write queries to the caller for the RocksDB engine * UI: optimized error messages for invalid graph definitions. Also fixed a graph renderer cleanrenderer cleanup error. * UI: document/edge editor now remembering their modes (e.g. code or tree) * UI: added a delay within the graph viewer while changing the colors of the graph. Necessary due different browser behavior. * fix removal of failed cluster nodes via web interface * back port of ClusterComm::wait fix in devel among other things this fixes too eager dropping of other followers in case one of the followers does not respond in time * transact interface in agency should not be inquired as of now * inquiry tests and blocking of inquiry on AgencyGeneralTransaction v3.2.8 (2017-11-18) ------------------- * fixed a race condition occuring when upgrading via linux package manager * fixed authentication issue during replication v3.2.7 (2017-11-13) ------------------- * Cluster customers, which have upgraded from 3.1 to 3.2 need to upgrade to 3.2.7. The cluster supervision is otherwise not operational. * Fixed issue #3597: AQL with path filters returns unexpected results In some cases breadth first search in combination with vertex filters yields wrong result, the filter was not applied correctly. * fixed some undefined behavior in some internal value caches for AQL GatherNodes and SortNodes, which could have led to sorted results being effectively not correctly sorted. * make the replication applier for the RocksDB engine start automatically after a restart of the server if the applier was configured with its `autoStart` property set to `true`. previously the replication appliers were only automatically restarted at server start for the MMFiles engine. * fixed arangodump batch size adaptivity in cluster mode and upped default batch size for arangodump these changes speed up arangodump in cluster context * smart graphs now return a proper inventory in response to replication inventory requests * fixed issue #3618: Inconsistent behavior of OR statement with object bind parameters * only users with read/write rights on the "_system" database can now execute "_admin/shutdown" as well as modify properties of the write-ahead log (WAL) * increase default maximum number of V8 contexts to at least 16 if not explicitly configured otherwise. the procedure for determining the actual maximum value of V8 contexts is unchanged apart from the value `16` and works as follows: - if explicitly set, the value of the configuration option `--javascript.v8-contexts` is used as the maximum number of V8 contexts - when the option is not set, the maximum number of V8 contexts is determined by the configuration option `--server.threads` if that option is set. if `--server.threads` is not set, then the maximum number of V8 contexts is the server's reported hardware concurrency (number of processors visible to the arangod process). if that would result in a maximum value of less than 16 in any of these two cases, then the maximum value will be increased to 16. * fixed issue #3447: ArangoError 1202: AQL: NotFound: (while executing) when updating collection * potential fix for issue #3581: Unexpected "rocksdb unique constraint violated" with unique hash index * fixed geo index optimizer rule for geo indexes with a single (array of coordinates) attribute. * improved the speed of the shards overview in cluster (API endpoint /_api/cluster/shardDistribution API) It is now guaranteed to return after ~2 seconds even if the entire cluster is unresponsive. * fix agency precondition check for complex objects this fixes issues with several CAS operations in the agency * several fixes for agency restart and shutdown * the cluster-internal representation of planned collection objects is now more lightweight than before, using less memory and not allocating any cache for indexes etc. * fixed issue #3403: How to kill long running AQL queries with the browser console's AQL (display issue) * fixed issue #3549: server reading ENGINE config file fails on common standard newline character * UI: fixed error notifications for collection modifications * several improvements for the truncate operation on collections: * the timeout for the truncate operation was increased in cluster mode in order to prevent too frequent "could not truncate collection" errors * after a truncate operation, collections in MMFiles still used disk space. to reclaim disk space used by truncated collection, the truncate actions in the web interface and from the ArangoShell now issue an extra WAL flush command (in cluster mode, this command is also propagated to all servers). the WAL flush allows all servers to write out any pending operations into the datafiles of the truncated collection. afterwards, a final journal rotate command is sent, which enables the compaction to entirely remove all datafiles and journals for the truncated collection, so that all disk space can be reclaimed * for MMFiles a special method will be called after a truncate operation so that all indexes of the collection can free most of their memory. previously some indexes (hash and skiplist indexes) partially kept already allocated memory in order to avoid future memory allocations * after a truncate operation in the RocksDB engine, an additional compaction will be triggered for the truncated collection. this compaction removes all deletions from the key space so that follow-up scans over the collection's key range do not have to filter out lots of already-removed values These changes make truncate operations potentially more time-consuming than before, but allow for memory/disk space savings afterwards. * enable JEMalloc background threads for purging and returning unused memory back to the operating system (Linux only) JEMalloc will create its background threads on demand. The number of background threads is capped by the number of CPUs or active arenas. The background threads run periodically and purge unused memory pages, allowing memory to be returned to the operating system. This change will make the arangod process create several additional threads. It is accompanied by an increased `TasksMax` value in the systemd service configuration file for the arangodb3 service. * upgraded bundled V8 engine to bugfix version v5.7.492.77 this upgrade fixes a memory leak in upstream V8 described in https://bugs.chromium.org/p/v8/issues/detail?id=5945 that will result in memory chunks only getting uncommitted but not unmapped v3.2.6 (2017-10-26) ------------------- * UI: fixed event cleanup in cluster shards view * UI: reduced cluster dashboard api calls * fixed a permission problem that prevented collection contents to be displayed in the web interface * removed posix_fadvise call from RocksDB's PosixSequentialFile::Read(). This is consistent with Facebook PR 2573 (#3505) this fix should improve the performance of the replication with the RocksDB storage engine * allow changing of collection replication factor for existing collections * UI: replicationFactor of a collection is now changeable in a cluster environment * several fixes for the cluster agency * fixed undefined behavior in the RocksDB-based geo index * fixed Foxxmaster failover * purging or removing the Debian/Ubuntu arangodb3 packages now properly stops the arangod instance before actuallying purging or removing v3.2.5 (2017-10-16) ------------------- * general-graph module and _api/gharial now accept cluster options for collection creation. It is now possible to set replicationFactor and numberOfShards for all collections created via this graph object. So adding a new collection will not result in a singleShard and no replication anymore. * fixed issue #3408: Hard crash in query for pagination * minimum number of V8 contexts in console mode must be 2, not 1. this is required to ensure the console gets one dedicated V8 context and all other operations have at least one extra context. This requirement was not enforced anymore. * fixed issue #3395: AQL: cannot instantiate CollectBlock with undetermined aggregation method * UI: fixed wrong user attribute name validation, issue #3228 * fix potential overflow in CRC marker check when a corrupted CRC marker is found at the very beginning of an MMFiles datafile * UI: fixed unresponsive events in cluster shards view * Add statistics about the V8 context counts and number of available/active/busy threads we expose through the server statistics interface. v3.2.4 (2017-09-26) ------------------- * UI: no default index selected during index creation * UI: added replicationFactor option during SmartGraph creation * make the MMFiles compactor perform less writes during normal compaction operation This partially fixes issue #3144 * make the MMFiles compactor configurable The following options have been added: * `--compaction.db-sleep-time`: sleep interval between two compaction runs (in s) * `--compaction.min-interval"`: minimum sleep time between two compaction runs (in s) * `--compaction.min-small-data-file-size`: minimal filesize threshold original datafiles have to be below for a compaction * `--compaction.dead-documents-threshold`: minimum unused count of documents in a datafile * `--compaction.dead-size-threshold`: how many bytes of the source data file are allowed to be unused at most * `--compaction.dead-size-percent-threshold`: how many percent of the source datafile should be unused at least * `--compaction.max-files`: Maximum number of files to merge to one file * `--compaction.max-result-file-size`: how large may the compaction result file become (in bytes) * `--compaction.max-file-size-factor`: how large the resulting file may be in comparison to the collection's `--database.maximal-journal-size' setting` * fix downwards-incompatibility in /_api/explain REST handler * fix Windows implementation for fs.getTempPath() to also create a sub-directory as we do on linux * fixed a multi-threading issue in cluster-internal communication * performance improvements for traversals and edge lookups * removed internal memory zone handling code. the memory zones were a leftover from the early ArangoDB days and did not provide any value in the current implementation. * (Enterprise only) added `skipInaccessibleCollections` option for AQL queries: if set, AQL queries (especially graph traversals) will treat collections to which a user has no access rights to as if these collections were empty. * adjusted scheduler thread handling to start and stop less threads in normal operations * leader-follower replication catchup code has been rewritten in C++ * early stage AQL optimization now also uses the C++ implementations of AQL functions if present. Previously it always referred to the JavaScript implementations and ignored the C++ implementations. This change gives more flexibility to the AQL optimizer. * ArangoDB tty log output is now colored for log messages with levels FATAL, ERR and WARN. * changed the return values of AQL functions `REGEX_TEST` and `REGEX_REPLACE` to `null` when the input regex is invalid. Previous versions of ArangoDB partly returned `false` for invalid regexes and partly `null`. * added `--log.role` option for arangod When set to `true`, this option will make the ArangoDB logger print a single character with the server's role into each logged message. The roles are: - U: undefined/unclear (used at startup) - S: single server - C: coordinator - P: primary - A: agent The default value for this option is `false`, so no roles will be logged. v3.2.3 (2017-09-07) ------------------- * fixed issue #3106: orphan collections could not be registered in general-graph module * fixed wrong selection of the database inside the internal cluster js api * added startup option `--server.check-max-memory-mappings` to make arangod check the number of memory mappings currently used by the process and compare it with the maximum number of allowed mappings as determined by /proc/sys/vm/max_map_count The default value is `true`, so the checks will be performed. When the current number of mappings exceeds 90% of the maximum number of mappings, the creation of further V8 contexts will be deferred. Note that this option is effective on Linux systems only. * arangoimp now has a `--remove-attribute` option * added V8 context lifetime control options `--javascript.v8-contexts-max-invocations` and `--javascript.v8-contexts-max-age` These options allow specifying after how many invocations a used V8 context is disposed, or after what time a V8 context is disposed automatically after its creation. If either of the two thresholds is reached, an idl V8 context will be disposed. The default value of `--javascript.v8-contexts-max-invocations` is 0, meaning that the maximum number of invocations per context is unlimited. The default value for `--javascript.v8-contexts-max-age` is 60 seconds. * fixed wrong UI cluster health information * fixed issue #3070: Add index in _jobs collection * fixed issue #3125: HTTP Foxx API JSON parsing * fixed issue #3120: Foxx queue: job isn't running when server.authentication = true * fixed supervision failure detection and handling, which happened with simultaneous agency leadership change v3.2.2 (2017-08-23) ------------------- * make "Rebalance shards" button work in selected database only, and not make it rebalance the shards of all databases * fixed issue #2847: adjust the response of the DELETE `/_api/users/database/*` calls * fixed issue #3075: Error when upgrading arangoDB on linux ubuntu 16.04 * fixed a buffer overrun in linenoise console input library for long input strings * increase size of the linenoise input buffer to 8 KB * abort compilation if the detected GCC or CLANG isn't in the range of compilers we support * fixed spurious cluster hangups by always sending AQL-query related requests to the correct servers, even after failover or when a follower drops The problem with the previous shard-based approach was that responsibilities for shards may change from one server to another at runtime, after the query was already instanciated. The coordinator and other parts of the query then sent further requests for the query to the servers now responsible for the shards. However, an AQL query must send all further requests to the same servers on which the query was originally instanciated, even in case of failover. Otherwise this would potentially send requests to servers that do not know about the query, and would also send query shutdown requests to the wrong servers, leading to abandoned queries piling up and using resources until they automatically time out. * fixed issue with RocksDB engine acquiring the collection count values too early, leading to the collection count values potentially being slightly off even in exclusive transactions (for which the exclusive access should provide an always-correct count value) * fixed some issues in leader-follower catch-up code, specifically for the RocksDB engine * make V8 log fatal errors to syslog before it terminates the process. This change is effective on Linux only. * fixed issue with MMFiles engine creating superfluous collection journals on shutdown * fixed issue #3067: Upgrade from 3.2 to 3.2.1 reset autoincrement keys * fixed issue #3044: ArangoDB server shutdown unexpectedly * fixed issue #3039: Incorrect filter interpretation * fixed issue #3037: Foxx, internal server error when I try to add a new service * improved MMFiles fulltext index document removal performance and fulltext index query performance for bigger result sets * ui: fixed a display bug within the slow and running queries view * ui: fixed a bug when success event triggers twice in a modal * ui: fixed the appearance of the documents filter * ui: graph vertex collections not restricted to 10 anymore * fixed issue #2835: UI detection of JWT token in case of server restart or upgrade * upgrade jemalloc version to 5.0.1 This fixes problems with the memory allocator returing "out of memory" when calling munmap to free memory in order to return it to the OS. It seems that calling munmap on Linux can increase the number of mappings, at least when a region is partially unmapped. This can lead to the process exceeding its maximum number of mappings, and munmap and future calls to mmap returning errors. jemalloc version 5.0.1 does not have the `--enable-munmap` configure option anymore, so the problem is avoided. To return memory to the OS eventually, jemalloc 5's background purge threads are used on Linux. * fixed issue #2978: log something more obvious when you log a Buffer * fixed issue #2982: AQL parse error? * fixed issue #3125: HTTP Foxx API Json parsing v3.2.1 (2017-08-09) ------------------- * added C++ implementations for AQL functions `LEFT()`, `RIGHT()` and `TRIM()` * fixed docs for issue #2968: Collection _key autoincrement value increases on error * fixed issue #3011: Optimizer rule reduce-extraction-to-projection breaks queries * Now allowing to restore users in a sharded environment as well It is still not possible to restore collections that are sharded differently than by _key. * fixed an issue with restoring of system collections and user rights. It was not possible to restore users into an authenticated server. * fixed issue #2977: Documentation for db._createDatabase is wrong * ui: added bind parameters to slow query history view * fixed issue #1751: Slow Query API should provide bind parameters, webui should display them * ui: fixed a bug when moving multiple documents was not possible * fixed docs for issue #2968: Collection _key autoincrement value increases on error * AQL CHAR_LENGTH(null) returns now 0. Since AQL TO_STRING(null) is '' (string of length 0) * ui: now supports single js file upload for Foxx services in addition to zip files * fixed a multi-threading issue in the agency when callElection was called while the Supervision was calling updateSnapshot * added startup option `--query.tracking-with-bindvars` This option controls whether the list of currently running queries and the list of slow queries should contain the bind variables used in the queries or not. The option can be changed at runtime using the commands // enables tracking of bind variables // set to false to turn tracking of bind variables off var value = true; require("@arangodb/aql/queries").properties({ trackBindVars: value }); * index selectivity estimates are now available in the cluster as well * fixed issue #2943: loadIndexesIntoMemory not returning the same structure as the rest of the collection APIs * fixed issue #2949: ArangoError 1208: illegal name * fixed issue #2874: Collection properties do not return `isVolatile` attribute * potential fix for issue #2939: Segmentation fault when starting coordinator node * fixed issue #2810: out of memory error when running UPDATE/REPLACE on medium-size collection * fix potential deadlock errors in collector thread * disallow the usage of volatile collections in the RocksDB engine by throwing an error when a collection is created with attribute `isVolatile` set to `true`. Volatile collections are unsupported by the RocksDB engine, so creating them should not succeed and silently create a non-volatile collection * prevent V8 from issuing SIGILL instructions when it runs out of memory Now arangod will attempt to log a FATAL error into its logfile in case V8 runs out of memory. In case V8 runs out of memory, it will still terminate the entire process. But at least there should be something in the ArangoDB logs indicating what the problem was. Apart from that, the arangod process should now be exited with SIGABRT rather than SIGILL as it shouldn't return into the V8 code that aborted the process with `__builtin_trap`. this potentially fixes issue #2920: DBServer crashing automatically post upgrade to 3.2 * Foxx queues and tasks now ensure that the scripts in them run with the same permissions as the Foxx code who started the task / queue * fixed issue #2928: Offset problems * fixed issue #2876: wrong skiplist index usage in edge collection * fixed issue #2868: cname missing from logger-follow results in rocksdb * fixed issue #2889: Traversal query using incorrect collection id * fixed issue #2884: AQL traversal uniqueness constraints "propagating" to other traversals? Weird results * arangoexport: added `--query` option for passing an AQL query to export the result * fixed issue #2879: No result when querying for the last record of a query * ui: allows now to edit default access level for collections in database _system for all users except the root user. * The _users collection is no longer accessible outside the arngod process, _queues is always read-only * added new option "--rocksdb.max-background-jobs" * removed options "--rocksdb.max-background-compactions", "--rocksdb.base-background-compactions" and "--rocksdb.max-background-flushes" * option "--rocksdb.compaction-read-ahead-size" now defaults to 2MB * change Windows build so that RocksDB doesn't enforce AVX optimizations by default This fixes startup crashes on servers that do not have AVX CPU extensions * speed up RocksDB secondary index creation and dropping * removed RocksDB note in Geo index docs v3.2.0 (2017-07-20) ------------------- * fixed UI issues * fixed multi-threading issues in Pregel * fixed Foxx resilience * added command-line option `--javascript.allow-admin-execute` This option can be used to control whether user-defined JavaScript code is allowed to be executed on server by sending via HTTP to the API endpoint `/_admin/execute` with an authenticated user account. The default value is `false`, which disables the execution of user-defined code. This is also the recommended setting for production. In test environments, it may be convenient to turn the option on in order to send arbitrary setup or teardown commands for execution on the server. v3.2.beta6 (2017-07-18) ----------------------- * various bugfixes v3.2.beta5 (2017-07-16) ----------------------- * numerous bugfixes v3.2.beta4 (2017-07-04) ----------------------- * ui: fixed document view _from and _to linking issue for special characters * added function `db._parse(query)` for parsing an AQL query and returning information about it * fixed one medium priority and two low priority security user interface issues found by owasp zap. * ui: added index deduplicate options * ui: fixed renaming of collections for the rocksdb storage engine * documentation and js fixes for secondaries * RocksDB storage format was changed, users of the previous beta/alpha versions must delete the database directory and re-import their data * enabled permissions on database and collection level * added and changed some user related REST APIs * added `PUT /_api/user/{user}/database/{database}/{collection}` to change collection permission * added `GET /_api/user/{user}/database/{database}/{collection}` * added optional `full` parameter to the `GET /_api/user/{user}/database/` REST call * added user functions in the arangoshell `@arangodb/users` module * added `grantCollection` and `revokeCollection` functions * added `permission(user, database, collection)` to retrieve collection specific rights * added "deduplicate" attribute for array indexes, which controls whether inserting duplicate index values from the same document into a unique array index will lead to an error or not: // with deduplicate = true, which is the default value: db._create("test"); db.test.ensureIndex({ type: "hash", fields: ["tags[*]"], deduplicate: true }); db.test.insert({ tags: ["a", "b"] }); db.test.insert({ tags: ["c", "d", "c"] }); // will work, because deduplicate = true db.test.insert({ tags: ["a"] }); // will fail // with deduplicate = false db._create("test"); db.test.ensureIndex({ type: "hash", fields: ["tags[*]"], deduplicate: false }); db.test.insert({ tags: ["a", "b"] }); db.test.insert({ tags: ["c", "d", "c"] }); // will not work, because deduplicate = false db.test.insert({ tags: ["a"] }); // will fail The "deduplicate" attribute is now also accepted by the index creation HTTP API endpoint POST /_api/index and is returned by GET /_api/index. * added optimizer rule "remove-filters-covered-by-traversal" * Debian/Ubuntu installer: make messages about future package upgrades more clear * fix a hangup in VST The problem happened when the two first chunks of a VST message arrived together on a connection that was newly switched to VST. * fix deletion of outdated WAL files in RocksDB engine * make use of selectivity estimates in hash, skiplist and persistent indexes in RocksDB engine * changed VM overcommit recommendation for user-friendliness * fix a shutdown bug in the cluster: a destroyed query could still be active * do not terminate the entire server process if a temp file cannot be created (Windows only) * fix log output in the front-end, it stopped in case of too many messages v3.2.beta3 (2017-06-27) ----------------------- * numerous bugfixes v3.2.beta2 (2017-06-20) ----------------------- * potentially fixed issue #2559: Duplicate _key generated on insertion * fix invalid results (too many) when a skipping LIMIT was used for a traversal. `LIMIT x` or `LIMIT 0, x` were not affected, but `LIMIT s, x` may have returned too many results * fix races in SSL communication code * fix invalid locking in JWT authentication cache, which could have crashed the server * fix invalid first group results for sorted AQL COLLECT when LIMIT was used * fix potential race, which could make arangod hang on startup * removed `exception` field from transaction error result; users should throw explicit `Error` instances to return custom exceptions (addresses issue #2561) * fixed issue #2613: Reduce log level when Foxx manager tries to self heal missing database * add a read only mode for users and collection level authorization * removed `exception` field from transaction error result; users should throw explicit `Error` instances to return custom exceptions (addresses issue #2561) * fixed issue #2677: Foxx disabling development mode creates non-deterministic service bundle * fixed issue #2684: Legacy service UI not working v3.2.beta1 (2017-06-12) ----------------------- * provide more context for index errors (addresses issue #342) * arangod now validates several OS/environment settings on startup and warns if the settings are non-ideal. Most of the checks are executed on Linux systems only. * fixed issue #2515: The replace-or-with-in optimization rule might prevent use of indexes * added `REGEX_REPLACE` AQL function * the RocksDB storage format was changed, users of the previous alpha versions must delete the database directory and re-import their data * added server startup option `--query.fail-on-warning` setting this option to `true` will abort any AQL query with an exception if it causes a warning at runtime. The value can be overridden per query by setting the `failOnWarning` attribute in a query's options. * added --rocksdb.num-uncompressed-levels to adjust number of non-compressed levels * added checks for memory managment and warn (i. e. if hugepages are enabled) * set default SSL cipher suite string to "HIGH:!EXPORT:!aNULL@STRENGTH" * fixed issue #2469: Authentication = true does not protect foxx-routes * fixed issue #2459: compile success but can not run with rocksdb * `--server.maximal-queue-size` is now an absolute maximum. If the queue is full, then 503 is returned. Setting it to 0 means "no limit". * (Enterprise only) added authentication against an LDAP server * fixed issue #2083: Foxx services aren't distributed to all coordinators * fixed issue #2384: new coordinators don't pick up existing Foxx services * fixed issue #2408: Foxx service validation causes unintended side-effects * extended HTTP API with routes for managing Foxx services * added distinction between hasUser and authorized within Foxx (cluster internal requests are authorized requests but don't have a user) * arangoimp now has a `--threads` option to enable parallel imports of data * PR #2514: Foxx services that can't be fixed by self-healing now serve a 503 error * added `time` function to `@arangodb` module v3.2.alpha4 (2017-04-25) ------------------------ * fixed issue #2450: Bad optimization plan on simple query * fixed issue #2448: ArangoDB Web UI takes no action when Delete button is clicked * fixed issue #2442: Frontend shows already deleted databases during login * added 'x-content-type-options: nosniff' to avoid MSIE bug * set default value for `--ssl.protocol` from TLSv1 to TLSv1.2. * AQL breaking change in cluster: The SHORTEST_PATH statement using edge-collection names instead of a graph name now requires to explicitly name the vertex-collection names within the AQL query in the cluster. It can be done by adding `WITH ` at the beginning of the query. Example: ``` FOR v,e IN OUTBOUND SHORTEST_PATH @start TO @target edges [...] ``` Now has to be: ``` WITH vertices FOR v,e IN OUTBOUND SHORTEST_PATH @start TO @target edges [...] ``` This change is due to avoid dead-lock sitations in clustered case. An error stating the above is included. * add implicit use of geo indexes when using SORT/FILTER in AQL, without the need to use the special-purpose geo AQL functions `NEAR` or `WITHIN`. the special purpose `NEAR` AQL function can now be substituted with the following AQL (provided there is a geo index present on the `doc.latitude` and `doc.longitude` attributes): FOR doc in geoSort SORT DISTANCE(doc.latitude, doc.longitude, 0, 0) LIMIT 5 RETURN doc `WITHIN` can be substituted with the following AQL: FOR doc in geoFilter FILTER DISTANCE(doc.latitude, doc.longitude, 0, 0) < 2000 RETURN doc Compared to using the special purpose AQL functions this approach has the advantage that it is more composable, and will also honor any `LIMIT` values used in the AQL query. * potential fix for shutdown hangs on OSX * added KB, MB, GB prefix for integer parameters, % for integer parameters with a base value * added JEMALLOC 4.5.0 * added `--vm.resident-limit` and `--vm.path` for file-backed memory mapping after reaching a configurable maximum RAM size * try recommended limit for file descriptors in case of unlimited hard limit * issue #2413: improve logging in case of lock timeout and deadlocks * added log topic attribute to /_admin/log api * removed internal build option `USE_DEV_TIMERS` Enabling this option activated some proprietary timers for only selected events in arangod. Instead better use `perf` to gather timings. v3.2.alpha3 (2017-03-22) ------------------------ * increase default collection lock timeout from 30 to 900 seconds * added function `db._engine()` for retrieval of storage engine information at server runtime There is also an HTTP REST handler at GET /_api/engine that returns engine information. * require at least cmake 3.2 for building ArangoDB * make arangod start with less V8 JavaScript contexts This speeds up the server start (a little bit) and makes it use less memory. Whenever a V8 context is needed by a Foxx action or some other operation and there is no usable V8 context, a new one will be created dynamically now. Up to `--javascript.v8-contexts` V8 contexts will be created, so this option will change its meaning. Previously as many V8 contexts as specified by this option were created at server start, and the number of V8 contexts did not change at runtime. Now up to this number of V8 contexts will be in use at the same time, but the actual number of V8 contexts is dynamic. The garbage collector thread will automatically delete unused V8 contexts after a while. The number of spare contexts will go down to as few as configured in the new option `--javascript.v8-contexts-minimum`. Actually that many V8 contexts are also created at server start. The first few requests in new V8 contexts will take longer than in contexts that have been there already. Performance may therefore suffer a bit for the initial requests sent to ArangoDB or when there are only few but performance- critical situations in which new V8 contexts will be created. If this is a concern, it can easily be fixed by setting `--javascipt.v8-contexts-minimum` and `--javascript.v8-contexts` to a relatively high value, which will guarantee that many number of V8 contexts to be created at startup and kept around even when unused. Waiting for an unused V8 context will now also abort if no V8 context can be acquired/created after 120 seconds. * improved diagnostic messages written to logfiles by supervisor process * fixed issue #2367 * added "bindVars" to attributes of currently running and slow queries * added "jsonl" as input file type for arangoimp * upgraded version of bundled zlib library from 1.2.8 to 1.2.11 * added input file type `auto` for arangoimp so it can automatically detect the type of the input file from the filename extension * fixed variables parsing in GraphQL * added `--translate` option for arangoimp to translate attribute names from the input files to attriubte names expected by ArangoDB The `--translate` option can be specified multiple times (once per translation to be executed). The following example renames the "id" column from the input file to "_key", and the "from" column to "_from", and the "to" column to "_to": arangoimp --type csv --file data.csv --translate "id=_key" --translate "from=_from" --translate "to=_to" `--translate` works for CSV and TSV inputs only. * changed default value for `--server.max-packet-size` from 128 MB to 256 MB * fixed issue #2350 * fixed issue #2349 * fixed issue #2346 * fixed issue #2342 * change default string truncation length from 80 characters to 256 characters for `print`/`printShell` functions in ArangoShell and arangod. This will emit longer prefixes of string values before truncating them with `...`, which is helpful for debugging. * always validate incoming JSON HTTP requests for duplicate attribute names Incoming JSON data with duplicate attribute names will now be rejected as invalid. Previous versions of ArangoDB only validated the uniqueness of attribute names inside incoming JSON for some API endpoints, but not consistently for all APIs. * don't let read-only transactions block the WAL collector * allow passing own `graphql-sync` module instance to Foxx GraphQL router * arangoexport can now export to csv format * arangoimp: fixed issue #2214 * Foxx: automatically add CORS response headers * added "OPTIONS" to CORS `access-control-allow-methods` header * Foxx: Fix arangoUser sometimes not being set correctly * fixed issue #1974 v3.2.alpha2 (2017-02-20) ------------------------ * ui: fixed issue #2065 * ui: fixed a dashboard related memory issue * Internal javascript rest actions will now hide their stack traces to the client unless maintainer mode is activated. Instead they will always log to the logfile * Removed undocumented internal HTTP API: * PUT _api/edges The documented GET _api/edges and the undocumented POST _api/edges remains unmodified. * updated V8 version to 5.7.0.0 * change undocumented behavior in case of invalid revision ids in If-Match and If-None-Match headers from 400 (BAD) to 412 (PRECONDITION FAILED). * change undocumented behavior in case of invalid revision ids in JavaScript document operations from 1239 ("illegal document revision") to 1200 ("conflict"). * added data export tool, arangoexport. arangoexport can be used to export collections to json, jsonl or xml and export a graph or collections to xgmml. * fixed a race condition when closing a connection * raised default hard limit on threads for very small to 64 * fixed negative counting of http connection in UI v3.2.alpha1 (2017-02-05) ------------------------ * added figure `httpRequests` to AQL query statistics * removed revisions cache intermediate layer implementation * obsoleted startup options `--database.revision-cache-chunk-size` and `--database.revision-cache-target-size` * fix potential port number over-/underruns * added startup option `--log.shorten-filenames` for controlling whether filenames in log messages should be shortened to just the filename with the absolute path * removed IndexThreadFeature, made `--database.index-threads` option obsolete * changed index filling to make it more parallel, dispatch tasks to boost::asio * more detailed stacktraces in Foxx apps * generated Foxx services now use swagger tags v3.1.24 (XXXX-XX-XX) -------------------- * fixed one more LIMIT issue in traversals v3.1.23 (2017-06-19) -------------------- * potentially fixed issue #2559: Duplicate _key generated on insertion * fix races in SSL communication code * fix invalid results (too many) when a skipping LIMIT was used for a traversal. `LIMIT x` or `LIMIT 0, x` were not affected, but `LIMIT s, x` may have returned too many results * fix invalid first group results for sorted AQL COLLECT when LIMIT was used * fix invalid locking in JWT authentication cache, which could have crashed the server * fix undefined behavior in traverser when traversals were used inside a FOR loop v3.1.22 (2017-06-07) -------------------- * fixed issue #2505: Problem with export + report of a bug * documented changed behavior of WITH * fixed ui glitch in aardvark * avoid agency compaction bug * fixed issue #2283: disabled proxy communication internally v3.1.21 (2017-05-22) -------------------- * fixed issue #2488: AQL operator IN error when data use base64 chars * more randomness in seeding RNG v3.1.20 (2016-05-16) -------------------- * fixed incorrect sorting for distributeShardsLike * improve reliability of AgencyComm communication with Agency * fixed shard numbering bug, where ids were erouneously incremented by 1 * remove an unnecessary precondition in createCollectionCoordinator * funny fail rotation fix * fix in SimpleHttpClient for correct advancement of readBufferOffset * forward SIG_HUP in supervisor process to the server process to fix logrotaion You need to stop the remaining arangod server process manually for the upgrade to work. v3.1.19 (2017-04-28) -------------------- * Fixed a StackOverflow issue in Traversal and ShortestPath. Occured if many (>1000) input values in a row do not return any result. Fixes issue: #2445 * fixed issue #2448 * fixed issue #2442 * added 'x-content-type-options: nosniff' to avoid MSIE bug * fixed issue #2441 * fixed issue #2440 * Fixed a StackOverflow issue in Traversal and ShortestPath. Occured if many (>1000) input values in a row do not return any result. Fixes issue: #2445 * fix occasional hanging shutdowns on OS X v3.1.18 (2017-04-18) -------------------- * fixed error in continuous synchronization of collections * fixed spurious hangs on server shutdown * better error messages during restore collection * completely overhaul supervision. More detailed tests * Fixed a dead-lock situation in cluster traversers, it could happen in rare cases if the computation on one DBServer could be completed much earlier than the other server. It could also be restricted to SmartGraphs only. * (Enterprise only) Fixed a bug in SmartGraph DepthFirstSearch. In some more complicated queries, the maxDepth limit of 1 was not considered strictly enough, causing the traverser to do unlimited depth searches. * fixed issue #2415 * fixed issue #2422 * fixed issue #1974 v3.1.17 (2017-04-04) -------------------- * (Enterprise only) fixed a bug where replicationFactor was not correctly forwarded in SmartGraph creation. * fixed issue #2404 * fixed issue #2397 * ui - fixed smart graph option not appearing * fixed issue #2389 * fixed issue #2400 v3.1.16 (2017-03-27) -------------------- * fixed issue #2392 * try to raise file descriptors to at least 8192, warn otherwise * ui - aql editor improvements + updated ace editor version (memory leak) * fixed lost HTTP requests * ui - fixed some event issues * avoid name resolution when given connection string is a valid ip address * helps with issue #1842, bug in COLLECT statement in connection with LIMIT. * fix locking bug in cluster traversals * increase lock timeout defaults * increase various cluster timeouts * limit default target size for revision cache to 1GB, which is better for tight RAM situations (used to be 40% of (totalRAM - 1GB), use --database.revision-cache-target-size to get back the old behavior * fixed a bug with restarted servers indicating status as "STARTUP" rather that "SERVING" in Nodes UI. v3.1.15 (2017-03-20) -------------------- * add logrotate configuration as requested in #2355 * fixed issue #2376 * ui - changed document api due a chrome bug * ui - fixed a submenu bug * added endpoint /_api/cluster/endpoints in cluster case to get all coordinator endpoints * fix documentation of /_api/endpoint, declaring this API obsolete. * Foxx response objects now have a `type` method for manipulating the content-type header * Foxx tests now support `xunit` and `tap` reporters v3.1.14 (2017-03-13) -------------------- * ui - added feature request (multiple start nodes within graph viewer) #2317 * added missing locks to authentication cache methods * ui - added feature request (multiple start nodes within graph viewer) #2317 * ui - fixed wrong merge of statistics information from different coordinators * ui - fixed issue #2316 * ui - fixed wrong protocol usage within encrypted environment * fixed compile error on Mac Yosemite * minor UI fixes v3.1.13 (2017-03-06) -------------------- * fixed variables parsing in GraphQL * fixed issue #2214 * fixed issue #2342 * changed thread handling to queue only user requests on coordinator * use exponential backoff when waiting for collection locks * repair short name server lookup in cluster in the case of a removed server v3.1.12 (2017-02-28) -------------------- * disable shell color escape sequences on Windows * fixed issue #2326 * fixed issue #2320 * fixed issue #2315 * fixed a race condition when closing a connection * raised default hard limit on threads for very small to 64 * fixed negative counting of http connection in UI * fixed a race when renaming collections * fixed a race when dropping databases v3.1.11 (2017-02-17) -------------------- * fixed a race between connection closing and sending out last chunks of data to clients when the "Connection: close" HTTP header was set in requests * ui: optimized smart graph creation usability * ui: fixed #2308 * fixed a race in async task cancellation via `require("@arangodb/tasks").unregisterTask()` * fixed spuriously hanging threads in cluster AQL that could sit idle for a few minutes * fixed potential numeric overflow for big index ids in index deletion API * fixed sort issue in cluster, occurring when one of the local sort buffers of a GatherNode was empty * reduce number of HTTP requests made for certain kinds of join queries in cluster, leading to speedup of some join queries * supervision deals with demised coordinators correctly again * implement a timeout in TraverserEngineRegistry * agent communication reduced in large batches of append entries RPCs * inception no longer estimates RAFT timings * compaction in agents has been moved to a separate thread * replicated logs hold local timestamps * supervision jobs failed leader and failed follower revisited for function in precarious stability situations * fixed bug in random number generator for 64bit int v3.1.10 (2017-02-02) -------------------- * updated versions of bundled node modules: - joi: from 8.4.2 to 9.2.0 - joi-to-json-schema: from 2.2.0 to 2.3.0 - sinon: from 1.17.4 to 1.17.6 - lodash: from 4.13.1 to 4.16.6 * added shortcut for AQL ternary operator instead of `condition ? true-part : false-part` it is now possible to also use a shortcut variant `condition ? : false-part`, e.g. FOR doc IN docs RETURN doc.value ?: 'not present' instead of FOR doc IN docs RETURN doc.value ? doc.value : 'not present' * fixed wrong sorting order in cluster, if an index was used to sort with many shards. * added --replication-factor, --number-of-shards and --wait-for-sync to arangobench * turn on UTF-8 string validation for VelocyPack values received via VST connections * fixed issue #2257 * upgraded Boost version to 1.62.0 * added optional detail flag for db..count() setting the flag to `true` will make the count operation returned the per-shard counts for the collection: db._create("test", { numberOfShards: 10 }); for (i = 0; i < 1000; ++i) { db.test.insert({value: i}); } db.test.count(true); { "s100058" : 99, "s100057" : 103, "s100056" : 100, "s100050" : 94, "s100055" : 90, "s100054" : 122, "s100051" : 109, "s100059" : 99, "s100053" : 95, "s100052" : 89 } * added optional memory limit for AQL queries: db._query("FOR i IN 1..100000 SORT i RETURN i", {}, { options: { memoryLimit: 100000 } }); This option limits the default maximum amount of memory (in bytes) that a single AQL query can use. When a single AQL query reaches the specified limit value, the query will be aborted with a *resource limit exceeded* exception. In a cluster, the memory accounting is done per shard, so the limit value is effectively a memory limit per query per shard. The global limit value can be overriden per query by setting the *memoryLimit* option value for individual queries when running an AQL query. * added server startup option `--query.memory-limit` * added convenience function to create vertex-centric indexes. Usage: `db.collection.ensureVertexCentricIndex("label", {type: "hash", direction: "outbound"})` That will create an index that can be used on OUTBOUND with filtering on the edge attribute `label`. * change default log output for tools to stdout (instead of stderr) * added option -D to define a configuration file environment key=value * changed encoding behavior for URLs encoded in the C++ code of ArangoDB: previously the special characters `-`, `_`, `~` and `.` were returned as-is after URL-encoding, now `.` will be encoded to be `%2e`. This also changes the behavior of how incoming URIs are processed: previously occurrences of `..` in incoming request URIs were collapsed (e.g. `a/../b/` was collapsed to a plain `b/`). Now `..` in incoming request URIs are not collapsed. * Foxx request URL suffix is no longer unescaped * @arangodb/request option json now defaults to `true` if the response body is not empty and encoding is not explicitly set to `null` (binary). The option can still be set to `false` to avoid unnecessary attempts at parsing the response as JSON. * Foxx configuration values for unknown options will be discarded when saving the configuration in production mode using the web interface * module.context.dependencies is now immutable * process.stdout.isTTY now returns `true` in arangosh and when running arangod with the `--console` flag * add support for Swagger tags in Foxx v3.1.9 (XXXX-XX-XX) ------------------- * macos CLI package: store databases and apps in the users home directory * ui: fixed re-login issue within a non system db, when tab was closed * fixed a race in the VelocyStream Commtask implementation * fixed issue #2256 v3.1.8 (2017-01-09) ------------------- * add Windows silent installer * add handling of debug symbols during Linux & windows release builds. * fixed issue #2181 * fixed issue #2248: reduce V8 max old space size from 3 GB to 1 GB on 32 bit systems * upgraded Boost version to 1.62.0 * fixed issue #2238 * fixed issue #2234 * agents announce new endpoints in inception phase to leader * agency leadership accepts updatet endpoints to given uuid * unified endpoints replace localhost with 127.0.0.1 * fix several problems within an authenticated cluster v3.1.7 (2016-12-29) ------------------- * fixed one too many elections in RAFT * new agency comm backported from devel v3.1.6 (2016-12-20) ------------------- * fixed issue #2227 * fixed issue #2220 * agency constituent/agent bug fixes in race conditions picking up leadership * supervision does not need waking up anymore as it is running regardless * agents challenge their leadership more rigorously v3.1.5 (2016-12-16) ------------------- * lowered default value of `--database.revision-cache-target-size` from 75% of RAM to less than 40% of RAM * fixed issue #2218 * fixed issue #2217 * Foxx router.get/post/etc handler argument can no longer accidentally omitted * fixed issue #2223 v3.1.4 (2016-12-08) ------------------- * fixed issue #2211 * fixed issue #2204 * at cluster start, coordinators wait until at least one DBserver is there, and either at least two DBservers are there or 15s have passed, before they initiate the bootstrap of system collections. * more robust agency startup from devel * supervision's AddFollower adds many followers at once * supervision has new FailedFollower job * agency's Node has new method getArray * agency RAFT timing estimates more conservative in waitForSync scenario * agency RAFT timing estimates capped at maximum 2.0/10.0 for low/high v3.1.3 (2016-12-02) ------------------- * fix a traversal bug when using skiplist indexes: if we have a skiplist of ["a", "unused", "_from"] and a traversal like: FOR v,e,p IN OUTBOUND @start @@edges FILTER p.edges[0].a == 'foo' RETURN v And the above index applied on "a" is considered better than EdgeIndex, than the executor got into undefined behavior. * fix endless loop when trying to create a collection with replicationFactor: -1 v3.1.2 (2016-11-24) ------------------- * added support for descriptions field in Foxx dependencies * (Enterprise only) fixed a bug in the statistic report for SmartGraph traversals. Now they state correctly how many documents were fetched from the index and how many have been filtered. * Prevent uniform shard distribution when replicationFactor == numServers v3.1.1 (2016-11-15) ------------------- * fixed issue #2176 * fixed issue #2168 * display index usage of traversals in AQL explainer output (previously missing) * fixed issue #2163 * preserve last-used HLC value across server starts * allow more control over handling of pre-3.1 _rev values this changes the server startup option `--database.check-30-revisions` from a boolean (true/false) parameter to a string parameter with the following possible values: - "fail": will validate _rev values of 3.0 collections on collection loading and throw an exception when invalid _rev values are found. in this case collections with invalid _rev values are marked as corrupted and cannot be used in the ArangoDB 3.1 instance. the fix procedure for such collections is to export the collections from 3.0 database with arangodump and restore them in 3.1 with arangorestore. collections that do not contain invalid _rev values are marked as ok and will not be re-checked on following loads. collections that contain invalid _rev values will be re-checked on following loads. - "true": will validate _rev values of 3.0 collections on collection loading and print a warning when invalid _rev values are found. in this case collections with invalid _rev values can be used in the ArangoDB 3.1 instance. however, subsequent operations on documents with invalid _rev values may silently fail or fail with explicit errors. the fix procedure for such collections is to export the collections from 3.0 database with arangodump and restore them in 3.1 with arangorestore. collections that do not contain invalid _rev values are marked as ok and will not be re-checked on following loads. collections that contain invalid _rev values will be re-checked on following loads. - "false": will not validate _rev values on collection loading and not print warnings. no hint is given when invalid _rev values are found. subsequent operations on documents with invalid _rev values may silently fail or fail with explicit errors. this setting does not affect whether collections are re-checked later. collections will be re-checked on following loads if `--database.check-30-revisions` is later set to either `true` or `fail`. The change also suppresses warnings that were printed when collections were restored using arangorestore, and the restore data contained invalid _rev values. Now these warnings are suppressed, and new HLC _rev values are generated for these documents as before. * added missing functions to AQL syntax highlighter in web interface * fixed display of `ANY` direction in traversal explainer output (direction `ANY` was shown as either `INBOUND` or `OUTBOUND`) * changed behavior of toJSON() function when serializing an object before saving it in the database if an object provides a toJSON() function, this function is still called for serializing it. the change is that the result of toJSON() is not stringified anymore, but saved as is. previous versions of ArangoDB called toJSON() and after that additionally stringified its result. This change will affect the saving of JS Buffer objects, which will now be saved as arrays of bytes instead of a comma-separated string of the Buffer's byte contents. * allow creating unique indexes on more attributes than present in shardKeys The following combinations of shardKeys and indexKeys are allowed/not allowed: shardKeys indexKeys a a ok a b not ok a a b ok a b a not ok a b b not ok a b a b ok a b a b c ok a b c a b not ok a b c a b c ok * fixed wrong version in web interface login screen (EE only) * make web interface not display an exclamation mark next to ArangoDB version number 3.1 * fixed search for arbitrary document attributes in web interface in case multiple search values were used on different attribute names. in this case, the search always produced an empty result * disallow updating `_from` and `_to` values of edges in Smart Graphs. Updating these attributes would lead to potential redistribution of edges to other shards, which must be avoided. * fixed issue #2148 * updated graphql-sync dependency to 0.6.2 * fixed issue #2156 * fixed CRC4 assembly linkage v3.1.0 (2016-10-29) ------------------- * AQL breaking change in cluster: from ArangoDB 3.1 onwards `WITH` is required for traversals in a clustered environment in order to avoid deadlocks. Note that for queries that access only a single collection or that have all collection names specified somewhere else in the query string, there is no need to use *WITH*. *WITH* is only useful when the AQL query parser cannot automatically figure out which collections are going to be used by the query. *WITH* is only useful for queries that dynamically access collections, e.g. via traversals, shortest path operations or the *DOCUMENT()* function. more info can be found [here](https://github.com/arangodb/arangodb/blob/devel/Documentation/Books/AQL/Operations/With.md) * added AQL function `DISTANCE` to calculate the distance between two arbitrary coordinates (haversine formula) * fixed issue #2110 * added Auto-aptation of RAFT timings as calculations only v3.1.rc2 (2016-10-10) --------------------- * second release candidate v3.1.rc1 (2016-09-30) --------------------- * first release candidate v3.1.alpha2 (2016-09-01) ------------------------ * added module.context.createDocumentationRouter to replace module.context.apiDocumentation * bug in RAFT implementation of reads. dethroned leader still answered requests in isolation * ui: added new graph viewer * ui: aql-editor added tabular & graph display * ui: aql-editor improved usability * ui: aql-editor: query profiling support * fixed issue #2109 * fixed issue #2111 * fixed issue #2075 * added AQL function `DISTANCE` to calculate the distance between two arbitrary coordinates (haversine formula) * rewrote scheduler and dispatcher based on boost::asio parameters changed: `--scheduler.threads` and `--server.threads` are now merged into a single one: `--server.threads` hidden `--server.extra-threads` has been removed hidden `--server.aql-threads` has been removed hidden `--server.backend` has been removed hidden `--server.show-backends` has been removed hidden `--server.thread-affinity` has been removed * fixed issue #2086 * fixed issue #2079 * fixed issue #2071 make the AQL query optimizer inject filter condition expressions referred to by variables during filter condition aggregation. For example, in the following query FOR doc IN collection LET cond1 = (doc.value == 1) LET cond2 = (doc.value == 2) FILTER cond1 || cond2 RETURN { doc, cond1, cond2 } the optimizer will now inject the conditions for `cond1` and `cond2` into the filter condition `cond1 || cond2`, expanding it to `(doc.value == 1) || (doc.value == 2)` and making these conditions available for index searching. Note that the optimizer previously already injected some conditions into other conditions, but only if the variable that defined the condition was not used elsewhere. For example, the filter condition in the query FOR doc IN collection LET cond = (doc.value == 1) FILTER cond RETURN { doc } already got optimized before because `cond` was only used once in the query and the optimizer decided to inject it into the place where it was used. This only worked for variables that were referred to once in the query. When a variable was used multiple times, the condition was not injected as in the following query: FOR doc IN collection LET cond = (doc.value == 1) FILTER cond RETURN { doc, cond } The fix for #2070 now will enable this optimization so that the query can use an index on `doc.value` if available. * changed behavior of AQL array comparison operators for empty arrays: * `ALL` and `ANY` now always return `false` when the left-hand operand is an empty array. The behavior for non-empty arrays does not change: * `[] ALL == 1` will return `false` * `[1] ALL == 1` will return `true` * `[1, 2] ALL == 1` will return `false` * `[2, 2] ALL == 1` will return `false` * `[] ANY == 1` will return `false` * `[1] ANY == 1` will return `true` * `[1, 2] ANY == 1` will return `true` * `[2, 2] ANY == 1` will return `false` * `NONE` now always returns `true` when the left-hand operand is an empty array. The behavior for non-empty arrays does not change: * `[] NONE == 1` will return `true` * `[1] NONE == 1` will return `false` * `[1, 2] NONE == 1` will return `false` * `[2, 2] NONE == 1` will return `true` * added experimental AQL functions `JSON_STRINGIFY` and `JSON_PARSE` * added experimental support for incoming gzip-compressed requests * added HTTP REST APIs for online log level adjustments: - GET `/_admin/log/level` returns the current log level settings - PUT `/_admin/log/level` modifies the current log level settings * PATCH /_api/gharial/{graph-name}/vertex/{collection-name}/{vertex-key} - changed default value for keepNull to true * PATCH /_api/gharial/{graph-name}/edge/{collection-name}/{edge-key} - changed default value for keepNull to true * renamed `maximalSize` attribute in parameter.json files to `journalSize` The `maximalSize` attribute will still be picked up from collections that have not been adjusted. Responses from the replication API will now also use `journalSize` instead of `maximalSize`. * added `--cluster.system-replication-factor` in order to adjust the replication factor for new system collections * fixed issue #2012 * added a memory expection in case V8 memory gets too low * added Optimizer Rule for other indexes in Traversals this allows AQL traversals to use other indexes than the edge index. So traversals with filters on edges can now make use of more specific indexes, e.g. FOR v, e, p IN 2 OUTBOUND @start @@edge FILTER p.edges[0].foo == "bar" will prefer a Hash Index on [_from, foo] above the EdgeIndex. * fixed epoch computation in hybrid logical clock * fixed thread affinity * replaced require("internal").db by require("@arangodb").db * added option `--skip-lines` for arangoimp this allows skipping the first few lines from the import file in case the CSV or TSV import are used * fixed periodic jobs: there should be only one instance running - even if it runs longer than the period * improved performance of primary index and edge index lookups * optimizations for AQL `[*]` operator in case no filter, no projection and no offset/limit are used * added AQL function `OUTERSECTION` to return the symmetric difference of its input arguments * Foxx manifests of installed services are now saved to disk with indentation * Foxx tests and scripts in development mode should now always respect updated files instead of loading stale modules * When disabling Foxx development mode the setup script is now re-run * Foxx now provides an easy way to directly serve GraphQL requests using the `@arangodb/foxx/graphql` module and the bundled `graphql-sync` dependency * Foxx OAuth2 module now correctly passes the `access_token` to the OAuth2 server * added iconv-lite and timezone modules * web interface now allows installing GitHub and zip services in legacy mode * added module.context.createDocumentationRouter to replace module.context.apiDocumentation * bug in RAFT implementation of reads. dethroned leader still answered requests in isolation * all lambdas in ClusterInfo might have been left with dangling references. * Agency bug fix for handling of empty json objects as values. * Foxx tests no longer support the Mocha QUnit interface as this resulted in weird inconsistencies in the BDD and TDD interfaces. This fixes the TDD interface as well as out-of-sequence problems when using the BDD before/after functions. * updated bundled JavaScript modules to latest versions; joi has been updated from 8.4 to 9.2 (see [joi 9.0.0 release notes](https://github.com/hapijs/joi/issues/920) for information on breaking changes and new features) * fixed issue #2139 * updated graphql-sync dependency to 0.6.2 * fixed issue #2156 v3.0.13 (XXXX-XX-XX) -------------------- * fixed issue #2315 * fixed issue #2210 v3.0.12 (2016-11-23) -------------------- * fixed issue #2176 * fixed issue #2168 * fixed issues #2149, #2159 * fixed error reporting for issue #2158 * fixed assembly linkage bug in CRC4 module * added support for descriptions field in Foxx dependencies v3.0.11 (2016-11-08) -------------------- * fixed issue #2140: supervisor dies instead of respawning child * fixed issue #2131: use shard key value entered by user in web interface * fixed issue #2129: cannot kill a long-run query * fixed issue #2110 * fixed issue #2081 * fixed issue #2038 * changes to Foxx service configuration or dependencies should now be stored correctly when options are cleared or omitted * Foxx tests no longer support the Mocha QUnit interface as this resulted in weird inconsistencies in the BDD and TDD interfaces. This fixes the TDD interface as well as out-of-sequence problems when using the BDD before/after functions. * fixed issue #2148 v3.0.10 (2016-09-26) -------------------- * fixed issue #2072 * fixed issue #2070 * fixed slow cluster starup issues. supervision will demonstrate more patience with db servers v3.0.9 (2016-09-21) ------------------- * fixed issue #2064 * fixed issue #2060 * speed up `collection.any()` and skiplist index creation * fixed multiple issues where ClusterInfo bug hung agency in limbo timeouting on multiple collection and database callbacks v3.0.8 (2016-09-14) ------------------- * fixed issue #2052 * fixed issue #2005 * fixed issue #2039 * fixed multiple issues where ClusterInfo bug hung agency in limbo timeouting on multiple collection and database callbacks v3.0.7 (2016-09-05) ------------------- * new supervision job handles db server failure during collection creation. v3.0.6 (2016-09-02) ------------------- * fixed issue #2026 * slightly better error diagnostics for AQL query compilation and replication * fixed issue #2018 * fixed issue #2015 * fixed issue #2012 * fixed wrong default value for arangoimp's `--on-duplicate` value * fix execution of AQL traversal expressions when there are multiple conditions that refer to variables set outside the traversal * properly return HTTP 503 in JS actions when backend is gone * supervision creates new key in agency for failed servers * new shards will not be allocated on failed or cleaned servers v3.0.5 (2016-08-18) ------------------- * execute AQL ternary operator via C++ if possible * fixed issue #1977 * fixed extraction of _id attribute in AQL traversal conditions * fix SSL agency endpoint * Minimum RAFT timeout was one order of magnitude to short. * Optimized RAFT RPCs from leader to followers for efficiency. * Optimized RAFT RPC handling on followers with respect to compaction. * Fixed bug in handling of duplicates and overlapping logs * Fixed bug in supervision take over after leadership change. v3.0.4 (2016-08-01) ------------------- * added missing lock for periodic jobs access * fix multiple Foxx related cluster issues * fix handling of empty AQL query strings * fixed issue in `INTERSECTION` AQL function with duplicate elements in the source arrays * fixed issue #1970 * fixed issue #1968 * fixed issue #1967 * fixed issue #1962 * fixed issue #1959 * replaced require("internal").db by require("@arangodb").db * fixed issue #1954 * fixed issue #1953 * fixed issue #1950 * fixed issue #1949 * fixed issue #1943 * fixed segfault in V8, by backporting https://bugs.chromium.org/p/v8/issues/detail?id=5033 * Foxx OAuth2 module now correctly passes the `access_token` to the OAuth2 server * fixed credentialed CORS requests properly respecting --http.trusted-origin * fixed a crash in V8Periodic task (forgotten lock) * fixed two bugs in synchronous replication (syncCollectionFinalize) v3.0.3 (2016-07-17) ------------------- * fixed issue #1942 * fixed issue #1941 * fixed array index batch insertion issues for hash indexes that caused problems when no elements remained for insertion * fixed AQL MERGE() function with External objects originating from traversals * fixed some logfile recovery errors with error message "document not found" * fixed issue #1937 * fixed issue #1936 * improved performance of arangorestore in clusters with synchronous replication * Foxx tests and scripts in development mode should now always respect updated files instead of loading stale modules * When disabling Foxx development mode the setup script is now re-run * Foxx manifests of installed services are now saved to disk with indentation v3.0.2 (2016-07-09) ------------------- * fixed assertion failure in case multiple remove operations were used in the same query * fixed upsert behavior in case upsert was used in a loop with the same document example * fixed issue #1930 * don't expose local file paths in Foxx error messages. * fixed issue #1929 * make arangodump dump the attribute `isSystem` when dumping the structure of a collection, additionally make arangorestore not fail when the attribute is missing * fixed "Could not extract custom attribute" issue when using COLLECT with MIN/MAX functions in some contexts * honor presence of persistent index for sorting * make AQL query optimizer not skip "use-indexes-rule", even if enough plans have been created already * make AQL optimizer not skip "use-indexes-rule", even if enough execution plans have been created already * fix double precision value loss in VelocyPack JSON parser * added missing SSL support for arangorestore * improved cluster import performance * fix Foxx thumbnails on DC/OS * fix Foxx configuration not being saved * fix Foxx app access from within the frontend on DC/OS * add option --default-replication-factor to arangorestore and simplify the control over the number of shards when restoring * fix a bug in the VPack -> V8 conversion if special attributes _key, _id, _rev, _from and _to had non-string values, which is allowed below the top level * fix malloc_usable_size for darwin v3.0.1 (2016-06-30) ------------------- * fixed periodic jobs: there should be only one instance running - even if it runs longer than the period * increase max. number of collections in AQL queries from 32 to 256 * fixed issue #1916: header "authorization" is required" when opening services page * fixed issue #1915: Explain: member out of range * fixed issue #1914: fix unterminated buffer * don't remove lockfile if we are the same (now stale) pid fixes docker setups (our pid will always be 1) * do not use revision id comparisons in compaction for determining whether a revision is obsolete, but marker memory addresses this ensures revision ids don't matter when compacting documents * escape Unicode characters in JSON HTTP responses this converts UTF-8 characters in HTTP responses of arangod into `\uXXXX` escape sequences. This makes the HTTP responses fit into the 7 bit ASCII character range, which speeds up HTTP response parsing for some clients, namely node.js/v8 * add write before read collections when starting a user transaction this allows specifying the same collection in both read and write mode without unintended side effects * fixed buffer overrun that occurred when building very large result sets * index lookup optimizations for primary index and edge index * fixed "collection is a nullptr" issue when starting a traversal from a transaction * enable /_api/import on coordinator servers v3.0.0 (2016-06-22) ------------------- * minor GUI fixxes * fix for replication and nonces v3.0.0-rc3 (2016-06-19) ----------------------- * renamed various Foxx errors to no longer refer to Foxx services as apps * adjusted various error messages in Foxx to be more informative * specifying "files" in a Foxx manifest to be mounted at the service root no longer results in 404s when trying to access non-file routes * undeclared path parameters in Foxx no longer break the service * trusted reverse proxy support is now handled more consistently * ArangoDB request compatibility and user are now exposed in Foxx * all bundled NPM modules have been upgraded to their latest versions v3.0.0-rc2 (2016-06-12) ----------------------- * added option `--server.max-packet-size` for client tools * renamed option `--server.ssl-protocol` to `--ssl.protocol` in client tools (was already done for arangod, but overlooked for client tools) * fix handling of `--ssl.protocol` value 5 (TLS v1.2) in client tools, which claimed to support it but didn't * config file can use '@include' to include a different config file as base v3.0.0-rc1 (2016-06-10) ----------------------- * the user management has changed: it now has users that are independent of databases. A user can have one or more database assigned to the user. * forward ported V8 Comparator bugfix for inline heuristics from https://github.com/v8/v8/commit/5ff7901e24c2c6029114567de5a08ed0f1494c81 * changed to-string conversion for AQL objects and arrays, used by the AQL function `TO_STRING()` and implicit to-string casts in AQL - arrays are now converted into their JSON-stringify equivalents, e.g. - `[ ]` is now converted to `[]` - `[ 1, 2, 3 ]` is now converted to `[1,2,3]` - `[ "test", 1, 2 ] is now converted to `["test",1,2]` Previous versions of ArangoDB converted arrays with no members into the empty string, and non-empty arrays into a comma-separated list of member values, without the surrounding angular brackets. Additionally, string array members were not enclosed in quotes in the result string: - `[ ]` was converted to `` - `[ 1, 2, 3 ]` was converted to `1,2,3` - `[ "test", 1, 2 ] was converted to `test,1,2` - objects are now converted to their JSON-stringify equivalents, e.g. - `{ }` is converted to `{}` - `{ a: 1, b: 2 }` is converted to `{"a":1,"b":2}` - `{ "test" : "foobar" }` is converted to `{"test":"foobar"}` Previous versions of ArangoDB always converted objects into the string `[object Object]` This change affects also the AQL functions `CONCAT()` and `CONCAT_SEPARATOR()` which treated array values differently in previous versions. Previous versions of ArangoDB automatically flattened array values on the first level of the array, e.g. `CONCAT([1, 2, 3, [ 4, 5, 6 ]])` produced `1,2,3,4,5,6`. Now this will produce `[1,2,3,[4,5,6]]`. To flatten array members on the top level, you can now use the more explicit `CONCAT(FLATTEN([1, 2, 3, [4, 5, 6]], 1))`. * added C++ implementations for AQL functions `SLICE()`, `CONTAINS()` and `RANDOM_TOKEN()` * as a consequence of the upgrade to V8 version 5, the implementation of the JavaScript `Buffer` object had to be changed. JavaScript `Buffer` objects in ArangoDB now always store their data on the heap. There is no shared pool for small Buffer values, and no pointing into existing Buffer data when extracting slices. This change may increase the cost of creating Buffers with short contents or when peeking into existing Buffers, but was required for safer memory management and to prevent leaks. * the `db` object's function `_listDatabases()` was renamed to just `_databases()` in order to make it more consistent with the existing `_collections()` function. Additionally the `db` object's `_listEndpoints()` function was renamed to just `_endpoints()`. * changed default value of `--server.authentication` from `false` to `true` in configuration files etc/relative/arangod.conf and etc/arangodb/arangod.conf.in. This means the server will be started with authentication enabled by default, requiring all client connections to provide authentication data when connecting to ArangoDB. Authentication can still be turned off via setting the value of `--server.authentication` to `false` in ArangoDB's configuration files or by specifying the option on the command-line. * Changed result format for querying all collections via the API GET `/_api/collection`. Previous versions of ArangoDB returned an object with an attribute named `collections` and an attribute named `names`. Both contained all available collections, but `collections` contained the collections as an array, and `names` contained the collections again, contained in an object in which the attribute names were the collection names, e.g. ``` { "collections": [ {"id":"5874437","name":"test","isSystem":false,"status":3,"type":2}, {"id":"17343237","name":"something","isSystem":false,"status":3,"type":2}, ... ], "names": { "test": {"id":"5874437","name":"test","isSystem":false,"status":3,"type":2}, "something": {"id":"17343237","name":"something","isSystem":false,"status":3,"type":2}, ... } } ``` This result structure was redundant, and therefore has been simplified to just ``` { "result": [ {"id":"5874437","name":"test","isSystem":false,"status":3,"type":2}, {"id":"17343237","name":"something","isSystem":false,"status":3,"type":2}, ... ] } ``` in ArangoDB 3.0. * added AQL functions `TYPENAME()` and `HASH()` * renamed arangob tool to arangobench * added AQL string comparison operator `LIKE` The operator can be used to compare strings like this: value LIKE search The operator is currently implemented by calling the already existing AQL function `LIKE`. This change also makes `LIKE` an AQL keyword. Using `LIKE` in either case as an attribute or collection name in AQL thus requires quoting. * make AQL optimizer rule "remove-unnecessary-calculations" fire in more cases The rule will now remove calculations that are used exactly once in other expressions (e.g. `LET a = doc RETURN a.value`) and calculations, or calculations that are just references (e.g. `LET a = b`). * renamed AQL optimizer rule "merge-traversal-filter" to "optimize-traversals" Additionally, the optimizer rule will remove unused edge and path result variables from the traversal in case they are specified in the `FOR` section of the traversal, but not referenced later in the query. This saves constructing edges and paths results. * added AQL optimizer rule "inline-subqueries" This rule can pull out certain subqueries that are used as an operand to a `FOR` loop one level higher, eliminating the subquery completely. For example, the query FOR i IN (FOR j IN [1,2,3] RETURN j) RETURN i will be transformed by the rule to: FOR i IN [1,2,3] RETURN i The query FOR name IN (FOR doc IN _users FILTER doc.status == 1 RETURN doc.name) LIMIT 2 RETURN name will be transformed into FOR tmp IN _users FILTER tmp.status == 1 LIMIT 2 RETURN tmp.name The rule will only fire when the subquery is used as an operand to a `FOR` loop, and if the subquery does not contain a `COLLECT` with an `INTO` variable. * added new endpoint "srv://" for DNS service records * The result order of the AQL functions VALUES and ATTRIBUTES has never been guaranteed and it only had the "correct" ordering by accident when iterating over objects that were not loaded from the database. This accidental behavior is now changed by introduction of VelocyPack. No ordering is guaranteed unless you specify the sort parameter. * removed configure option `--enable-logger` * added AQL array comparison operators All AQL comparison operators now also exist in an array variant. In the array variant, the operator is preceded with one of the keywords *ALL*, *ANY* or *NONE*. Using one of these keywords changes the operator behavior to execute the comparison operation for all, any, or none of its left hand argument values. It is therefore expected that the left hand argument of an array operator is an array. Examples: [ 1, 2, 3 ] ALL IN [ 2, 3, 4 ] // false [ 1, 2, 3 ] ALL IN [ 1, 2, 3 ] // true [ 1, 2, 3 ] NONE IN [ 3 ] // false [ 1, 2, 3 ] NONE IN [ 23, 42 ] // true [ 1, 2, 3 ] ANY IN [ 4, 5, 6 ] // false [ 1, 2, 3 ] ANY IN [ 1, 42 ] // true [ 1, 2, 3 ] ANY == 2 // true [ 1, 2, 3 ] ANY == 4 // false [ 1, 2, 3 ] ANY > 0 // true [ 1, 2, 3 ] ANY <= 1 // true [ 1, 2, 3 ] NONE < 99 // false [ 1, 2, 3 ] NONE > 10 // true [ 1, 2, 3 ] ALL > 2 // false [ 1, 2, 3 ] ALL > 0 // true [ 1, 2, 3 ] ALL >= 3 // false ["foo", "bar"] ALL != "moo" // true ["foo", "bar"] NONE == "bar" // false ["foo", "bar"] ANY == "foo" // true * improved AQL optimizer to remove unnecessary sort operations in more cases * allow enclosing AQL identifiers in forward ticks in addition to using backward ticks This allows for convenient writing of AQL queries in JavaScript template strings (which are delimited with backticks themselves), e.g. var q = `FOR doc IN ´collection´ RETURN doc.´name´`; * allow to set `print.limitString` to configure the number of characters to output before truncating * make logging configurable per log "topic" `--log.level ` sets the global log level to , e.g. `info`, `debug`, `trace`. `--log.level topic=` sets the log level for a specific topic. Currently, the following topics exist: `collector`, `compactor`, `mmap`, `performance`, `queries`, and `requests`. `performance` and `requests` are set to FATAL by default. `queries` is set to info. All others are set to the global level by default. The new log option `--log.output ` allows directing the global or per-topic log output to different outputs. The output definition "" can be one of "-" for stdin "+" for stderr "syslog://" "syslog:///" "file://" The option can be specified multiple times in order to configure the output for different log topics. To set up a per-topic output configuration, use `--log.output =`, e.g. queries=file://queries.txt logs all queries to the file "queries.txt". * the option `--log.requests-file` is now deprecated. Instead use `--log.level requests=info` `--log.output requests=file://requests.txt` * the option `--log.facility` is now deprecated. Instead use `--log.output requests=syslog://facility` * the option `--log.performance` is now deprecated. Instead use `--log.level performance=trace` * removed option `--log.source-filter` * removed configure option `--enable-logger` * change collection directory names to include a random id component at the end The new pattern is `collection--`, where `` is the collection id and `` is a random number. Previous versions of ArangoDB used a pattern `collection-` without the random number. ArangoDB 3.0 understands both the old and name directory name patterns. * removed mostly unused internal spin-lock implementation * removed support for pre-Windows 7-style locks. This removes compatibility for Windows versions older than Windows 7 (e.g. Windows Vista, Windows XP) and Windows 2008R2 (e.g. Windows 2008). * changed names of sub-threads started by arangod * added option `--default-number-of-shards` to arangorestore, allowing creating collections with a specifiable number of shards from a non-cluster dump * removed support for CoffeeScript source files * removed undocumented SleepAndRequeue * added WorkMonitor to inspect server threads * when downloading a Foxx service from the web interface the suggested filename is now based on the service's mount path instead of simply "app.zip" * the `@arangodb/request` response object now stores the parsed JSON response body in a property `json` instead of `body` when the request was made using the `json` option. The `body` instead contains the response body as a string. * the Foxx API has changed significantly, 2.8 services are still supported using a backwards-compatible "legacy mode" v2.8.12 (XXXX-XX-XX) -------------------- * issue #2091: decrease connect timeout to 5 seconds on startup * fixed issue #2072 * slightly better error diagnostics for some replication errors * fixed issue #1977 * fixed issue in `INTERSECTION` AQL function with duplicate elements in the source arrays * fixed issue #1962 * fixed issue #1959 * export aqlQuery template handler as require('org/arangodb').aql for forwards-compatibility v2.8.11 (2016-07-13) -------------------- * fixed array index batch insertion issues for hash indexes that caused problems when no elements remained for insertion * fixed issue #1937 v2.8.10 (2016-07-01) -------------------- * make sure next local _rev value used for a document is at least as high as the _rev value supplied by external sources such as replication * make adding a collection in both read- and write-mode to a transaction behave as expected (write includes read). This prevents the `unregister collection used in transaction` error * fixed sometimes invalid result for `byExample(...).count()` when an index plus post-filtering was used * fixed "collection is a nullptr" issue when starting a traversal from a transaction * honor the value of startup option `--database.wait-for-sync` (that is used to control whether new collections are created with `waitForSync` set to `true` by default) also when creating collections via the HTTP API (and thus the ArangoShell). When creating a collection via these mechanisms, the option was ignored so far, which was inconsistent. * fixed issue #1826: arangosh --javascript.execute: internal error (geo index issue) * fixed issue #1823: Arango crashed hard executing very simple query on windows v2.8.9 (2016-05-13) ------------------- * fixed escaping and quoting of extra parameters for executables in Mac OS X App * added "waiting for" status variable to web interface collection figures view * fixed undefined behavior in query cache invaldation * fixed access to /_admin/statistics API in case statistics are disable via option `--server.disable-statistics` * Foxx manager will no longer fail hard when Foxx store is unreachable unless installing a service from the Foxx store (e.g. when behind a firewall or GitHub is unreachable). v2.8.8 (2016-04-19) ------------------- * fixed issue #1805: Query: internal error (location: arangod/Aql/AqlValue.cpp:182). Please report this error to arangodb.com (while executing) * allow specifying collection name prefixes for `_from` and `_to` in arangoimp: To avoid specifying complete document ids (consisting of collection names and document keys) for *_from* and *_to* values when importing edges with arangoimp, there are now the options *--from-collection-prefix* and *--to-collection-prefix*. If specified, these values will be automatically prepended to each value in *_from* (or *_to* resp.). This allows specifying only document keys inside *_from* and/or *_to*. *Example* > arangoimp --from-collection-prefix users --to-collection-prefix products ... Importing the following document will then create an edge between *users/1234* and *products/4321*: ```js { "_from" : "1234", "_to" : "4321", "desc" : "users/1234 is connected to products/4321" } ``` * requests made with the interactive system API documentation in the web interface (Swagger) will now respect the active database instead of always using `_system` v2.8.7 (2016-04-07) ------------------- * optimized primary=>secondary failover * fix to-boolean conversion for documents in AQL * expose the User-Agent HTTP header from the ArangoShell since Github seems to require it now, and we use the ArangoShell for fetching Foxx repositories from Github * work with http servers that only send * fixed potential race condition between compactor and collector threads * fix removal of temporary directories on arangosh exit * javadoc-style comments in Foxx services are no longer interpreted as Foxx comments outside of controller/script/exports files (#1748) * removed remaining references to class syntax for Foxx Model and Repository from the documentation * added a safe-guard for corrupted master-pointer v2.8.6 (2016-03-23) ------------------- * arangosh can now execute JavaScript script files that contain a shebang in the first line of the file. This allows executing script files directly. Provided there is a script file `/path/to/script.js` with the shebang `#!arangosh --javascript.execute`: > cat /path/to/script.js #!arangosh --javascript.execute print("hello from script.js"); If the script file is made executable > chmod a+x /path/to/script.js it can be invoked on the shell directly and use arangosh for its execution: > /path/to/script.js hello from script.js This did not work in previous versions of ArangoDB, as the whole script contents (including the shebang) were treated as JavaScript code. Now shebangs in script files will now be ignored for all files passed to arangosh's `--javascript.execute` parameter. The alternative way of executing a JavaScript file with arangosh still works: > arangosh --javascript.execute /path/to/script.js hello from script.js * added missing reset of traversal state for nested traversals. The state of nested traversals (a traversal in an AQL query that was located in a repeatedly executed subquery or inside another FOR loop) was not reset properly, so that multiple invocations of the same nested traversal with different start vertices led to the nested traversal always using the start vertex provided on the first invocation. * fixed issue #1781: ArangoDB startup time increased tremendously * fixed issue #1783: SIGHUP should rotate the log v2.8.5 (2016-03-11) ------------------- * Add OpenSSL handler for TLS V1.2 as sugested by kurtkincaid in #1771 * fixed issue #1765 (The webinterface should display the correct query time) and #1770 (Display ACTUAL query time in aardvark's AQL editor) * Windows: the unhandled exception handler now calls the windows logging facilities directly without locks. This fixes lockups on crashes from the logging framework. * improve nullptr handling in logger. * added new endpoint "srv://" for DNS service records * `org/arangodb/request` no longer sets the content-type header to the string "undefined" when no content-type header should be sent (issue #1776) v2.8.4 (2016-03-01) ------------------- * global modules are no longer incorrectly resolved outside the ArangoDB JavaScript directory or the Foxx service's root directory (issue #1577) * improved error messages from Foxx and JavaScript (issues #1564, #1565, #1744) v2.8.3 (2016-02-22) ------------------- * fixed AQL filter condition collapsing for deeply-nested cases, potentially enabling usage of indexes in some dedicated cases * added parentheses in AQL explain command output to correctly display precedence of logical and arithmetic operators * Foxx Model event listeners defined on the model are now correctly invoked by the Repository methods (issue #1665) * Deleting a Foxx service in the frontend should now always succeed even if the files no longer exist on the file system (issue #1358) * Routing actions loaded from the database no longer throw exceptions when trying to load other modules using "require" * The `org/arangodb/request` response object now sets a property `json` to the parsed JSON response body in addition to overwriting the `body` property when the request was made using the `json` option. * Improved Windows stability * Fixed a bug in the interactive API documentation that would escape slashes in document-handle fields. Document handles are now provided as separate fields for collection name and document key. v2.8.2 (2016-02-09) ------------------- * the continuous replication applier will now prevent the master's WAL logfiles from being removed if they are still needed by the applier on the slave. This should help slaves that suffered from masters garbage collection WAL logfiles which would have been needed by the slave later. The initial synchronization will block removal of still needed WAL logfiles on the master for 10 minutes initially, and will extend this period when further requests are made to the master. Initial synchronization hands over its handle for blocking logfile removal to the continuous replication when started via the *setupReplication* function. In this case, continuous replication will extend the logfile removal blocking period for the required WAL logfiles when the slave makes additional requests. All handles that block logfile removal will time out automatically after at most 5 minutes should a master not be contacted by the slave anymore (e.g. in case the slave's replication is turned off, the slaves loses the connection to the master or the slave goes down). * added all-in-one function *setupReplication* to synchronize data from master to slave and start the continuous replication: require("@arangodb/replication").setupReplication(configuration); The command will return when the initial synchronization is finished and the continuous replication has been started, or in case the initial synchronization has failed. If the initial synchronization is successful, the command will store the given configuration on the slave. It also configures the continuous replication to start automatically if the slave is restarted, i.e. *autoStart* is set to *true*. If the command is run while the slave's replication applier is already running, it will first stop the running applier, drop its configuration and do a resynchronization of data with the master. It will then use the provided configration, overwriting any previously existing replication configuration on the slave. The following example demonstrates how to use the command for setting up replication for the *_system* database. Note that it should be run on the slave and not the master: db._useDatabase("_system"); require("@arangodb/replication").setupReplication({ endpoint: "tcp://master.domain.org:8529", username: "myuser", password: "mypasswd", verbose: false, includeSystem: false, incremental: true, autoResync: true }); * the *sync* and *syncCollection* functions now always start the data synchronization as an asynchronous server job. The call to *sync* or *syncCollection* will block until synchronization is either complete or has failed with an error. The functions will automatically poll the slave periodically for status updates. The main benefit is that the connection to the slave does not need to stay open permanently and is thus not affected by timeout issues. Additionally the caller does not need to query the synchronization status from the slave manually as this is now performed automatically by these functions. * fixed undefined behavior when explaining some types of AQL traversals, fixed display of some types of traversals in AQL explain output v2.8.1 (2016-01-29) ------------------- * Improved AQL Pattern matching by allowing to specify a different traversal direction for one or many of the edge collections. FOR v, e, p IN OUTBOUND @start @@ec1, INBOUND @@ec2, @@ec3 will traverse *ec1* and *ec3* in the OUTBOUND direction and for *ec2* it will use the INBOUND direction. These directions can be combined in arbitrary ways, the direction defined after *IN [steps]* will we used as default direction and can be overriden for specific collections. This feature is only available for collection lists, it is not possible to combine it with graph names. * detect more types of transaction deadlocks early * fixed display of relational operators in traversal explain output * fixed undefined behavior in AQL function `PARSE_IDENTIFIER` * added "engines" field to Foxx services generated in the admin interface * added AQL function `IS_SAME_COLLECTION`: *IS_SAME_COLLECTION(collection, document)*: Return true if *document* has the same collection id as the collection specified in *collection*. *document* can either be a [document handle](../Glossary/README.md#document-handle) string, or a document with an *_id* attribute. The function does not validate whether the collection actually contains the specified document, but only compares the name of the specified collection with the collection name part of the specified document. If *document* is neither an object with an *id* attribute nor a *string* value, the function will return *null* and raise a warning. /* true */ IS_SAME_COLLECTION('_users', '_users/my-user') IS_SAME_COLLECTION('_users', { _id: '_users/my-user' }) /* false */ IS_SAME_COLLECTION('_users', 'foobar/baz') IS_SAME_COLLECTION('_users', { _id: 'something/else' }) v2.8.0 (2016-01-25) ------------------- * avoid recursive locking v2.8.0-beta8 (2016-01-19) ------------------------- * improved internal datafile statistics for compaction and compaction triggering conditions, preventing excessive growth of collection datafiles under some workloads. This should also fix issue #1596. * renamed AQL optimizer rule `remove-collect-into` to `remove-collect-variables` * fixed primary and edge index lookups prematurely aborting searches when the specified id search value contained a different collection than the collection the index was created for v2.8.0-beta7 (2016-01-06) ------------------------- * added vm.runInThisContext * added AQL keyword `AGGREGATE` for use in AQL `COLLECT` statement Using `AGGREGATE` allows more efficient aggregation (incrementally while building the groups) than previous versions of AQL, which built group aggregates afterwards from the total of all group values. `AGGREGATE` can be used inside a `COLLECT` statement only. If used, it must follow the declaration of grouping keys: FOR doc IN collection COLLECT gender = doc.gender AGGREGATE minAge = MIN(doc.age), maxAge = MAX(doc.age) RETURN { gender, minAge, maxAge } or, if no grouping keys are used, it can follow the `COLLECT` keyword: FOR doc IN collection COLLECT AGGREGATE minAge = MIN(doc.age), maxAge = MAX(doc.age) RETURN { minAge, maxAge } Only specific expressions are allowed on the right-hand side of each `AGGREGATE` assignment: - on the top level the expression must be a call to one of the supported aggregation functions `LENGTH`, `MIN`, `MAX`, `SUM`, `AVERAGE`, `STDDEV_POPULATION`, `STDDEV_SAMPLE`, `VARIANCE_POPULATION`, or `VARIANCE_SAMPLE` - the expression must not refer to variables introduced in the `COLLECT` itself * Foxx: mocha test paths with wildcard characters (asterisks) now work on Windows * reserved AQL keyword `NONE` for future use * web interface: fixed a graph display bug concerning dashboard view * web interface: fixed several bugs during the dashboard initialize process * web interface: included several bugfixes: #1597, #1611, #1623 * AQL query optimizer now converts `LENGTH(collection-name)` to an optimized expression that returns the number of documents in a collection * adjusted the behavior of the expansion (`[*]`) operator in AQL for non-array values In ArangoDB 2.8, calling the expansion operator on a non-array value will always return an empty array. Previous versions of ArangoDB expanded non-array values by calling the `TO_ARRAY()` function for the value, which for example returned an array with a single value for boolean, numeric and string input values, and an array with the object's values for an object input value. This behavior was inconsistent with how the expansion operator works for the array indexes in 2.8, so the behavior is now unified: - if the left-hand side operand of `[*]` is an array, the array will be returned as is when calling `[*]` on it - if the left-hand side operand of `[*]` is not an array, an empty array will be returned by `[*]` AQL queries that rely on the old behavior can be changed by either calling `TO_ARRAY` explicitly or by using the `[*]` at the correct position. The following example query will change its result in 2.8 compared to 2.7: LET values = "foo" RETURN values[*] In 2.7 the query has returned the array `[ "foo" ]`, but in 2.8 it will return an empty array `[ ]`. To make it return the array `[ "foo" ]` again, an explicit `TO_ARRAY` function call is needed in 2.8 (which in this case allows the removal of the `[*]` operator altogether). This also works in 2.7: LET values = "foo" RETURN TO_ARRAY(values) Another example: LET values = [ { name: "foo" }, { name: "bar" } ] RETURN values[*].name[*] The above returned `[ [ "foo" ], [ "bar" ] ] in 2.7. In 2.8 it will return `[ [ ], [ ] ]`, because the value of `name` is not an array. To change the results to the 2.7 style, the query can be changed to LET values = [ { name: "foo" }, { name: "bar" } ] RETURN values[* RETURN TO_ARRAY(CURRENT.name)] The above also works in 2.7. The following types of queries won't change: LET values = [ 1, 2, 3 ] RETURN values[*] LET values = [ { name: "foo" }, { name: "bar" } ] RETURN values[*].name LET values = [ { names: [ "foo", "bar" ] }, { names: [ "baz" ] } ] RETURN values[*].names[*] LET values = [ { names: [ "foo", "bar" ] }, { names: [ "baz" ] } ] RETURN values[*].names[**] * slightly adjusted V8 garbage collection strategy so that collection eventually happens in all contexts that hold V8 external references to documents and collections. also adjusted default value of `--javascript.gc-frequency` from 10 seconds to 15 seconds, as less internal operations are carried out in JavaScript. * fixes for AQL optimizer and traversal * added `--create-collection-type` option to arangoimp This allows specifying the type of the collection to be created when `--create-collection` is set to `true`. * Foxx export cache should no longer break if a broken app is loaded in the web admin interface. v2.8.0-beta2 (2015-12-16) ------------------------- * added AQL query optimizer rule "sort-in-values" This rule pre-sorts the right-hand side operand of the `IN` and `NOT IN` operators so the operation can use a binary search with logarithmic complexity instead of a linear search. The rule is applied when the right-hand side operand of an `IN` or `NOT IN` operator in a filter condition is a variable that is defined in a different loop/scope than the operator itself. Additionally, the filter condition must consist of solely the `IN` or `NOT IN` operation in order to avoid any side-effects. * changed collection status terminology in web interface for collections for which an unload request has been issued from `in the process of being unloaded` to `will be unloaded`. * unloading a collection via the web interface will now trigger garbage collection in all v8 contexts and force a WAL flush. This increases the chances of perfoming the unload faster. * added the following attributes to the result of `collection.figures()` and the corresponding HTTP API at `PUT /_api/collection//figures`: - `documentReferences`: The number of references to documents in datafiles that JavaScript code currently holds. This information can be used for debugging compaction and unload issues. - `waitingFor`: An optional string value that contains information about which object type is at the head of the collection's cleanup queue. This information can be used for debugging compaction and unload issues. - `compactionStatus.time`: The point in time the compaction for the collection was last executed. This information can be used for debugging compaction issues. - `compactionStatus.message`: The action that was performed when the compaction was last run for the collection. This information can be used for debugging compaction issues. Note: `waitingFor` and `compactionStatus` may be empty when called on a coordinator in a cluster. * the compaction will now provide queryable status info that can be used to track its progress. The compaction status is displayed in the web interface, too. * better error reporting for arangodump and arangorestore * arangodump will now fail by default when trying to dump edges that refer to already dropped collections. This can be circumvented by specifying the option `--force true` when invoking arangodump * fixed cluster upgrade procedure * the AQL functions `NEAR` and `WITHIN` now have stricter validations for their input parameters `limit`, `radius` and `distance`. They may now throw exceptions when invalid parameters are passed that may have not led to exceptions in previous versions. * deprecation warnings now log stack traces * Foxx: improved backwards compatibility with 2.5 and 2.6 - reverted Model and Repository back to non-ES6 "classes" because of compatibility issues when using the extend method with a constructor - removed deprecation warnings for extend and controller.del - restored deprecated method Model.toJSONSchema - restored deprecated `type`, `jwt` and `sessionStorageApp` options in Controller#activateSessions * Fixed a deadlock problem in the cluster v2.8.0-beta1 (2015-12-06) ------------------------- * added AQL function `IS_DATESTRING(value)` Returns true if *value* is a string that can be used in a date function. This includes partial dates such as *2015* or *2015-10* and strings containing invalid dates such as *2015-02-31*. The function will return false for all non-string values, even if some of them may be usable in date functions. v2.8.0-alpha1 (2015-12-03) -------------------------- * added AQL keywords `GRAPH`, `OUTBOUND`, `INBOUND` and `ANY` for use in graph traversals, reserved AQL keyword `ALL` for future use Usage of these keywords as collection names, variable names or attribute names in AQL queries will not be possible without quoting. For example, the following AQL query will still work as it uses a quoted collection name and a quoted attribute name: FOR doc IN `OUTBOUND` RETURN doc.`any` * issue #1593: added AQL `POW` function for exponentation * added cluster execution site info in explain output for AQL queries * replication improvements: - added `autoResync` configuration parameter for continuous replication. When set to `true`, a replication slave will automatically trigger a full data re-synchronization with the master when the master cannot provide the log data the slave had asked for. Note that `autoResync` will only work when the option `requireFromPresent` is also set to `true` for the continuous replication, or when the continuous syncer is started and detects that no start tick is present. Automatic re-synchronization may transfer a lot of data from the master to the slave and may be expensive. It is therefore turned off by default. When turned off, the slave will never perform an automatic re-synchronization with the master. - added `idleMinWaitTime` and `idleMaxWaitTime` configuration parameters for continuous replication. These parameters can be used to control the minimum and maximum wait time the slave will (intentionally) idle and not poll for master log changes in case the master had sent the full logs already. The `idleMaxWaitTime` value will only be used when `adapativePolling` is set to `true`. When `adaptivePolling` is disable, only `idleMinWaitTime` will be used as a constant time span in which the slave will not poll the master for further changes. The default values are 0.5 seconds for `idleMinWaitTime` and 2.5 seconds for `idleMaxWaitTime`, which correspond to the hard-coded values used in previous versions of ArangoDB. - added `initialSyncMaxWaitTime` configuration parameter for initial and continuous replication This option controls the maximum wait time (in seconds) that the initial synchronization will wait for a response from the master when fetching initial collection data. If no response is received within this time period, the initial synchronization will give up and fail. This option is also relevant for continuous replication in case *autoResync* is set to *true*, as then the continuous replication may trigger a full data re-synchronization in case the master cannot the log data the slave had asked for. - HTTP requests sent from the slave to the master during initial synchronization will now be retried if they fail with connection problems. - the initial synchronization now logs its progress so it can be queried using the regular replication status check APIs. - added `async` attribute for `sync` and `syncCollection` operations called from the ArangoShell. Setthing this attribute to `true` will make the synchronization job on the server go into the background, so that the shell does not block. The status of the started asynchronous synchronization job can be queried from the ArangoShell like this: /* starts initial synchronization */ var replication = require("@arangodb/replication"); var id = replication.sync({ endpoint: "tcp://master.domain.org:8529", username: "myuser", password: "mypasswd", async: true }); /* now query the id of the returned async job and print the status */ print(replication.getSyncResult(id)); The result of `getSyncResult()` will be `false` while the server-side job has not completed, and different to `false` if it has completed. When it has completed, all job result details will be returned by the call to `getSyncResult()`. * fixed non-deterministic query results in some cluster queries * fixed issue #1589 * return HTTP status code 410 (gone) instead of HTTP 408 (request timeout) for server-side operations that are canceled / killed. Sending 410 instead of 408 prevents clients from re-starting the same (canceled) operation. Google Chrome for example sends the HTTP request again in case it is responded with an HTTP 408, and this is exactly the opposite of the desired behavior when an operation is canceled / killed by the user. * web interface: queries in AQL editor now cancelable * web interface: dashboard - added replication information * web interface: AQL editor now supports bind parameters * added startup option `--server.hide-product-header` to make the server not send the HTTP response header `"Server: ArangoDB"` in its HTTP responses. By default, the option is turned off so the header is still sent as usual. * added new AQL function `UNSET_RECURSIVE` to recursively unset attritutes from objects/documents * switched command-line editor in ArangoShell and arangod to linenoise-ng * added automatic deadlock detection for transactions In case a deadlock is detected, a multi-collection operation may be rolled back automatically and fail with error 29 (`deadlock detected`). Client code for operations containing more than one collection should be aware of this potential error and handle it accordingly, either by giving up or retrying the transaction. * Added C++ implementations for the AQL arithmetic operations and the following AQL functions: - ABS - APPEND - COLLECTIONS - CURRENT_DATABASE - DOCUMENT - EDGES - FIRST - FIRST_DOCUMENT - FIRST_LIST - FLATTEN - FLOOR - FULLTEXT - LAST - MEDIAN - MERGE_RECURSIVE - MINUS - NEAR - NOT_NULL - NTH - PARSE_IDENTIFIER - PERCENTILE - POP - POSITION - PUSH - RAND - RANGE - REMOVE_NTH - REMOVE_VALUE - REMOVE_VALUES - ROUND - SHIFT - SQRT - STDDEV_POPULATION - STDDEV_SAMPLE - UNSHIFT - VARIANCE_POPULATION - VARIANCE_SAMPLE - WITHIN - ZIP * improved performance of skipping over many documents in an AQL query when no indexes and no filters are used, e.g. FOR doc IN collection LIMIT 1000000, 10 RETURN doc * Added array indexes Hash indexes and skiplist indexes can now optionally be defined for array values so they index individual array members. To define an index for array values, the attribute name is extended with the expansion operator `[*]` in the index definition: arangosh> db.colName.ensureHashIndex("tags[*]"); When given the following document { tags: [ "AQL", "ArangoDB", "Index" ] } the index will now contain the individual values `"AQL"`, `"ArangoDB"` and `"Index"`. Now the index can be used for finding all documents having `"ArangoDB"` somewhere in their tags array using the following AQL query: FOR doc IN colName FILTER "ArangoDB" IN doc.tags[*] RETURN doc * rewrote AQL query optimizer rule `use-index-range` and renamed it to `use-indexes`. The name change affects rule names in the optimizer's output. * rewrote AQL execution node `IndexRangeNode` and renamed it to `IndexNode`. The name change affects node names in the optimizer's explain output. * added convenience function `db._explain(query)` for human-readable explanation of AQL queries * module resolution as used by `require` now behaves more like in node.js * the `org/arangodb/request` module now returns response bodies for error responses by default. The old behavior of not returning bodies for error responses can be re-enabled by explicitly setting the option `returnBodyOnError` to `false` (#1437) v2.7.6 (2016-01-30) ------------------- * detect more types of transaction deadlocks early v2.7.5 (2016-01-22) ------------------- * backported added automatic deadlock detection for transactions In case a deadlock is detected, a multi-collection operation may be rolled back automatically and fail with error 29 (`deadlock detected`). Client code for operations containing more than one collection should be aware of this potential error and handle it accordingly, either by giving up or retrying the transaction. * improved internal datafile statistics for compaction and compaction triggering conditions, preventing excessive growth of collection datafiles under some workloads. This should also fix issue #1596. * Foxx export cache should no longer break if a broken app is loaded in the web admin interface. * Foxx: removed some incorrect deprecation warnings. * Foxx: mocha test paths with wildcard characters (asterisks) now work on Windows v2.7.4 (2015-12-21) ------------------- * slightly adjusted V8 garbage collection strategy so that collection eventually happens in all contexts that hold V8 external references to documents and collections. * added the following attributes to the result of `collection.figures()` and the corresponding HTTP API at `PUT /_api/collection//figures`: - `documentReferences`: The number of references to documents in datafiles that JavaScript code currently holds. This information can be used for debugging compaction and unload issues. - `waitingFor`: An optional string value that contains information about which object type is at the head of the collection's cleanup queue. This information can be used for debugging compaction and unload issues. - `compactionStatus.time`: The point in time the compaction for the collection was last executed. This information can be used for debugging compaction issues. - `compactionStatus.message`: The action that was performed when the compaction was last run for the collection. This information can be used for debugging compaction issues. Note: `waitingFor` and `compactionStatus` may be empty when called on a coordinator in a cluster. * the compaction will now provide queryable status info that can be used to track its progress. The compaction status is displayed in the web interface, too. v2.7.3 (2015-12-17) ------------------- * fixed some replication value conversion issues when replication applier properties were set via ArangoShell * fixed disappearing of documents for collections transferred via `sync` or `syncCollection` if the collection was dropped right before synchronization and drop and (re-)create collection markers were located in the same WAL file * fixed an issue where overwriting the system sessions collection would break the web interface when authentication is enabled v2.7.2 (2015-12-01) ------------------- * replication improvements: - added `autoResync` configuration parameter for continuous replication. When set to `true`, a replication slave will automatically trigger a full data re-synchronization with the master when the master cannot provide the log data the slave had asked for. Note that `autoResync` will only work when the option `requireFromPresent` is also set to `true` for the continuous replication, or when the continuous syncer is started and detects that no start tick is present. Automatic re-synchronization may transfer a lot of data from the master to the slave and may be expensive. It is therefore turned off by default. When turned off, the slave will never perform an automatic re-synchronization with the master. - added `idleMinWaitTime` and `idleMaxWaitTime` configuration parameters for continuous replication. These parameters can be used to control the minimum and maximum wait time the slave will (intentionally) idle and not poll for master log changes in case the master had sent the full logs already. The `idleMaxWaitTime` value will only be used when `adapativePolling` is set to `true`. When `adaptivePolling` is disable, only `idleMinWaitTime` will be used as a constant time span in which the slave will not poll the master for further changes. The default values are 0.5 seconds for `idleMinWaitTime` and 2.5 seconds for `idleMaxWaitTime`, which correspond to the hard-coded values used in previous versions of ArangoDB. - added `initialSyncMaxWaitTime` configuration parameter for initial and continuous replication This option controls the maximum wait time (in seconds) that the initial synchronization will wait for a response from the master when fetching initial collection data. If no response is received within this time period, the initial synchronization will give up and fail. This option is also relevant for continuous replication in case *autoResync* is set to *true*, as then the continuous replication may trigger a full data re-synchronization in case the master cannot the log data the slave had asked for. - HTTP requests sent from the slave to the master during initial synchronization will now be retried if they fail with connection problems. - the initial synchronization now logs its progress so it can be queried using the regular replication status check APIs. * fixed non-deterministic query results in some cluster queries * added missing lock instruction for primary index in compactor size calculation * fixed issue #1589 * fixed issue #1583 * fixed undefined behavior when accessing the top level of a document with the `[*]` operator * fixed potentially invalid pointer access in shaper when the currently accessed document got re-located by the WAL collector at the very same time * Foxx: optional configuration options no longer log validation errors when assigned empty values (#1495) * Foxx: constructors provided to Repository and Model sub-classes via extend are now correctly called (#1592) v2.7.1 (2015-11-07) ------------------- * switch to linenoise next generation * exclude `_apps` collection from replication The slave has its own `_apps` collection which it populates on server start. When replicating data from the master to the slave, the data from the master may clash with the slave's own data in the `_apps` collection. Excluding the `_apps` collection from replication avoids this. * disable replication appliers when starting in modes `--upgrade`, `--no-server` and `--check-upgrade` * more detailed output in arango-dfdb * fixed "no start tick" issue in replication applier This error could occur after restarting a slave server after a shutdown when no data was ever transferred from the master to the slave via the continuous replication * fixed problem during SSL client connection abort that led to scheduler thread staying at 100% CPU saturation * fixed potential segfault in AQL `NEIGHBORS` function implementation when C++ function variant was used and collection names were passed as strings * removed duplicate target for some frontend JavaScript files from the Makefile * make AQL function `MERGE()` work on a single array parameter, too. This allows combining the attributes of multiple objects from an array into a single object, e.g. RETURN MERGE([ { foo: 'bar' }, { quux: 'quetzalcoatl', ruled: true }, { bar: 'baz', foo: 'done' } ]) will now return: { "foo": "done", "quux": "quetzalcoatl", "ruled": true, "bar": "baz" } * fixed potential deadlock in collection status changing on Windows * fixed hard-coded `incremental` parameter in shell implementation of `syncCollection` function in replication module * fix for GCC5: added check for '-stdlib' option v2.7.0 (2015-10-09) ------------------- * fixed request statistics aggregation When arangod was started in supervisor mode, the request statistics always showed 0 requests, as the statistics aggregation thread did not run then. * read server configuration files before dropping privileges. this ensures that the SSL keyfile specified in the configuration can be read with the server's start privileges (i.e. root when using a standard ArangoDB package). * fixed replication with a 2.6 replication configuration and issues with a 2.6 master * raised default value of `--server.descriptors-minimum` to 1024 * allow Foxx apps to be installed underneath URL path `/_open/`, so they can be (intentionally) accessed without authentication. * added *allowImplicit* sub-attribute in collections declaration of transactions. The *allowImplicit* attributes allows making transactions fail should they read-access a collection that was not explicitly declared in the *collections* array of the transaction. * added "special" password ARANGODB_DEFAULT_ROOT_PASSWORD. If you pass ARANGODB_DEFAULT_ROOT_PASSWORD as password, it will read the password from the environment variable ARANGODB_DEFAULT_ROOT_PASSWORD v2.7.0-rc2 (2015-09-22) ----------------------- * fix over-eager datafile compaction This should reduce the need to compact directly after loading a collection when a collection datafile contained many insertions and updates for the same documents. It should also prevent from re-compacting already merged datafiles in case not many changes were made. Compaction will also make fewer index lookups than before. * added `syncCollection()` function in module `org/arangodb/replication` This allows synchronizing the data of a single collection from a master to a slave server. Synchronization can either restore the whole collection by transferring all documents from the master to the slave, or incrementally by only transferring documents that differ. This is done by partitioning the collection's entire key space into smaller chunks and comparing the data chunk-wise between master and slave. Only chunks that are different will be re-transferred. The `syncCollection()` function can be used as follows: require("org/arangodb/replication").syncCollection(collectionName, options); e.g. require("org/arangodb/replication").syncCollection("myCollection", { endpoint: "tcp://127.0.0.1:8529", /* master */ username: "root", /* username for master */ password: "secret", /* password for master */ incremental: true /* use incremental mode */ }); * additionally allow the following characters in document keys: `(` `)` `+` `,` `=` `;` `$` `!` `*` `'` `%` v2.7.0-rc1 (2015-09-17) ----------------------- * removed undocumented server-side-only collection functions: * collection.OFFSET() * collection.NTH() * collection.NTH2() * collection.NTH3() * upgraded Swagger to version 2.0 for the Documentation This gives the user better prepared test request structures. More conversions will follow so finally client libraries can be auto-generated. * added extra AQL functions for date and time calculation and manipulation. These functions were contributed by GitHub users @CoDEmanX and @friday. A big thanks for their work! The following extra date functions are available from 2.7 on: * `DATE_DAYOFYEAR(date)`: Returns the day of year number of *date*. The return values range from 1 to 365, or 366 in a leap year respectively. * `DATE_ISOWEEK(date)`: Returns the ISO week date of *date*. The return values range from 1 to 53. Monday is considered the first day of the week. There are no fractional weeks, thus the last days in December may belong to the first week of the next year, and the first days in January may be part of the previous year's last week. * `DATE_LEAPYEAR(date)`: Returns whether the year of *date* is a leap year. * `DATE_QUARTER(date)`: Returns the quarter of the given date (1-based): * 1: January, February, March * 2: April, May, June * 3: July, August, September * 4: October, November, December - *DATE_DAYS_IN_MONTH(date)*: Returns the number of days in *date*'s month (28..31). * `DATE_ADD(date, amount, unit)`: Adds *amount* given in *unit* to *date* and returns the calculated date. *unit* can be either of the following to specify the time unit to add or subtract (case-insensitive): - y, year, years - m, month, months - w, week, weeks - d, day, days - h, hour, hours - i, minute, minutes - s, second, seconds - f, millisecond, milliseconds *amount* is the number of *unit*s to add (positive value) or subtract (negative value). * `DATE_SUBTRACT(date, amount, unit)`: Subtracts *amount* given in *unit* from *date* and returns the calculated date. It works the same as `DATE_ADD()`, except that it subtracts. It is equivalent to calling `DATE_ADD()` with a negative amount, except that `DATE_SUBTRACT()` can also subtract ISO durations. Note that negative ISO durations are not supported (i.e. starting with `-P`, like `-P1Y`). * `DATE_DIFF(date1, date2, unit, asFloat)`: Calculate the difference between two dates in given time *unit*, optionally with decimal places. Returns a negative value if *date1* is greater than *date2*. * `DATE_COMPARE(date1, date2, unitRangeStart, unitRangeEnd)`: Compare two partial dates and return true if they match, false otherwise. The parts to compare are defined by a range of time units. The full range is: years, months, days, hours, minutes, seconds, milliseconds. Pass the unit to start from as *unitRangeStart*, and the unit to end with as *unitRangeEnd*. All units in between will be compared. Leave out *unitRangeEnd* to only compare *unitRangeStart*. * `DATE_FORMAT(date, format)`: Format a date according to the given format string. It supports the following placeholders (case-insensitive): - %t: timestamp, in milliseconds since midnight 1970-01-01 - %z: ISO date (0000-00-00T00:00:00.000Z) - %w: day of week (0..6) - %y: year (0..9999) - %yy: year (00..99), abbreviated (last two digits) - %yyyy: year (0000..9999), padded to length of 4 - %yyyyyy: year (-009999 .. +009999), with sign prefix and padded to length of 6 - %m: month (1..12) - %mm: month (01..12), padded to length of 2 - %d: day (1..31) - %dd: day (01..31), padded to length of 2 - %h: hour (0..23) - %hh: hour (00..23), padded to length of 2 - %i: minute (0..59) - %ii: minute (00..59), padded to length of 2 - %s: second (0..59) - %ss: second (00..59), padded to length of 2 - %f: millisecond (0..999) - %fff: millisecond (000..999), padded to length of 3 - %x: day of year (1..366) - %xxx: day of year (001..366), padded to length of 3 - %k: ISO week date (1..53) - %kk: ISO week date (01..53), padded to length of 2 - %l: leap year (0 or 1) - %q: quarter (1..4) - %a: days in month (28..31) - %mmm: abbreviated English name of month (Jan..Dec) - %mmmm: English name of month (January..December) - %www: abbreviated English name of weekday (Sun..Sat) - %wwww: English name of weekday (Sunday..Saturday) - %&: special escape sequence for rare occasions - %%: literal % - %: ignored * new WAL logfiles and datafiles are now created non-sparse This prevents SIGBUS signals being raised when memory of a sparse datafile is accessed and the disk is full and the accessed file part is not actually disk-backed. In this case the mapped memory region is not necessarily backed by physical memory, and accessing the memory may raise SIGBUS and crash arangod. * the `internal.download()` function and the module `org/arangodb/request` used some internal library function that handled the sending of HTTP requests from inside of ArangoDB. This library unconditionally set an HTTP header `Accept-Encoding: gzip` in all outgoing HTTP requests. This has been fixed in 2.7, so `Accept-Encoding: gzip` is not set automatically anymore. Additionally, the header `User-Agent: ArangoDB` is not set automatically either. If client applications desire to send these headers, they are free to add it when constructing the requests using the `download` function or the request module. * fixed issue #1436: org/arangodb/request advertises deflate without supporting it * added template string generator function `aqlQuery` for generating AQL queries This can be used to generate safe AQL queries with JavaScript parameter variables or expressions easily: var name = 'test'; var attributeName = '_key'; var query = aqlQuery`FOR u IN users FILTER u.name == ${name} RETURN u.${attributeName}`; db._query(query); * report memory usage for document header data (revision id, pointer to data etc.) in `db.collection.figures()`. The memory used for document headers will now show up in the already existing attribute `indexes.size`. Due to that, the index sizes reported by `figures()` in 2.7 will be higher than those reported by 2.6, but the 2.7 values are more accurate. * IMPORTANT CHANGE: the filenames in dumps created by arangodump now contain not only the name of the dumped collection, but also an additional 32-digit hash value. This is done to prevent overwriting dump files in case-insensitive file systems when there exist multiple collections with the same name (but with different cases). For example, if a database has two collections: `test` and `Test`, previous versions of ArangoDB created the files * `test.structure.json` and `test.data.json` for collection `test` * `Test.structure.json` and `Test.data.json` for collection `Test` This did not work for case-insensitive filesystems, because the files for the second collection would have overwritten the files of the first. arangodump in 2.7 will create the following filenames instead: * `test_098f6bcd4621d373cade4e832627b4f6.structure.json` and `test_098f6bcd4621d373cade4e832627b4f6.data.json` * `Test_0cbc6611f5540bd0809a388dc95a615b.structure.json` and `Test_0cbc6611f5540bd0809a388dc95a615b.data.json` These filenames will be unambiguous even in case-insensitive filesystems. * IMPORTANT CHANGE: make arangod actually close lingering client connections when idle for at least the duration specified via `--server.keep-alive-timeout`. In previous versions of ArangoDB, connections were not closed by the server when the timeout was reached and the client was still connected. Now the connection is properly closed by the server in case of timeout. Client applications relying on the old behavior may now need to reconnect to the server when their idle connections time out and get closed (note: connections being idle for a long time may be closed by the OS or firewalls anyway - client applications should be aware of that and try to reconnect). * IMPORTANT CHANGE: when starting arangod, the server will drop the process privileges to the specified values in options `--server.uid` and `--server.gid` instantly after parsing the startup options. That means when either `--server.uid` or `--server.gid` are set, the privilege change will happen earlier. This may prevent binding the server to an endpoint with a port number lower than 1024 if the arangodb user has no privileges for that. Previous versions of ArangoDB changed the privileges later, so some startup actions were still carried out under the invoking user (i.e. likely *root* when started via init.d or system scripts) and especially binding to low port numbers was still possible there. The default privileges for user *arangodb* will not be sufficient for binding to port numbers lower than 1024. To have an ArangoDB 2.7 bind to a port number lower than 1024, it needs to be started with either a different privileged user, or the privileges of the *arangodb* user have to raised manually beforehand. * added AQL optimizer rule `patch-update-statements` * Linux startup scripts and systemd configuration for arangod now try to adjust the NOFILE (number of open files) limits for the process. The limit value is set to 131072 (128k) when ArangoDB is started via start/stop commands * When ArangoDB is started/stopped manually via the start/stop commands, the main process will wait for up to 10 seconds after it forks the supervisor and arangod child processes. If the startup fails within that period, the start/stop script will fail with an exit code other than zero. If the startup of the supervisor or arangod is still ongoing after 10 seconds, the main program will still return with exit code 0. The limit of 10 seconds is arbitrary because the time required for a startup is not known in advance. * added startup option `--database.throw-collection-not-loaded-error` Accessing a not-yet loaded collection will automatically load a collection on first access. This flag controls what happens in case an operation would need to wait for another thread to finalize loading a collection. If set to *true*, then the first operation that accesses an unloaded collection will load it. Further threads that try to access the same collection while it is still loading immediately fail with an error (1238, *collection not loaded*). This is to prevent all server threads from being blocked while waiting on the same collection to finish loading. When the first thread has completed loading the collection, the collection becomes regularly available, and all operations from that point on can be carried out normally, and error 1238 will not be thrown anymore for that collection. If set to *false*, the first thread that accesses a not-yet loaded collection will still load it. Other threads that try to access the collection while loading will not fail with error 1238 but instead block until the collection is fully loaded. This configuration might lead to all server threads being blocked because they are all waiting for the same collection to complete loading. Setting the option to *true* will prevent this from happening, but requires clients to catch error 1238 and react on it (maybe by scheduling a retry for later). The default value is *false*. * added better control-C support in arangosh When CTRL-C is pressed in arangosh, it will now print a `^C` first. Pressing CTRL-C again will reset the prompt if something was entered before, or quit arangosh if no command was entered directly before. This affects the arangosh version build with Readline-support only (Linux and MacOS). The MacOS version of ArangoDB for Homebrew now depends on Readline, too. The Homebrew formula has been changed accordingly. When self-compiling ArangoDB on MacOS without Homebrew, Readline now is a prerequisite. * increased default value for collection-specific `indexBuckets` value from 1 to 8 Collections created from 2.7 on will use the new default value of `8` if not overridden on collection creation or later using `collection.properties({ indexBuckets: ... })`. The `indexBuckets` value determines the number of buckets to use for indexes of type `primary`, `hash` and `edge`. Having multiple index buckets allows splitting an index into smaller components, which can be filled in parallel when a collection is loading. Additionally, resizing and reallocation of indexes are faster and less intrusive if the index uses multiple buckets, because resize and reallocation will affect only data in a single bucket instead of all index values. The index buckets will be filled in parallel when loading a collection if the collection has an `indexBuckets` value greater than 1 and the collection contains a significant amount of documents/edges (the current threshold is 256K documents but this value may change in future versions of ArangoDB). * changed HTTP client to use poll instead of select on Linux and MacOS This affects the ArangoShell and user-defined JavaScript code running inside arangod that initiates its own HTTP calls. Using poll instead of select allows using arbitrary high file descriptors (bigger than the compiled in FD_SETSIZE). Server connections are still handled using epoll, which has never been affected by FD_SETSIZE. * implemented AQL `LIKE` function using ICU regexes * added `RETURN DISTINCT` for AQL queries to return unique results: FOR doc IN collection RETURN DISTINCT doc.status This change also introduces `DISTINCT` as an AQL keyword. * removed `createNamedQueue()` and `addJob()` functions from org/arangodb/tasks * use less locks and more atomic variables in the internal dispatcher and V8 context handling implementations. This leads to improved throughput in some ArangoDB internals and allows for higher HTTP request throughput for many operations. A short overview of the improvements can be found here: https://www.arangodb.com/2015/08/throughput-enhancements/ * added shorthand notation for attribute names in AQL object literals: LET name = "Peter" LET age = 42 RETURN { name, age } The above is the shorthand equivalent of the generic form LET name = "Peter" LET age = 42 RETURN { name : name, age : age } * removed configure option `--enable-timings` This option did not have any effect. * removed configure option `--enable-figures` This option previously controlled whether HTTP request statistics code was compiled into ArangoDB or not. The previous default value was `true` so statistics code was available in official packages. Setting the option to `false` led to compile errors so it is doubtful the default value was ever changed. By removing the option some internal statistics code was also simplified. * removed run-time manipulation methods for server endpoints: * `db._removeEndpoint()` * `db._configureEndpoint()` * HTTP POST `/_api/endpoint` * HTTP DELETE `/_api/endpoint` * AQL query result cache The query result cache can optionally cache the complete results of all or selected AQL queries. It can be operated in the following modes: * `off`: the cache is disabled. No query results will be stored * `on`: the cache will store the results of all AQL queries unless their `cache` attribute flag is set to `false` * `demand`: the cache will store the results of AQL queries that have their `cache` attribute set to `true`, but will ignore all others The mode can be set at server startup using the `--database.query-cache-mode` configuration option and later changed at runtime. The following HTTP REST APIs have been added for controlling the query cache: * HTTP GET `/_api/query-cache/properties`: returns the global query cache configuration * HTTP PUT `/_api/query-cache/properties`: modifies the global query cache configuration * HTTP DELETE `/_api/query-cache`: invalidates all results in the query cache The following JavaScript functions have been added for controlling the query cache: * `require("org/arangodb/aql/cache").properties()`: returns the global query cache configuration * `require("org/arangodb/aql/cache").properties(properties)`: modifies the global query cache configuration * `require("org/arangodb/aql/cache").clear()`: invalidates all results in the query cache * do not link arangoimp against V8 * AQL function call arguments optimization This will lead to arguments in function calls inside AQL queries not being copied but passed by reference. This may speed up calls to functions with bigger argument values or queries that call functions a lot of times. * upgraded V8 version to 4.3.61 * removed deprecated AQL `SKIPLIST` function. This function was introduced in older versions of ArangoDB with a less powerful query optimizer to retrieve data from a skiplist index using a `LIMIT` clause. It was marked as deprecated in ArangoDB 2.6. Since ArangoDB 2.3 the behavior of the `SKIPLIST` function can be emulated using regular AQL constructs, e.g. FOR doc IN @@collection FILTER doc.value >= @value SORT doc.value DESC LIMIT 1 RETURN doc * the `skip()` function for simple queries does not accept negative input any longer. This feature was deprecated in 2.6.0. * fix exception handling In some cases JavaScript exceptions would re-throw without information of the original problem. Now the original exception is logged for failure analysis. * based REST API method PUT `/_api/simple/all` on the cursor API and make it use AQL internally. The change speeds up this REST API method and will lead to additional query information being returned by the REST API. Clients can use this extra information or ignore it. * Foxx Queue job success/failure handlers arguments have changed from `(jobId, jobData, result, jobFailures)` to `(result, jobData, job)`. * added Foxx Queue job options `repeatTimes`, `repeatUntil` and `repeatDelay` to automatically re-schedule jobs when they are completed. * added Foxx manifest configuration type `password` to mask values in the web interface. * fixed default values in Foxx manifest configurations sometimes not being used as defaults. * fixed optional parameters in Foxx manifest configurations sometimes not being cleared correctly. * Foxx dependencies can now be marked as optional using a slightly more verbose syntax in your manifest file. * converted Foxx constructors to ES6 classes so you can extend them using class syntax. * updated aqb to 2.0. * updated chai to 3.0. * Use more madvise calls to speed up things when memory is tight, in particular at load time but also for random accesses later. * Overhauled web interface The web interface now has a new design. The API documentation for ArangoDB has been moved from "Tools" to "Links" in the web interface. The "Applications" tab in the web interfaces has been renamed to "Services". v2.6.12 (2015-12-02) -------------------- * fixed disappearing of documents for collections transferred via `sync` if the the collection was dropped right before synchronization and drop and (re-)create collection markers were located in the same WAL file * added missing lock instruction for primary index in compactor size calculation * fixed issue #1589 * fixed issue #1583 * Foxx: optional configuration options no longer log validation errors when assigned empty values (#1495) v2.6.11 (2015-11-18) -------------------- * fixed potentially invalid pointer access in shaper when the currently accessed document got re-located by the WAL collector at the very same time v2.6.10 (2015-11-10) -------------------- * disable replication appliers when starting in modes `--upgrade`, `--no-server` and `--check-upgrade` * more detailed output in arango-dfdb * fixed potential deadlock in collection status changing on Windows * issue #1521: Can't dump/restore with user and password v2.6.9 (2015-09-29) ------------------- * added "special" password ARANGODB_DEFAULT_ROOT_PASSWORD. If you pass ARANGODB_DEFAULT_ROOT_PASSWORD as password, it will read the password from the environment variable ARANGODB_DEFAULT_ROOT_PASSWORD * fixed failing AQL skiplist, sort and limit combination When using a Skiplist index on an attribute (say "a") and then using sort and skip on this attribute caused the result to be empty e.g.: require("internal").db.test.ensureSkiplist("a"); require("internal").db._query("FOR x IN test SORT x.a LIMIT 10, 10"); Was always empty no matter how many documents are stored in test. This is now fixed. v2.6.8 (2015-09-09) ------------------- * ARM only: The ArangoDB packages for ARM require the kernel to allow unaligned memory access. How the kernel handles unaligned memory access is configurable at runtime by checking and adjusting the contents `/proc/cpu/alignment`. In order to operate on ARM, ArangoDB requires the bit 1 to be set. This will make the kernel trap and adjust unaligned memory accesses. If this bit is not set, the kernel may send a SIGBUS signal to ArangoDB and terminate it. To set bit 1 in `/proc/cpu/alignment` use the following command as a privileged user (e.g. root): echo "2" > /proc/cpu/alignment Note that this setting affects all user processes and not just ArangoDB. Setting the alignment with the above command will also not make the setting permanent, so it will be lost after a restart of the system. In order to make the setting permanent, it should be executed during system startup or before starting arangod. The ArangoDB start/stop scripts do not adjust the alignment setting, but rely on the environment to have the correct alignment setting already. The reason for this is that the alignment settings also affect all other user processes (which ArangoDB is not aware of) and thus may have side-effects outside of ArangoDB. It is therefore more reasonable to have the system administrator carry out the change. v2.6.7 (2015-08-25) ------------------- * improved AssocMulti index performance when resizing. This makes the edge index perform less I/O when under memory pressure. v2.6.6 (2015-08-23) ------------------- * added startup option `--server.additional-threads` to create separate queues for slow requests. v2.6.5 (2015-08-17) ------------------- * added startup option `--database.throw-collection-not-loaded-error` Accessing a not-yet loaded collection will automatically load a collection on first access. This flag controls what happens in case an operation would need to wait for another thread to finalize loading a collection. If set to *true*, then the first operation that accesses an unloaded collection will load it. Further threads that try to access the same collection while it is still loading immediately fail with an error (1238, *collection not loaded*). This is to prevent all server threads from being blocked while waiting on the same collection to finish loading. When the first thread has completed loading the collection, the collection becomes regularly available, and all operations from that point on can be carried out normally, and error 1238 will not be thrown anymore for that collection. If set to *false*, the first thread that accesses a not-yet loaded collection will still load it. Other threads that try to access the collection while loading will not fail with error 1238 but instead block until the collection is fully loaded. This configuration might lead to all server threads being blocked because they are all waiting for the same collection to complete loading. Setting the option to *true* will prevent this from happening, but requires clients to catch error 1238 and react on it (maybe by scheduling a retry for later). The default value is *false*. * fixed busy wait loop in scheduler threads that sometimes consumed 100% CPU while waiting for events on connections closed unexpectedly by the client side * handle attribute `indexBuckets` when restoring collections via arangorestore. Previously the `indexBuckets` attribute value from the dump was ignored, and the server default value for `indexBuckets` was used when restoring a collection. * fixed "EscapeValue already set error" crash in V8 actions that might have occurred when canceling V8-based operations. v2.6.4 (2015-08-01) ------------------- * V8: Upgrade to version 4.1.0.27 - this is intended to be the stable V8 version. * fixed issue #1424: Arango shell should not processing arrows pushing on keyboard v2.6.3 (2015-07-21) ------------------- * issue #1409: Document values with null character truncated v2.6.2 (2015-07-04) ------------------- * fixed issue #1383: bindVars for HTTP API doesn't work with empty string * fixed handling of default values in Foxx manifest configurations * fixed handling of optional parameters in Foxx manifest configurations * fixed a reference error being thrown in Foxx queues when a function-based job type is used that is not available and no options object is passed to queue.push v2.6.1 (2015-06-24) ------------------- * Add missing swagger files to cmake build. fixes #1368 * fixed documentation errors v2.6.0 (2015-06-20) ------------------- * using negative values for `SimpleQuery.skip()` is deprecated. This functionality will be removed in future versions of ArangoDB. * The following simple query functions are now deprecated: * collection.near * collection.within * collection.geo * collection.fulltext * collection.range * collection.closedRange This also lead to the following REST API methods being deprecated from now on: * PUT /_api/simple/near * PUT /_api/simple/within * PUT /_api/simple/fulltext * PUT /_api/simple/range It is recommended to replace calls to these functions or APIs with equivalent AQL queries, which are more flexible because they can be combined with other operations: FOR doc IN NEAR(@@collection, @latitude, @longitude, @limit) RETURN doc FOR doc IN WITHIN(@@collection, @latitude, @longitude, @radius, @distanceAttributeName) RETURN doc FOR doc IN FULLTEXT(@@collection, @attributeName, @queryString, @limit) RETURN doc FOR doc IN @@collection FILTER doc.value >= @left && doc.value < @right LIMIT @skip, @limit RETURN doc` The above simple query functions and REST API methods may be removed in future versions of ArangoDB. * deprecated now-obsolete AQL `SKIPLIST` function The function was introduced in older versions of ArangoDB with a less powerful query optimizer to retrieve data from a skiplist index using a `LIMIT` clause. Since 2.3 the same goal can be achieved by using regular AQL constructs, e.g. FOR doc IN collection FILTER doc.value >= @value SORT doc.value DESC LIMIT 1 RETURN doc * fixed issues when switching the database inside tasks and during shutdown of database cursors These features were added during 2.6 alpha stage so the fixes affect devel/2.6-alpha builds only * issue #1360: improved foxx-manager help * added `--enable-tcmalloc` configure option. When this option is set, arangod and the client tools will be linked against tcmalloc, which replaces the system allocator. When the option is set, a tcmalloc library must be present on the system under one of the names `libtcmalloc`, `libtcmalloc_minimal` or `libtcmalloc_debug`. As this is a configure option, it is supported for manual builds on Linux-like systems only. tcmalloc support is currently experimental. * issue #1353: Windows: HTTP API - incorrect path in errorMessage * issue #1347: added option `--create-database` for arangorestore. Setting this option to `true` will now create the target database if it does not exist. When creating the target database, the username and passwords passed to arangorestore will be used to create an initial user for the new database. * issue #1345: advanced debug information for User Functions * issue #1341: Can't use bindvars in UPSERT * fixed vulnerability in JWT implementation. * changed default value of option `--database.ignore-datafile-errors` from `true` to `false` If the new default value of `false` is used, then arangod will refuse loading collections that contain datafiles with CRC mismatches or other errors. A collection with datafile errors will then become unavailable. This prevents follow up errors from happening. The only way to access such collection is to use the datafile debugger (arango-dfdb) and try to repair or truncate the datafile with it. If `--database.ignore-datafile-errors` is set to `true`, then collections will become available even if parts of their data cannot be loaded. This helps availability, but may cause (partial) data loss and follow up errors. * added server startup option `--server.session-timeout` for controlling the timeout of user sessions in the web interface * add sessions and cookie authentication for ArangoDB's web interface ArangoDB's built-in web interface now uses sessions. Session information ids are stored in cookies, so clients using the web interface must accept cookies in order to use it * web interface: display query execution time in AQL editor * web interface: renamed AQL query *submit* button to *execute* * web interface: added query explain feature in AQL editor * web interface: demo page added. only working if demo data is available, hidden otherwise * web interface: added support for custom app scripts with optional arguments and results * web interface: mounted apps that need to be configured are now indicated in the app overview * web interface: added button for running tests to app details * web interface: added button for configuring app dependencies to app details * web interface: upgraded API documentation to use Swagger 2 * INCOMPATIBLE CHANGE removed startup option `--log.severity` The docs for `--log.severity` mentioned lots of severities (e.g. `exception`, `technical`, `functional`, `development`) but only a few severities (e.g. `all`, `human`) were actually used, with `human` being the default and `all` enabling the additional logging of requests. So the option pretended to control a lot of things which it actually didn't. Additionally, the option `--log.requests-file` was around for a long time already, also controlling request logging. Because the `--log.severity` option effectively did not control that much, it was removed. A side effect of removing the option is that 2.5 installations which used `--log.severity all` will not log requests after the upgrade to 2.6. This can be adjusted by setting the `--log.requests-file` option. * add backtrace to fatal log events * added optional `limit` parameter for AQL function `FULLTEXT` * make fulltext index also index text values contained in direct sub-objects of the indexed attribute. Previous versions of ArangoDB only indexed the attribute value if it was a string. Sub-attributes of the index attribute were ignored when fulltext indexing. Now, if the index attribute value is an object, the object's values will each be included in the fulltext index if they are strings. If the index attribute value is an array, the array's values will each be included in the fulltext index if they are strings. For example, with a fulltext index present on the `translations` attribute, the following text values will now be indexed: var c = db._create("example"); c.ensureFulltextIndex("translations"); c.insert({ translations: { en: "fox", de: "Fuchs", fr: "renard", ru: "лиса" } }); c.insert({ translations: "Fox is the English translation of the German word Fuchs" }); c.insert({ translations: [ "ArangoDB", "document", "database", "Foxx" ] }); c.fulltext("translations", "лиса").toArray(); // returns only first document c.fulltext("translations", "Fox").toArray(); // returns first and second documents c.fulltext("translations", "prefix:Fox").toArray(); // returns all three documents * added batch document removal and lookup commands: collection.lookupByKeys(keys) collection.removeByKeys(keys) These commands can be used to perform multi-document lookup and removal operations efficiently from the ArangoShell. The argument to these operations is an array of document keys. Also added HTTP APIs for batch document commands: * PUT /_api/simple/lookup-by-keys * PUT /_api/simple/remove-by-keys * properly prefix document address URLs with the current database name for calls to the REST API method GET `/_api/document?collection=...` (that method will return partial URLs to all documents in the collection). Previous versions of ArangoDB returned the URLs starting with `/_api/` but without the current database name, e.g. `/_api/document/mycollection/mykey`. Starting with 2.6, the response URLs will include the database name as well, e.g. `/_db/_system/_api/document/mycollection/mykey`. * added dedicated collection export HTTP REST API ArangoDB now provides a dedicated collection export API, which can take snapshots of entire collections more efficiently than the general-purpose cursor API. The export API is useful to transfer the contents of an entire collection to a client application. It provides optional filtering on specific attributes. The export API is available at endpoint `POST /_api/export?collection=...`. The API has the same return value structure as the already established cursor API (`POST /_api/cursor`). An introduction to the export API is given in this blog post: http://jsteemann.github.io/blog/2015/04/04/more-efficient-data-exports/ * subquery optimizations for AQL queries This optimization avoids copying intermediate results into subqueries that are not required by the subquery. A brief description can be found here: http://jsteemann.github.io/blog/2015/05/04/subquery-optimizations/ * return value optimization for AQL queries This optimization avoids copying the final query result inside the query's main `ReturnNode`. A brief description can be found here: http://jsteemann.github.io/blog/2015/05/04/return-value-optimization-for-aql/ * speed up AQL queries containing big `IN` lists for index lookups `IN` lists used for index lookups had performance issues in previous versions of ArangoDB. These issues have been addressed in 2.6 so using bigger `IN` lists for filtering is much faster. A brief description can be found here: http://jsteemann.github.io/blog/2015/05/07/in-list-improvements/ * allow `@` and `.` characters in document keys, too This change also leads to document keys being URL-encoded when returned in HTTP `location` response headers. * added alternative implementation for AQL COLLECT The alternative method uses a hash table for grouping and does not require its input elements to be sorted. It will be taken into account by the optimizer for `COLLECT` statements that do not use an `INTO` clause. In case a `COLLECT` statement can use the hash table variant, the optimizer will create an extra plan for it at the beginning of the planning phase. In this plan, no extra `SORT` node will be added in front of the `COLLECT` because the hash table variant of `COLLECT` does not require sorted input. Instead, a `SORT` node will be added after it to sort its output. This `SORT` node may be optimized away again in later stages. If the sort order of the result is irrelevant to the user, adding an extra `SORT null` after a hash `COLLECT` operation will allow the optimizer to remove the sorts altogether. In addition to the hash table variant of `COLLECT`, the optimizer will modify the original plan to use the regular `COLLECT` implementation. As this implementation requires sorted input, the optimizer will insert a `SORT` node in front of the `COLLECT`. This `SORT` node may be optimized away in later stages. The created plans will then be shipped through the regular optimization pipeline. In the end, the optimizer will pick the plan with the lowest estimated total cost as usual. The hash table variant does not require an up-front sort of the input, and will thus be preferred over the regular `COLLECT` if the optimizer estimates many input elements for the `COLLECT` node and cannot use an index to sort them. The optimizer can be explicitly told to use the regular *sorted* variant of `COLLECT` by suffixing a `COLLECT` statement with `OPTIONS { "method" : "sorted" }`. This will override the optimizer guesswork and only produce the *sorted* variant of `COLLECT`. A blog post on the new `COLLECT` implementation can be found here: http://jsteemann.github.io/blog/2015/04/22/collecting-with-a-hash-table/ * refactored HTTP REST API for cursors The HTTP REST API for cursors (`/_api/cursor`) has been refactored to improve its performance and use less memory. A post showing some of the performance improvements can be found here: http://jsteemann.github.io/blog/2015/04/01/improvements-for-the-cursor-api/ * simplified return value syntax for data-modification AQL queries ArangoDB 2.4 since version allows to return results from data-modification AQL queries. The syntax for this was quite limited and verbose: FOR i IN 1..10 INSERT { value: i } IN test LET inserted = NEW RETURN inserted The `LET inserted = NEW RETURN inserted` was required literally to return the inserted documents. No calculations could be made using the inserted documents. This is now more flexible. After a data-modification clause (e.g. `INSERT`, `UPDATE`, `REPLACE`, `REMOVE`, `UPSERT`) there can follow any number of `LET` calculations. These calculations can refer to the pseudo-values `OLD` and `NEW` that are created by the data-modification statements. This allows returning projections of inserted or updated documents, e.g.: FOR i IN 1..10 INSERT { value: i } IN test RETURN { _key: NEW._key, value: i } Still not every construct is allowed after a data-modification clause. For example, no functions can be called that may access documents. More information can be found here: http://jsteemann.github.io/blog/2015/03/27/improvements-for-data-modification-queries/ * added AQL `UPSERT` statement This adds an `UPSERT` statement to AQL that is a combination of both `INSERT` and `UPDATE` / `REPLACE`. The `UPSERT` will search for a matching document using a user-provided example. If no document matches the example, the *insert* part of the `UPSERT` statement will be executed. If there is a match, the *update* / *replace* part will be carried out: UPSERT { page: 'index.html' } /* search example */ INSERT { page: 'index.html', pageViews: 1 } /* insert part */ UPDATE { pageViews: OLD.pageViews + 1 } /* update part */ IN pageViews `UPSERT` can be used with an `UPDATE` or `REPLACE` clause. The `UPDATE` clause will perform a partial update of the found document, whereas the `REPLACE` clause will replace the found document entirely. The `UPDATE` or `REPLACE` parts can refer to the pseudo-value `OLD`, which contains all attributes of the found document. `UPSERT` statements can optionally return values. In the following query, the return attribute `found` will return the found document before the `UPDATE` was applied. If no document was found, `found` will contain a value of `null`. The `updated` result attribute will contain the inserted / updated document: UPSERT { page: 'index.html' } /* search example */ INSERT { page: 'index.html', pageViews: 1 } /* insert part */ UPDATE { pageViews: OLD.pageViews + 1 } /* update part */ IN pageViews RETURN { found: OLD, updated: NEW } A more detailed description of `UPSERT` can be found here: http://jsteemann.github.io/blog/2015/03/27/preview-of-the-upsert-command/ * adjusted default configuration value for `--server.backlog-size` from 10 to 64. * issue #1231: bug xor feature in AQL: LENGTH(null) == 4 This changes the behavior of the AQL `LENGTH` function as follows: - if the single argument to `LENGTH()` is `null`, then the result will now be `0`. In previous versions of ArangoDB, the result of `LENGTH(null)` was `4`. - if the single argument to `LENGTH()` is `true`, then the result will now be `1`. In previous versions of ArangoDB, the result of `LENGTH(true)` was `4`. - if the single argument to `LENGTH()` is `false`, then the result will now be `0`. In previous versions of ArangoDB, the result of `LENGTH(false)` was `5`. The results of `LENGTH()` with string, numeric, array object argument values do not change. * issue #1298: Bulk import if data already exists (#1298) This change extends the HTTP REST API for bulk imports as follows: When documents are imported and the `_key` attribute is specified for them, the import can be used for inserting and updating/replacing documents. Previously, the import could be used for inserting new documents only, and re-inserting a document with an existing key would have failed with a *unique key constraint violated* error. The above behavior is still the default. However, the API now allows controlling the behavior in case of a unique key constraint error via the optional URL parameter `onDuplicate`. This parameter can have one of the following values: - `error`: when a unique key constraint error occurs, do not import or update the document but report an error. This is the default. - `update`: when a unique key constraint error occurs, try to (partially) update the existing document with the data specified in the import. This may still fail if the document would violate secondary unique indexes. Only the attributes present in the import data will be updated and other attributes already present will be preserved. The number of updated documents will be reported in the `updated` attribute of the HTTP API result. - `replace`: when a unique key constraint error occurs, try to fully replace the existing document with the data specified in the import. This may still fail if the document would violate secondary unique indexes. The number of replaced documents will be reported in the `updated` attribute of the HTTP API result. - `ignore`: when a unique key constraint error occurs, ignore this error. There will be no insert, update or replace for the particular document. Ignored documents will be reported separately in the `ignored` attribute of the HTTP API result. The result of the HTTP import API will now contain the attributes `ignored` and `updated`, which contain the number of ignored and updated documents respectively. These attributes will contain a value of zero unless the `onDuplicate` URL parameter is set to either `update` or `replace` (in this case the `updated` attribute may contain non-zero values) or `ignore` (in this case the `ignored` attribute may contain a non-zero value). To support the feature, arangoimp also has a new command line option `--on-duplicate` which can have one of the values `error`, `update`, `replace`, `ignore`. The default value is `error`. A few examples for using arangoimp with the `--on-duplicate` option can be found here: http://jsteemann.github.io/blog/2015/04/14/updating-documents-with-arangoimp/ * changed behavior of `db._query()` in the ArangoShell: if the command's result is printed in the shell, the first 10 results will be printed. Previously only a basic description of the underlying query result cursor was printed. Additionally, if the cursor result contains more than 10 results, the cursor is assigned to a global variable `more`, which can be used to iterate over the cursor result. Example: arangosh [_system]> db._query("FOR i IN 1..15 RETURN i") [object ArangoQueryCursor, count: 15, hasMore: true] [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] type 'more' to show more documents arangosh [_system]> more [object ArangoQueryCursor, count: 15, hasMore: false] [ 11, 12, 13, 14, 15 ] * Disallow batchSize value 0 in HTTP `POST /_api/cursor`: The HTTP REST API `POST /_api/cursor` does not accept a `batchSize` parameter value of `0` any longer. A batch size of 0 never made much sense, but previous versions of ArangoDB did not check for this value. Now creating a cursor using a `batchSize` value 0 will result in an HTTP 400 error response * REST Server: fix memory leaks when failing to add jobs * 'EDGES' AQL Function The AQL function `EDGES` got a new fifth option parameter. Right now only one option is available: 'includeVertices'. This is a boolean parameter that allows to modify the result of the `EDGES` function. Default is 'includeVertices: false' which does not have any effect. 'includeVertices: true' modifies the result, such that {vertex: , edge: } is returned. * INCOMPATIBLE CHANGE: The result format of the AQL function `NEIGHBORS` has been changed. Before it has returned an array of objects containing 'vertex' and 'edge'. Now it will only contain the vertex directly. Also an additional option 'includeData' has been added. This is used to define if only the 'vertex._id' value should be returned (false, default), or if the vertex should be looked up in the collection and the complete JSON should be returned (true). Using only the id values can lead to significantly improved performance if this is the only information required. In order to get the old result format prior to ArangoDB 2.6, please use the function EDGES instead. Edges allows for a new option 'includeVertices' which, set to true, returns exactly the format of NEIGHBORS. Example: NEIGHBORS(, , , , ) This can now be achieved by: EDGES(, , , , {includeVertices: true}) If you are nesting several NEIGHBORS steps you can speed up their performance in the following way: Old Example: FOR va IN NEIGHBORS(Users, relations, 'Users/123', 'outbound') FOR vc IN NEIGHBORS(Products, relations, va.vertex._id, 'outbound') RETURN vc This can now be achieved by: FOR va IN NEIGHBORS(Users, relations, 'Users/123', 'outbound') FOR vc IN NEIGHBORS(Products, relations, va, 'outbound', null, {includeData: true}) RETURN vc ^^^^ ^^^^^^^^^^^^^^^^^^^ Use intermediate directly include Data for final * INCOMPATIBLE CHANGE: The AQL function `GRAPH_NEIGHBORS` now provides an additional option `includeData`. This option allows controlling whether the function should return the complete vertices or just their IDs. Returning only the IDs instead of the full vertices can lead to improved performance . If provided, `includeData` is set to `true`, all vertices in the result will be returned with all their attributes. The default value of `includeData` is `false`. This makes the default function results incompatible with previous versions of ArangoDB. To get the old result style in ArangoDB 2.6, please set the options as follows in calls to `GRAPH_NEIGHBORS`: GRAPH_NEIGHBORS(, , { includeData: true }) * INCOMPATIBLE CHANGE: The AQL function `GRAPH_COMMON_NEIGHBORS` now provides an additional option `includeData`. This option allows controlling whether the function should return the complete vertices or just their IDs. Returning only the IDs instead of the full vertices can lead to improved performance . If provided, `includeData` is set to `true`, all vertices in the result will be returned with all their attributes. The default value of `includeData` is `false`. This makes the default function results incompatible with previous versions of ArangoDB. To get the old result style in ArangoDB 2.6, please set the options as follows in calls to `GRAPH_COMMON_NEIGHBORS`: GRAPH_COMMON_NEIGHBORS(, , , { includeData: true }, { includeData: true }) * INCOMPATIBLE CHANGE: The AQL function `GRAPH_SHORTEST_PATH` now provides an additional option `includeData`. This option allows controlling whether the function should return the complete vertices and edges or just their IDs. Returning only the IDs instead of full vertices and edges can lead to improved performance . If provided, `includeData` is set to `true`, all vertices and edges in the result will be returned with all their attributes. There is also an optional parameter `includePath` of type object. It has two optional sub-attributes `vertices` and `edges`, both of type boolean. Both can be set individually and the result will include all vertices on the path if `includePath.vertices == true` and all edges if `includePath.edges == true` respectively. The default value of `includeData` is `false`, and paths are now excluded by default. This makes the default function results incompatible with previous versions of ArangoDB. To get the old result style in ArangoDB 2.6, please set the options as follows in calls to `GRAPH_SHORTEST_PATH`: GRAPH_SHORTEST_PATH(, , , { includeData: true, includePath: { edges: true, vertices: true } }) The attributes `startVertex` and `vertex` that were present in the results of `GRAPH_SHORTEST_PATH` in previous versions of ArangoDB will not be produced in 2.6. To calculate these attributes in 2.6, please extract the first and last elements from the `vertices` result attribute. * INCOMPATIBLE CHANGE: The AQL function `GRAPH_DISTANCE_TO` will now return only the id the destination vertex in the `vertex` attribute, and not the full vertex data with all vertex attributes. * INCOMPATIBLE CHANGE: All graph measurements functions in JavaScript module `general-graph` that calculated a single figure previously returned an array containing just the figure. Now these functions will return the figure directly and not put it inside an array. The affected functions are: * `graph._absoluteEccentricity` * `graph._eccentricity` * `graph._absoluteCloseness` * `graph._closeness` * `graph._absoluteBetweenness` * `graph._betweenness` * `graph._radius` * `graph._diameter` * Create the `_graphs` collection in new databases with `waitForSync` attribute set to `false` The previous `waitForSync` value was `true`, so default the behavior when creating and dropping graphs via the HTTP REST API changes as follows if the new settings are in effect: * `POST /_api/graph` by default returns `HTTP 202` instead of `HTTP 201` * `DELETE /_api/graph/graph-name` by default returns `HTTP 202` instead of `HTTP 201` If the `_graphs` collection still has its `waitForSync` value set to `true`, then the HTTP status code will not change. * Upgraded ICU to version 54; this increases performance in many places. based on https://code.google.com/p/chromium/issues/detail?id=428145 * added support for HTTP push aka chunked encoding * issue #1051: add info whether server is running in service or user mode? This will add a "mode" attribute to the result of the result of HTTP GET `/_api/version?details=true` "mode" can have the following values: - `standalone`: server was started manually (e.g. on command-line) - `service`: service is running as Windows service, in daemon mode or under the supervisor * improve system error messages in Windows port * increased default value of `--server.request-timeout` from 300 to 1200 seconds for client tools (arangosh, arangoimp, arangodump, arangorestore) * increased default value of `--server.connect-timeout` from 3 to 5 seconds for client tools (arangosh, arangoimp, arangodump, arangorestore) * added startup option `--server.foxx-queues-poll-interval` This startup option controls the frequency with which the Foxx queues manager is checking the queue (or queues) for jobs to be executed. The default value is `1` second. Lowering this value will result in the queue manager waking up and checking the queues more frequently, which may increase CPU usage of the server. When not using Foxx queues, this value can be raised to save some CPU time. * added startup option `--server.foxx-queues` This startup option controls whether the Foxx queue manager will check queue and job entries. Disabling this option can reduce server load but will prevent jobs added to Foxx queues from being processed at all. The default value is `true`, enabling the Foxx queues feature. * make Foxx queues really database-specific. Foxx queues were and are stored in a database-specific collection `_queues`. However, a global cache variable for the queues led to the queue names being treated database-independently, which was wrong. Since 2.6, Foxx queues names are truly database-specific, so the same queue name can be used in two different databases for two different queues. Until then, it is advisable to think of queues as already being database-specific, and using the database name as a queue name prefix to be avoid name conflicts, e.g.: var queueName = "myQueue"; var Foxx = require("org/arangodb/foxx"); Foxx.queues.create(db._name() + ":" + queueName); * added support for Foxx queue job types defined as app scripts. The old job types introduced in 2.4 are still supported but are known to cause issues in 2.5 and later when the server is restarted or the job types are not defined in every thread. The new job types avoid this issue by storing an explicit mount path and script name rather than an assuming the job type is defined globally. It is strongly recommended to convert your job types to the new script-based system. * renamed Foxx sessions option "sessionStorageApp" to "sessionStorage". The option now also accepts session storages directly. * Added the following JavaScript methods for file access: * fs.copyFile() to copy single files * fs.copyRecursive() to copy directory trees * fs.chmod() to set the file permissions (non-Windows only) * Added process.env for accessing the process environment from JavaScript code * Cluster: kickstarter shutdown routines will more precisely follow the shutdown of its nodes. * Cluster: don't delete agency connection objects that are currently in use. * Cluster: improve passing along of HTTP errors * fixed issue #1247: debian init script problems * multi-threaded index creation on collection load When a collection contains more than one secondary index, they can be built in memory in parallel when the collection is loaded. How many threads are used for parallel index creation is determined by the new configuration parameter `--database.index-threads`. If this is set to 0, indexes are built by the opening thread only and sequentially. This is equivalent to the behavior in 2.5 and before. * speed up building up primary index when loading collections * added `count` attribute to `parameters.json` files of collections. This attribute indicates the number of live documents in the collection on unload. It is read when the collection is (re)loaded to determine the initial size for the collection's primary index * removed remainders of MRuby integration, removed arangoirb * simplified `controllers` property in Foxx manifests. You can now specify a filename directly if you only want to use a single file mounted at the base URL of your Foxx app. * simplified `exports` property in Foxx manifests. You can now specify a filename directly if you only want to export variables from a single file in your Foxx app. * added support for node.js-style exports in Foxx exports. Your Foxx exports file can now export arbitrary values using the `module.exports` property instead of adding properties to the `exports` object. * added `scripts` property to Foxx manifests. You should now specify the `setup` and `teardown` files as properties of the `scripts` object in your manifests and can define custom, app-specific scripts that can be executed from the web interface or the CLI. * added `tests` property to Foxx manifests. You can now define test cases using the `mocha` framework which can then be executed inside ArangoDB. * updated `joi` package to 6.0.8. * added `extendible` package. * added Foxx model lifecycle events to repositories. See #1257. * speed up resizing of edge index. * allow to split an edge index into buckets which are resized individually. This is controlled by the `indexBuckets` attribute in the `properties` of the collection. * fix a cluster deadlock bug in larger clusters by marking a thread waiting for a lock on a DBserver as blocked v2.5.7 (2015-08-02) ------------------- * V8: Upgrade to version 4.1.0.27 - this is intended to be the stable V8 version. v2.5.6 (2015-07-21) ------------------- * alter Windows build infrastructure so we can properly store pdb files. * potentially fixed issue #1313: Wrong metric calculation at dashboard Escape whitespace in process name when scanning /proc/pid/stats This fixes statistics values read from that file * Fixed variable naming in AQL `COLLECT INTO` results in case the COLLECT is placed in a subquery which itself is followed by other constructs that require variables v2.5.5 (2015-05-29) ------------------- * fixed vulnerability in JWT implementation. * fixed format string for reading /proc/pid/stat * take into account barriers used in different V8 contexts v2.5.4 (2015-05-14) ------------------- * added startup option `--log.performance`: specifying this option at startup will log performance-related info messages, mainly timings via the regular logging mechanisms * cluster fixes * fix for recursive copy under Windows v2.5.3 (2015-04-29) ------------------- * Fix fs.move to work across filesystem borders; Fixes Foxx app installation problems; issue #1292. * Fix Foxx app install when installed on a different drive on Windows * issue #1322: strange AQL result * issue #1318: Inconsistent db._create() syntax * issue #1315: queries to a collection fail with an empty response if the collection contains specific JSON data * issue #1300: Make arangodump not fail if target directory exists but is empty * allow specifying higher values than SOMAXCONN for `--server.backlog-size` Previously, arangod would not start when a `--server.backlog-size` value was specified that was higher than the platform's SOMAXCONN header value. Now, arangod will use the user-provided value for `--server.backlog-size` and pass it to the listen system call even if the value is higher than SOMAXCONN. If the user-provided value is higher than SOMAXCONN, arangod will log a warning on startup. * Fixed a cluster deadlock bug. Mark a thread that is in a RemoteBlock as blocked to allow for additional dispatcher threads to be started. * Fix locking in cluster by using another ReadWriteLock class for collections. * Add a second DispatcherQueue for AQL in the cluster. This fixes a cluster-AQL thread explosion bug. v2.5.2 (2015-04-11) ------------------- * modules stored in _modules are automatically flushed when changed * added missing query-id parameter in documentation of HTTP DELETE `/_api/query` endpoint * added iterator for edge index in AQL queries this change may lead to less edges being read when used together with a LIMIT clause * make graph viewer in web interface issue less expensive queries for determining a random vertex from the graph, and for determining vertex attributes * issue #1285: syntax error, unexpected $undefined near '@_to RETURN obj this allows AQL bind parameter names to also start with underscores * moved /_api/query to C++ * issue #1289: Foxx models created from database documents expose an internal method * added `Foxx.Repository#exists` * parallelize initialization of V8 context in multiple threads * fixed a possible crash when the debug-level was TRACE * cluster: do not initialize statistics collection on each coordinator, this fixes a race condition at startup * cluster: fix a startup race w.r.t. the _configuration collection * search for db:// JavaScript modules only after all local files have been considered, this speeds up the require command in a cluster considerably * general cluster speedup in certain areas v2.5.1 (2015-03-19) ------------------- * fixed bug that caused undefined behavior when an AQL query was killed inside a calculation block * fixed memleaks in AQL query cleanup in case out-of-memory errors are thrown * by default, Debian and RedHat packages are built with debug symbols * added option `--database.ignore-logfile-errors` This option controls how collection datafiles with a CRC mismatch are treated. If set to `false`, CRC mismatch errors in collection datafiles will lead to a collection not being loaded at all. If a collection needs to be loaded during WAL recovery, the WAL recovery will also abort (if not forced with `--wal.ignore-recovery-errors true`). Setting this flag to `false` protects users from unintentionally using a collection with corrupted datafiles, from which only a subset of the original data can be recovered. If set to `true`, CRC mismatch errors in collection datafiles will lead to the datafile being partially loaded. All data up to until the mismatch will be loaded. This will enable users to continue with collection datafiles that are corrupted, but will result in only a partial load of the data. The WAL recovery will still abort when encountering a collection with a corrupted datafile, at least if `--wal.ignore-recovery-errors` is not set to `true`. The default value is *true*, so for collections with corrupted datafiles there might be partial data loads once the WAL recovery has finished. If the WAL recovery will need to load a collection with a corrupted datafile, it will still stop when using the default values. * INCOMPATIBLE CHANGE: make the arangod server refuse to start if during startup it finds a non-readable `parameter.json` file for a database or a collection. Stopping the startup process in this case requires manual intervention (fixing the unreadable files), but prevents follow-up errors due to ignored databases or collections from happening. * datafiles and `parameter.json` files written by arangod are now created with read and write privileges for the arangod process user, and with read and write privileges for the arangod process group. Previously, these files were created with user read and write permissions only. * INCOMPATIBLE CHANGE: abort WAL recovery if one of the collection's datafiles cannot be opened * INCOMPATIBLE CHANGE: never try to raise the privileges after dropping them, this can lead to a race condition while running the recovery If you require to run ArangoDB on a port lower than 1024, you must run ArangoDB as root. * fixed inefficiencies in `remove` methods of general-graph module * added option `--database.slow-query-threshold` for controlling the default AQL slow query threshold value on server start * add system error strings for Windows on many places * rework service startup so we announce 'RUNNING' only when we're finished starting. * use the Windows eventlog for FATAL and ERROR - log messages * fix service handling in NSIS Windows installer, specify human readable name * add the ICU_DATA environment variable to the fatal error messages * fixed issue #1265: arangod crashed with SIGSEGV * fixed issue #1241: Wildcards in examples v2.5.0 (2015-03-09) ------------------- * installer fixes for Windows * fix for downloading Foxx * fixed issue #1258: http pipelining not working? v2.5.0-beta4 (2015-03-05) ------------------------- * fixed issue #1247: debian init script problems v2.5.0-beta3 (2015-02-27) ------------------------- * fix Windows install path calculation in arango * fix Windows logging of long strings * fix possible undefinedness of const strings in Windows v2.5.0-beta2 (2015-02-23) ------------------------- * fixed issue #1256: agency binary not found #1256 * fixed issue #1230: API: document/col-name/_key and cursor return different floats * front-end: dashboard tries not to (re)load statistics if user has no access * V8: Upgrade to version 3.31.74.1 * etcd: Upgrade to version 2.0 - This requires go 1.3 to compile at least. * refuse to startup if ICU wasn't initialized, this will i.e. prevent errors from being printed, and libraries from being loaded. * front-end: unwanted removal of index table header after creating new index * fixed issue #1248: chrome: applications filtering not working * fixed issue #1198: queries remain in aql editor (front-end) if you navigate through different tabs * Simplify usage of Foxx Thanks to our user feedback we learned that Foxx is a powerful, yet rather complicated concept. With this release we tried to make it less complicated while keeping all its strength. That includes a rewrite of the documentation as well as some code changes as listed below: * Moved Foxx applications to a different folder. The naming convention now is: /_db///APP Before it was: /databases//: This caused some trouble as apps where cached based on name and version and updates did not apply. Hence the path on filesystem and the app's access URL had no relation to one another. Now the path on filesystem is identical to the URL (except for slashes and the appended APP) * Rewrite of Foxx routing The routing of Foxx has been exposed to major internal changes we adjusted because of user feedback. This allows us to set the development mode per mount point without having to change paths and hold apps at separate locations. * Foxx Development mode The development mode used until 2.4 is gone. It has been replaced by a much more mature version. This includes the deprecation of the javascript.dev-app-path parameter, which is useless since 2.5. Instead of having two separate app directories for production and development, apps now reside in one place, which is used for production as well as for development. Apps can still be put into development mode, changing their behavior compared to production mode. Development mode apps are still reread from disk at every request, and still they ship more debug output. This change has also made the startup options `--javascript.frontend-development-mode` and `--javascript.dev-app-path` obsolete. The former option will not have any effect when set, and the latter option is only read and used during the upgrade to 2.5 and does not have any effects later. * Foxx install process Installing Foxx apps has been a two step process: import them into ArangoDB and mount them at a specific mount point. These operations have been joined together. You can install an app at one mount point, that's it. No fetch, mount, unmount, purge cycle anymore. The commands have been simplified to just: * install: get your Foxx app up and running * uninstall: shut it down and erase it from disk * Foxx error output Until 2.4 the errors produced by Foxx were not optimal. Often, the error message was just `unable to parse manifest` and contained only an internal stack trace. In 2.5 we made major improvements there, including a much more fine-grained error output that helps you debug your Foxx apps. The error message printed is now much closer to its source and should help you track it down. Also we added the default handlers for unhandled errors in Foxx apps: * You will get a nice internal error page whenever your Foxx app is called but was not installed due to any error * You will get a proper error message when having an uncaught error appears in any app route In production mode the messages above will NOT contain any information about your Foxx internals and are safe to be exposed to third party users. In development mode the messages above will contain the stacktrace (if available), making it easier for your in-house devs to track down errors in the application. * added `console` object to Foxx apps. All Foxx apps now have a console object implementing the familiar Console API in their global scope, which can be used to log diagnostic messages to the database. * added `org/arangodb/request` module, which provides a simple API for making HTTP requests to external services. * added optimizer rule `propagate-constant-attributes` This rule will look inside `FILTER` conditions for constant value equality comparisons, and insert the constant values in other places in `FILTER`s. For example, the rule will insert `42` instead of `i.value` in the second `FILTER` of the following query: FOR i IN c1 FOR j IN c2 FILTER i.value == 42 FILTER j.value == i.value RETURN 1 * added `filtered` value to AQL query execution statistics This value indicates how many documents were filtered by `FilterNode`s in the AQL query. Note that `IndexRangeNode`s can also filter documents by selecting only the required ranges from the index. The `filtered` value will not include the work done by `IndexRangeNode`s, but only the work performed by `FilterNode`s. * added support for sparse hash and skiplist indexes Hash and skiplist indexes can optionally be made sparse. Sparse indexes exclude documents in which at least one of the index attributes is either not set or has a value of `null`. As such documents are excluded from sparse indexes, they may contain fewer documents than their non-sparse counterparts. This enables faster indexing and can lead to reduced memory usage in case the indexed attribute does occur only in some, but not all documents of the collection. Sparse indexes will also reduce the number of collisions in non-unique hash indexes in case non-existing or optional attributes are indexed. In order to create a sparse index, an object with the attribute `sparse` can be added to the index creation commands: db.collection.ensureHashIndex(attributeName, { sparse: true }); db.collection.ensureHashIndex(attributeName1, attributeName2, { sparse: true }); db.collection.ensureUniqueConstraint(attributeName, { sparse: true }); db.collection.ensureUniqueConstraint(attributeName1, attributeName2, { sparse: true }); db.collection.ensureSkiplist(attributeName, { sparse: true }); db.collection.ensureSkiplist(attributeName1, attributeName2, { sparse: true }); db.collection.ensureUniqueSkiplist(attributeName, { sparse: true }); db.collection.ensureUniqueSkiplist(attributeName1, attributeName2, { sparse: true }); Note that in place of the above specialized index creation commands, it is recommended to use the more general index creation command `ensureIndex`: ```js db.collection.ensureIndex({ type: "hash", sparse: true, unique: true, fields: [ attributeName ] }); db.collection.ensureIndex({ type: "skiplist", sparse: false, unique: false, fields: [ "a", "b" ] }); ``` When not explicitly set, the `sparse` attribute defaults to `false` for new indexes. This causes a change in behavior when creating a unique hash index without specifying the sparse flag: in 2.4, unique hash indexes were implicitly sparse, always excluding `null` values. There was no option to control this behavior, and sparsity was neither supported for non-unique hash indexes nor skiplists in 2.4. This implicit sparsity of unique hash indexes was considered an inconsistency, and therefore the behavior was cleaned up in 2.5. As of 2.5, indexes will only be created sparse if sparsity is explicitly requested. Existing unique hash indexes from 2.4 or before will automatically be migrated so they are still sparse after the upgrade to 2.5. Geo indexes are implicitly sparse, meaning documents without the indexed location attribute or containing invalid location coordinate values will be excluded from the index automatically. This is also a change when compared to pre-2.5 behavior, when documents with missing or invalid coordinate values may have caused errors on insertion when the geo index' `unique` flag was set and its `ignoreNull` flag was not. This was confusing and has been rectified in 2.5. The method `ensureGeoConstaint()` now does the same as `ensureGeoIndex()`. Furthermore, the attributes `constraint`, `unique`, `ignoreNull` and `sparse` flags are now completely ignored when creating geo indexes. The same is true for fulltext indexes. There is no need to specify non-uniqueness or sparsity for geo or fulltext indexes. They will always be non-unique and sparse. As sparse indexes may exclude some documents, they cannot be used for every type of query. Sparse hash indexes cannot be used to find documents for which at least one of the indexed attributes has a value of `null`. For example, the following AQL query cannot use a sparse index, even if one was created on attribute `attr`: FOR doc In collection FILTER doc.attr == null RETURN doc If the lookup value is non-constant, a sparse index may or may not be used, depending on the other types of conditions in the query. If the optimizer can safely determine that the lookup value cannot be `null`, a sparse index may be used. When uncertain, the optimizer will not make use of a sparse index in a query in order to produce correct results. For example, the following queries cannot use a sparse index on `attr` because the optimizer will not know beforehand whether the comparison values for `doc.attr` will include `null`: FOR doc In collection FILTER doc.attr == SOME_FUNCTION(...) RETURN doc FOR other IN otherCollection FOR doc In collection FILTER doc.attr == other.attr RETURN doc Sparse skiplist indexes can be used for sorting if the optimizer can safely detect that the index range does not include `null` for any of the index attributes. * inspection of AQL data-modification queries will now detect if the data-modification part of the query can run in lockstep with the data retrieval part of the query, or if the data retrieval part must be executed before the data modification can start. Executing the two in lockstep allows using much smaller buffers for intermediate results and starts the actual data-modification operations much earlier than if the two phases were executed separately. * Allow dynamic attribute names in AQL object literals This allows using arbitrary expressions to construct attribute names in object literals specified in AQL queries. To disambiguate expressions and other unquoted attribute names, dynamic attribute names need to be enclosed in brackets (`[` and `]`). Example: FOR i IN 1..100 RETURN { [ CONCAT('value-of-', i) ] : i } * make AQL optimizer rule "use-index-for-sort" remove sort also in case a non-sorted index (e.g. a hash index) is used for only equality lookups and all sort attributes are covered by the index. Example that does not require an extra sort (needs hash index on `value`): FOR doc IN collection FILTER doc.value == 1 SORT doc.value RETURN doc Another example that does not require an extra sort (with hash index on `value1`, `value2`): FOR doc IN collection FILTER doc.value1 == 1 && doc.value2 == 2 SORT doc.value1, doc.value2 RETURN doc * make AQL optimizer rule "use-index-for-sort" remove sort also in case the sort criteria excludes the left-most index attributes, but the left-most index attributes are used by the index for equality-only lookups. Example that can use the index for sorting (needs skiplist index on `value1`, `value2`): FOR doc IN collection FILTER doc.value1 == 1 SORT doc.value2 RETURN doc * added selectivity estimates for primary index, edge index, and hash index The selectivity estimates are returned by the `GET /_api/index` REST API method in a sub-attribute `selectivityEstimate` for each index that supports it. This attribute will be omitted for indexes that do not provide selectivity estimates. If provided, the selectivity estimate will be a numeric value between 0 and 1. Selectivity estimates will also be reported in the result of `collection.getIndexes()` for all indexes that support this. If no selectivity estimate can be determined for an index, the attribute `selectivityEstimate` will be omitted here, too. The web interface also shows selectivity estimates for each index that supports this. Currently the following index types can provide selectivity estimates: - primary index - edge index - hash index (unique and non-unique) No selectivity estimates will be provided when running in cluster mode. * fixed issue #1226: arangod log issues * added additional logger if arangod is started in foreground mode on a tty * added AQL optimizer rule "move-calculations-down" * use exclusive native SRWLocks on Windows instead of native mutexes * added AQL functions `MD5`, `SHA1`, and `RANDOM_TOKEN`. * reduced number of string allocations when parsing certain AQL queries parsing numbers (integers or doubles) does not require a string allocation per number anymore * RequestContext#bodyParam now accepts arbitrary joi schemas and rejects invalid (but well-formed) request bodies. * enforce that AQL user functions are wrapped inside JavaScript function () declarations AQL user functions were always expected to be wrapped inside a JavaScript function, but previously this was not enforced when registering a user function. Enforcing the AQL user functions to be contained inside functions prevents functions from doing some unexpected things that may have led to undefined behavior. * Windows service uninstalling: only remove service if it points to the currently running binary, or --force was specified. * Windows (debug only): print stacktraces on crash and run minidump * Windows (cygwin): if you run arangosh in a cygwin shell or via ssh we will detect this and use the appropriate output functions. * Windows: improve process management * fix IPv6 reverse ip lookups - so far we only did IPv4 addresses. * improve join documentation, add outer join example * run jslint for unit tests too, to prevent "memory leaks" by global js objects with native code. * fix error logging for exceptions - we wouldn't log the exception message itself so far. * improve error reporting in the http client (Windows & *nix) * improve error reports in cluster * Standard errors can now contain custom messages. v2.4.7 (XXXX-XX-XX) ------------------- * fixed issue #1282: Geo WITHIN_RECTANGLE for nested lat/lng v2.4.6 (2015-03-18) ------------------- * added option `--database.ignore-logfile-errors` This option controls how collection datafiles with a CRC mismatch are treated. If set to `false`, CRC mismatch errors in collection datafiles will lead to a collection not being loaded at all. If a collection needs to be loaded during WAL recovery, the WAL recovery will also abort (if not forced with `--wal.ignore-recovery-errors true`). Setting this flag to `false` protects users from unintentionally using a collection with corrupted datafiles, from which only a subset of the original data can be recovered. If set to `true`, CRC mismatch errors in collection datafiles will lead to the datafile being partially loaded. All data up to until the mismatch will be loaded. This will enable users to continue with a collection datafiles that are corrupted, but will result in only a partial load of the data. The WAL recovery will still abort when encountering a collection with a corrupted datafile, at least if `--wal.ignore-recovery-errors` is not set to `true`. The default value is *true*, so for collections with corrupted datafiles there might be partial data loads once the WAL recovery has finished. If the WAL recovery will need to load a collection with a corrupted datafile, it will still stop when using the default values. * INCOMPATIBLE CHANGE: make the arangod server refuse to start if during startup it finds a non-readable `parameter.json` file for a database or a collection. Stopping the startup process in this case requires manual intervention (fixing the unreadable files), but prevents follow-up errors due to ignored databases or collections from happening. * datafiles and `parameter.json` files written by arangod are now created with read and write privileges for the arangod process user, and with read and write privileges for the arangod process group. Previously, these files were created with user read and write permissions only. * INCOMPATIBLE CHANGE: abort WAL recovery if one of the collection's datafiles cannot be opened * INCOMPATIBLE CHANGE: never try to raise the privileges after dropping them, this can lead to a race condition while running the recovery If you require to run ArangoDB on a port lower than 1024, you must run ArangoDB as root. * fixed inefficiencies in `remove` methods of general-graph module * added option `--database.slow-query-threshold` for controlling the default AQL slow query threshold value on server start v2.4.5 (2015-03-16) ------------------- * added elapsed time to HTTP request logging output (`--log.requests-file`) * added AQL current and slow query tracking, killing of AQL queries This change enables retrieving the list of currently running AQL queries inside the selected database. AQL queries with an execution time beyond a certain threshold can be moved to a "slow query" facility and retrieved from there. Queries can also be killed by specifying the query id. This change adds the following HTTP REST APIs: - `GET /_api/query/current`: for retrieving the list of currently running queries - `GET /_api/query/slow`: for retrieving the list of slow queries - `DELETE /_api/query/slow`: for clearing the list of slow queries - `GET /_api/query/properties`: for retrieving the properties for query tracking - `PUT /_api/query/properties`: for adjusting the properties for query tracking - `DELETE /_api/query/`: for killing an AQL query The following JavaScript APIs have been added: - require("org/arangodb/aql/queries").current(); - require("org/arangodb/aql/queries").slow(); - require("org/arangodb/aql/queries").clearSlow(); - require("org/arangodb/aql/queries").properties(); - require("org/arangodb/aql/queries").kill(); * fixed issue #1265: arangod crashed with SIGSEGV * fixed issue #1241: Wildcards in examples * fixed comment parsing in Foxx controllers v2.4.4 (2015-02-24) ------------------- * fixed the generation template for foxx apps. It now does not create deprecated functions anymore * add custom visitor functionality for `GRAPH_NEIGHBORS` function, too * increased default value of traversal option *maxIterations* to 100 times of its previous default value v2.4.3 (2015-02-06) ------------------- * fix multi-threading with openssl when running under Windows * fix timeout on socket operations when running under Windows * Fixed an error in Foxx routing which caused some apps that worked in 2.4.1 to fail with status 500: `undefined is not a function` errors in 2.4.2 This error was occurring due to seldom internal rerouting introduced by the malformed application handler. v2.4.2 (2015-01-30) ------------------- * added custom visitor functionality for AQL traversals This allows more complex result processing in traversals triggered by AQL. A few examples are shown in [this article](http://jsteemann.github.io/blog/2015/01/28/using-custom-visitors-in-aql-graph-traversals/). * improved number of results estimated for nodes of type EnumerateListNode and SubqueryNode in AQL explain output * added AQL explain helper to explain arbitrary AQL queries The helper function prints the query execution plan and the indexes to be used in the query. It can be invoked from the ArangoShell or the web interface as follows: require("org/arangodb/aql/explainer").explain(query); * enable use of indexes for certain AQL conditions with non-equality predicates, in case the condition(s) also refer to indexed attributes The following queries will now be able to use indexes: FILTER a.indexed == ... && a.indexed != ... FILTER a.indexed == ... && a.nonIndexed != ... FILTER a.indexed == ... && ! (a.indexed == ...) FILTER a.indexed == ... && ! (a.nonIndexed == ...) FILTER a.indexed == ... && ! (a.indexed != ...) FILTER a.indexed == ... && ! (a.nonIndexed != ...) FILTER (a.indexed == ... && a.nonIndexed == ...) || (a.indexed == ... && a.nonIndexed == ...) FILTER (a.indexed == ... && a.nonIndexed != ...) || (a.indexed == ... && a.nonIndexed != ...) * Fixed spuriously occurring "collection not found" errors when running queries on local collections on a cluster DB server * Fixed upload of Foxx applications to the server for apps exceeding approx. 1 MB zipped. * Malformed Foxx applications will now return a more useful error when any route is requested. In Production a Foxx app mounted on /app will display an html page on /app/* stating a 503 Service temporarily not available. It will not state any information about your Application. Before it was a 404 Not Found without any information and not distinguishable from a correct not found on your route. In Development Mode the html page also contains information about the error occurred. * Unhandled errors thrown in Foxx routes are now handled by the Foxx framework itself. In Production the route will return a status 500 with a body {error: "Error statement"}. In Development the route will return a status 500 with a body {error: "Error statement", stack: "..."} Before, it was status 500 with a plain text stack including ArangoDB internal routing information. * The Applications tab in web interface will now request development apps more often. So if you have a fixed a syntax error in your app it should always be visible after reload. v2.4.1 (2015-01-19) ------------------- * improved WAL recovery output * fixed certain OR optimizations in AQL optimizer * better diagnostics for arangoimp * fixed invalid result of HTTP REST API method `/_admin/foxx/rescan` * fixed possible segmentation fault when passing a Buffer object into a V8 function as a parameter * updated AQB module to 1.8.0. v2.4.0 (2015-01-13) ------------------- * updated AQB module to 1.7.0. * fixed V8 integration-related crashes * make `fs.move(src, dest)` also fail when both `src` and `dest` are existing directories. This ensures the same behavior of the move operation on different platforms. * fixed AQL insert operation for multi-shard collections in cluster * added optional return value for AQL data-modification queries. This allows returning the documents inserted, removed or updated with the query, e.g. FOR doc IN docs REMOVE doc._key IN docs LET removed = OLD RETURN removed FOR doc IN docs INSERT { } IN docs LET inserted = NEW RETURN inserted FOR doc IN docs UPDATE doc._key WITH { } IN docs LET previous = OLD RETURN previous FOR doc IN docs UPDATE doc._key WITH { } IN docs LET updated = NEW RETURN updated The variables `OLD` and `NEW` are automatically available when a `REMOVE`, `INSERT`, `UPDATE` or `REPLACE` statement is immediately followed by a `LET` statement. Note that the `LET` and `RETURN` statements in data-modification queries are not as flexible as the general versions of `LET` and `RETURN`. When returning documents from data-modification operations, only a single variable can be assigned using `LET`, and the assignment can only be either `OLD` or `NEW`, but not an arbitrary expression. The `RETURN` statement also allows using the just-created variable only, and no arbitrary expressions. v2.4.0-beta1 (2014-12-26) -------------------------- * fixed superstates in FoxxGenerator * fixed issue #1065: Aardvark: added creation of documents and edges with _key property * fixed issue #1198: Aardvark: current AQL editor query is now cached * Upgraded V8 version from 3.16.14 to 3.29.59 The built-in version of V8 has been upgraded from 3.16.14 to 3.29.59. This activates several ES6 (also dubbed *Harmony* or *ES.next*) features in ArangoDB, both in the ArangoShell and the ArangoDB server. They can be used for scripting and in server-side actions such as Foxx routes, traversals etc. The following ES6 features are available in ArangoDB 2.4 by default: * iterators * the `of` operator * symbols * predefined collections types (Map, Set etc.) * typed arrays Many other ES6 features are disabled by default, but can be made available by starting arangod or arangosh with the appropriate options: * arrow functions * proxies * generators * String, Array, and Number enhancements * constants * enhanced object and numeric literals To activate all these ES6 features in arangod or arangosh, start it with the following options: arangosh --javascript.v8-options="--harmony --harmony_generators" More details on the available ES6 features can be found in [this blog](https://jsteemann.github.io/blog/2014/12/19/using-es6-features-in-arangodb/). * Added Foxx generator for building Hypermedia APIs A more detailed description is [here](https://www.arangodb.com/2014/12/08/building-hypermedia-apis-foxxgenerator) * New `Applications` tab in web interface: The `applications` tab got a complete redesign. It will now only show applications that are currently running on ArangoDB. For a selected application, a new detailed view has been created. This view provides a better overview of the app: * author * license * version * contributors * download links * API documentation To install a new application, a new dialog is now available. It provides the features already available in the console application `foxx-manager` plus some more: * install an application from Github * install an application from a zip file * install an application from ArangoDB's application store * create a new application from scratch: this feature uses a generator to create a Foxx application with pre-defined CRUD methods for a given list of collections. The generated Foxx app can either be downloaded as a zip file or be installed on the server. Starting with a new Foxx app has never been easier. * fixed issue #1102: Aardvark: Layout bug in documents overview The documents overview was entirely destroyed in some situations on Firefox. We replaced the plugin we used there. * fixed issue #1168: Aardvark: pagination buttons jumping * fixed issue #1161: Aardvark: Click on Import JSON imports previously uploaded file * removed configure options `--enable-all-in-one-v8`, `--enable-all-in-one-icu`, and `--enable-all-in-one-libev`. * global internal rename to fix naming incompatibilities with JSON: Internal functions with names containing `array` have been renamed to `object`, internal functions with names containing `list` have been renamed to `array`. The renaming was mainly done in the C++ parts. The documentation has also been adjusted so that the correct JSON type names are used in most places. The change also led to the addition of a few function aliases in AQL: * `TO_LIST` now is an alias of the new `TO_ARRAY` * `IS_LIST` now is an alias of the new `IS_ARRAY` * `IS_DOCUMENT` now is an alias of the new `IS_OBJECT` The changed also renamed the option `mergeArrays` to `mergeObjects` for AQL data-modification query options and HTTP document modification API * AQL: added optimizer rule "remove-filter-covered-by-index" This rule removes FilterNodes and CalculationNodes from an execution plan if the filter is already covered by a previous IndexRangeNode. Removing the CalculationNode and the FilterNode will speed up query execution because the query requires less computation. * AQL: added optimizer rule "remove-sort-rand" This rule removes a `SORT RAND()` expression from a query and moves the random iteration into the appropriate `EnumerateCollectionNode`. This is more efficient than individually enumerating and then sorting randomly. * AQL: range optimizations for IN and OR This change enables usage of indexes for several additional cases. Filters containing the `IN` operator can now make use of indexes, and multiple OR- or AND-combined filter conditions can now also use indexes if the filters are accessing the same indexed attribute. Here are a few examples of queries that can now use indexes but couldn't before: FOR doc IN collection FILTER doc.indexedAttribute == 1 || doc.indexedAttribute > 99 RETURN doc FOR doc IN collection FILTER doc.indexedAttribute IN [ 3, 42 ] || doc.indexedAttribute > 99 RETURN doc FOR doc IN collection FILTER (doc.indexedAttribute > 2 && doc.indexedAttribute < 10) || (doc.indexedAttribute > 23 && doc.indexedAttribute < 42) RETURN doc * fixed issue #500: AQL parentheses issue This change allows passing subqueries as AQL function parameters without using duplicate brackets (e.g. `FUNC(query)` instead of `FUNC((query))` * added optional `COUNT` clause to AQL `COLLECT` This allows more efficient group count calculation queries, e.g. FOR doc IN collection COLLECT age = doc.age WITH COUNT INTO length RETURN { age: age, count: length } A count-only query is also possible: FOR doc IN collection COLLECT WITH COUNT INTO length RETURN length * fixed missing makeDirectory when fetching a Foxx application from a zip file * fixed issue #1134: Change the default endpoint to localhost This change will modify the IP address ArangoDB listens on to 127.0.0.1 by default. This will make new ArangoDB installations unaccessible from clients other than localhost unless changed. This is a security feature. To make ArangoDB accessible from any client, change the server's configuration (`--server.endpoint`) to either `tcp://0.0.0.0:8529` or the server's publicly visible IP address. * deprecated `Repository#modelPrototype`. Use `Repository#model` instead. * IMPORTANT CHANGE: by default, system collections are included in replication and all replication API return values. This will lead to user accounts and credentials data being replicated from master to slave servers. This may overwrite slave-specific database users. If this is undesired, the `_users` collection can be excluded from replication easily by setting the `includeSystem` attribute to `false` in the following commands: * replication.sync({ includeSystem: false }); * replication.applier.properties({ includeSystem: false }); This will exclude all system collections (including `_aqlfunctions`, `_graphs` etc.) from the initial synchronization and the continuous replication. If this is also undesired, it is also possible to specify a list of collections to exclude from the initial synchronization and the continuous replication using the `restrictCollections` attribute, e.g.: replication.applier.properties({ includeSystem: true, restrictType: "exclude", restrictCollections: [ "_users", "_graphs", "foo" ] }); The HTTP API methods for fetching the replication inventory and for dumping collections also support the `includeSystem` control flag via a URL parameter. * removed DEPRECATED replication methods: * `replication.logger.start()` * `replication.logger.stop()` * `replication.logger.properties()` * HTTP PUT `/_api/replication/logger-start` * HTTP PUT `/_api/replication/logger-stop` * HTTP GET `/_api/replication/logger-config` * HTTP PUT `/_api/replication/logger-config` * fixed issue #1174, which was due to locking problems in distributed AQL execution * improved cluster locking for AQL avoiding deadlocks * use DistributeNode for modifying queries with REPLACE and UPDATE, if possible v2.3.6 (2015-XX-XX) ------------------- * fixed AQL subquery optimization that produced wrong result when multiple subqueries directly followed each other and and a directly following `LET` statement did refer to any but the first subquery. v2.3.5 (2015-01-16) ------------------- * fixed intermittent 404 errors in Foxx apps after mounting or unmounting apps * fixed issue #1200: Expansion operator results in "Cannot call method 'forEach' of null" * fixed issue #1199: Cannot unlink root node of plan v2.3.4 (2014-12-23) ------------------- * fixed cerberus path for MyArangoDB v2.3.3 (2014-12-17) ------------------- * fixed error handling in instantiation of distributed AQL queries, this also fixes a bug in cluster startup with many servers * issue #1185: parse non-fractional JSON numbers with exponent (e.g. `4e-261`) * issue #1159: allow --server.request-timeout and --server.connect-timeout of 0 v2.3.2 (2014-12-09) ------------------- * fixed issue #1177: Fix bug in the user app's storage * fixed issue #1173: AQL Editor "Save current query" resets user password * fixed missing makeDirectory when fetching a Foxx application from a zip file * put in warning about default changed: fixed issue #1134: Change the default endpoint to localhost * fixed issue #1163: invalid fullCount value returned from AQL * fixed range operator precedence * limit default maximum number of plans created by AQL optimizer to 256 (from 1024) * make AQL optimizer not generate an extra plan if an index can be used, but modify existing plans in place * fixed AQL cursor ttl (time-to-live) issue Any user-specified cursor ttl value was not honored since 2.3.0. * fixed segfault in AQL query hash index setup with unknown shapes * fixed memleaks * added AQL optimizer rule for removing `INTO` from a `COLLECT` statement if not needed * fixed issue #1131 This change provides the `KEEP` clause for `COLLECT ... INTO`. The `KEEP` clause allows controlling which variables will be kept in the variable created by `INTO`. * fixed issue #1147, must protect dispatcher ID for etcd v2.3.1 (2014-11-28) ------------------- * recreate password if missing during upgrade * fixed issue #1126 * fixed non-working subquery index optimizations * do not restrict summary of Foxx applications to 60 characters * fixed display of "required" path parameters in Foxx application documentation * added more optimizations of constants values in AQL FILTER conditions * fixed invalid or-to-in optimization for FILTERs containing comparisons with boolean values * fixed replication of `_graphs` collection * added AQL list functions `PUSH`, `POP`, `UNSHIFT`, `SHIFT`, `REMOVE_VALUES`, `REMOVE_VALUE`, `REMOVE_NTH` and `APPEND` * added AQL functions `CALL` and `APPLY` to dynamically call other functions * fixed AQL optimizer cost estimation for LIMIT node * prevent Foxx queues from permanently writing to the journal even when server is idle * fixed AQL COLLECT statement with INTO clause, which copied more variables than v2.2 and thus lead to too much memory consumption. This deals with #1107. * fixed AQL COLLECT statement, this concerned every COLLECT statement, only the first group had access to the values of the variables before the COLLECT statement. This deals with #1127. * fixed some AQL internals, where sometimes too many items were fetched from upstream in the presence of a LIMIT clause. This should generally improve performance. v2.3.0 (2014-11-18) ------------------- * fixed syslog flags. `--log.syslog` is deprecated and setting it has no effect, `--log.facility` now works as described. Application name has been changed from `triagens` to `arangod`. It can be changed using `--log.application`. The syslog will only contain the actual log message. The datetime prefix is omitted. * fixed deflate in SimpleHttpClient * fixed issue #1104: edgeExamples broken or changed * fixed issue #1103: Error while importing user queries * fixed issue #1100: AQL: HAS() fails on doc[attribute_name] * fixed issue #1098: runtime error when creating graph vertex * hide system applications in **Applications** tab by default Display of system applications can be toggled by using the *system applications* toggle in the UI. * added HTTP REST API for managing tasks (`/_api/tasks`) * allow passing character lists as optional parameter to AQL functions `TRIM`, `LTRIM` and `RTRIM` These functions now support trimming using custom character lists. If no character lists are specified, all whitespace characters will be removed as previously: TRIM(" foobar\t \r\n ") // "foobar" TRIM(";foo;bar;baz, ", "; ") // "foo;bar;baz" * added AQL string functions `LTRIM`, `RTRIM`, `FIND_FIRST`, `FIND_LAST`, `SPLIT`, `SUBSTITUTE` * added AQL functions `ZIP`, `VALUES` and `PERCENTILE` * made AQL functions `CONCAT` and `CONCAT_SEPARATOR` work with list arguments * dynamically create extra dispatcher threads if required * fixed issue #1097: schemas in the API docs no longer show required properties as optional v2.3.0-beta2 (2014-11-08) ------------------------- * front-end: new icons for uploading and downloading JSON documents into a collection * front-end: fixed documents pagination css display error * front-end: fixed flickering of the progress view * front-end: fixed missing event for documents filter function * front-end: jsoneditor: added CMD+Return (Mac) CTRL+Return (Linux/Win) shortkey for saving a document * front-end: added information tooltip for uploading json documents. * front-end: added database management view to the collapsed navigation menu * front-end: added collection truncation feature * fixed issue #1086: arangoimp: Odd errors if arguments are not given properly * performance improvements for AQL queries that use JavaScript-based expressions internally * added AQL geo functions `WITHIN_RECTANGLE` and `IS_IN_POLYGON` * fixed non-working query results download in AQL editor of web interface * removed debug print message in AQL editor query export routine * fixed issue #1075: Aardvark: user name required even if auth is off #1075 The fix for this prefills the username input field with the current user's account name if any and `root` (the default username) otherwise. Additionally, the tooltip text has been slightly adjusted. * fixed issue #1069: Add 'raw' link to swagger ui so that the raw swagger json can easily be retrieved This adds a link to the Swagger API docs to an application's detail view in the **Applications** tab of the web interface. The link produces the Swagger JSON directly. If authentication is turned on, the link requires authentication, too. * documentation updates v2.3.0-beta1 (2014-11-01) ------------------------- * added dedicated `NOT IN` operator for AQL Previously, a `NOT IN` was only achievable by writing a negated `IN` condition: FOR i IN ... FILTER ! (i IN [ 23, 42 ]) ... This can now alternatively be expressed more intuitively as follows: FOR i IN ... FILTER i NOT IN [ 23, 42 ] ... * added alternative logical operator syntax for AQL Previously, the logical operators in AQL could only be written as: - `&&`: logical and - `||`: logical or - `!`: negation ArangoDB 2.3 introduces the alternative variants for these operators: - `AND`: logical and - `OR`: logical or - `NOT`: negation The new syntax is just an alternative to the old syntax, allowing easier migration from SQL. The old syntax is still fully supported and will be. * improved output of `ArangoStatement.parse()` and POST `/_api/query` If an AQL query can be parsed without problems, The return value of `ArangoStatement.parse()` now contains an attribute `ast` with the abstract syntax tree of the query (before optimizations). Though this is an internal representation of the query and is subject to change, it can be used to inspect how ArangoDB interprets a given query. * improved `ArangoStatement.explain()` and POST `/_api/explain` The commands for explaining AQL queries have been improved. * added command-line option `--javascript.v8-contexts` to control the number of V8 contexts created in arangod. Previously, the number of V8 contexts was equal to the number of server threads (as specified by option `--server.threads`). However, it may be sensible to create different amounts of threads and V8 contexts. If the option is not specified, the number of V8 contexts created will be equal to the number of server threads. Thus no change in configuration is required to keep the old behavior. If you are using the default config files or merge them with your local config files, please review if the default number of server threads is okay in your environment. Additionally you should verify that the number of V8 contexts created (as specified in option `--javascript.v8-contexts`) is okay. * the number of server.threads specified is now the minimum of threads started. There are situation in which threads are waiting for results of distributed database servers. In this case the number of threads is dynamically increased. * removed index type "bitarray" Bitarray indexes were only half-way documented and integrated in previous versions of ArangoDB so their benefit was limited. The support for bitarray indexes has thus been removed in ArangoDB 2.3. It is not possible to create indexes of type "bitarray" with ArangoDB 2.3. When a collection is opened that contains a bitarray index definition created with a previous version of ArangoDB, ArangoDB will ignore it and log the following warning: index type 'bitarray' is not supported in this version of ArangoDB and is ignored Future versions of ArangoDB may automatically remove such index definitions so the warnings will eventually disappear. * removed internal "_admin/modules/flush" in order to fix requireApp * added basic support for handling binary data in Foxx Requests with binary payload can be processed in Foxx applications by using the new method `res.rawBodyBuffer()`. This will return the unparsed request body as a Buffer object. There is now also the method `req.requestParts()` available in Foxx to retrieve the individual components of a multipart HTTP request. Buffer objects can now be used when setting the response body of any Foxx action. Additionally, `res.send()` has been added as a convenience method for returning strings, JSON objects or buffers from a Foxx action: res.send("

some HTML

"); res.send({ success: true }); res.send(new Buffer("some binary data")); The convenience method `res.sendFile()` can now be used to easily return the contents of a file from a Foxx action: res.sendFile(applicationContext.foxxFilename("image.png")); `fs.write` now accepts not only strings but also Buffer objects as second parameter: fs.write(filename, "some data"); fs.write(filename, new Buffer("some binary data")); `fs.readBuffer` can be used to return the contents of a file in a Buffer object. * improved performance of insertion into non-unique hash indexes significantly in case many duplicate keys are used in the index * issue #1042: set time zone in log output the command-line option `--log.use-local-time` was added to print dates and times in the server-local timezone instead of UTC * command-line options that require a boolean value now validate the value given on the command-line This prevents issues if no value is specified for an option that requires a boolean value. For example, the following command-line would have caused trouble in 2.2, because `--server.endpoint` would have been used as the value for the `--server.disable-authentication` options (which requires a boolean value): arangod --server.disable-authentication --server.endpoint tcp://127.0.0.1:8529 data In 2.3, running this command will fail with an error and requires to be modified to: arangod --server.disable-authentication true --server.endpoint tcp://127.0.0.1:8529 data * improved performance of CSV import in arangoimp * fixed issue #1027: Stack traces are off-by-one * fixed issue #1026: Modules loaded in different files within the same app should refer to the same module * fixed issue #1025: Traversal not as expected in undirected graph * added a _relation function in the general-graph module. This deprecated _directedRelation and _undirectedRelation. ArangoDB does not offer any constraints for undirected edges which caused some confusion of users how undirected relations have to be handled. Relation now only supports directed relations and the user can actively simulate undirected relations. * changed return value of Foxx.applicationContext#collectionName: Previously, the function could return invalid collection names because invalid characters were not replaced in the application name prefix, only in the collection name passed. Now, the function replaces invalid characters also in the application name prefix, which might to slightly different results for application names that contained any characters outside the ranges [a-z], [A-Z] and [0-9]. * prevent XSS in AQL editor and logs view * integrated tutorial into ArangoShell and web interface * added option `--backslash-escape` for arangoimp when running CSV file imports * front-end: added download feature for (filtered) documents * front-end: added download feature for the results of a user query * front-end: added function to move documents to another collection * front-end: added sort-by attribute to the documents filter * front-end: added sorting feature to database, graph management and user management view. * issue #989: front-end: Databases view not refreshing after deleting a database * issue #991: front-end: Database search broken * front-end: added infobox which shows more information about a document (_id, _rev, _key) or an edge (_id, _rev, _key, _from, _to). The from and to attributes are clickable and redirect to their document location. * front-end: added edit-mode for deleting multiple documents at the same time. * front-end: added delete button to the detailed document/edge view. * front-end: added visual feedback for saving documents/edges inside the editor (error/success). * front-end: added auto-focusing for the first input field in a modal. * front-end: added validation for user input in a modal. * front-end: user defined queries are now stored inside the database and are bound to the current user, instead of using the local storage functionality of the browsers. The outcome of this is that user defined queries are now independently usable from any device. Also queries can now be edited through the standard document editor of the front-end through the _users collection. * front-end: added import and export functionality for user defined queries. * front-end: added new keywords and functions to the aql-editor theme * front-end: applied tile-style to the graph view * front-end: now using the new graph api including multi-collection support * front-end: foxx apps are now deletable * front-end: foxx apps are now installable and updateable through github, if github is their origin. * front-end: added foxx app version control. Multiple versions of a single foxx app are now installable and easy to manage and are also arranged in groups. * front-end: the user-set filter of a collection is now stored until the user navigates to another collection. * front-end: fetching and filtering of documents, statistics, and query operations are now handled with asynchronous ajax calls. * front-end: added progress indicator if the front-end is waiting for a server operation. * front-end: fixed wrong count of documents in the documents view of a collection. * front-end: fixed unexpected styling of the manage db view and navigation. * front-end: fixed wrong handling of select fields in a modal view. * front-end: fixed wrong positioning of some tooltips. * automatically call `toJSON` function of JavaScript objects (if present) when serializing them into database documents. This change allows storing JavaScript date objects in the database in a sensible manner. v2.2.7 (2014-11-19) ------------------- * fixed issue #998: Incorrect application URL for non-system Foxx apps * fixed issue #1079: AQL editor: keyword WITH in UPDATE query is not highlighted * fix memory leak in cluster nodes * fixed registration of AQL user-defined functions in Web UI (JS shell) * fixed error display in Web UI for certain errors (now error message is printed instead of 'undefined') * fixed issue #1059: bug in js module console * fixed issue #1056: "fs": zip functions fail with passwords * fixed issue #1063: Docs: measuring unit of --wal.logfile-size? * fixed issue #1062: Docs: typo in 14.2 Example data v2.2.6 (2014-10-20) ------------------- * fixed issue #972: Compilation Issue * fixed issue #743: temporary directories are now unique and one can read off the tool that created them, if empty, they are removed atexit * Highly improved performance of all AQL GRAPH_* functions. * Orphan collections in general graphs can now be found via GRAPH_VERTICES if either "any" or no direction is defined * Fixed documentation for AQL function GRAPH_NEIGHBORS. The option "vertexCollectionRestriction" is meant to filter the target vertices only, and should not filter the path. * Fixed a bug in GRAPH_NEIGHBORS which enforced only empty results under certain conditions v2.2.5 (2014-10-09) ------------------- * fixed issue #961: allow non-JSON values in undocument request bodies * fixed issue 1028: libicu is now statically linked * fixed cached lookups of collections on the server, which may have caused spurious problems after collection rename operations v2.2.4 (2014-10-01) ------------------- * fixed accessing `_from` and `_to` attributes in `collection.byExample` and `collection.firstExample` These internal attributes were not handled properly in the mentioned functions, so searching for them did not always produce documents * fixed issue #1030: arangoimp 2.2.3 crashing, not logging on large Windows CSV file * fixed issue #1025: Traversal not as expected in undirected graph * fixed issue #1020 This requires re-introducing the startup option `--database.force-sync-properties`. This option can again be used to force fsyncs of collection, index and database properties stored as JSON strings on disk in files named `parameter.json`. Syncing these files after a write may be necessary if the underlying storage does not sync file contents by itself in a "sensible" amount of time after a file has been written and closed. The default value is `true` so collection, index and database properties will always be synced to disk immediately. This affects creating, renaming and dropping collections as well as creating and dropping databases and indexes. Each of these operations will perform an additional fsync on the `parameter.json` file if the option is set to `true`. It might be sensible to set this option to `false` for workloads that create and drop a lot of collections (e.g. test runs). Document operations such as creating, updating and dropping documents are not affected by this option. * fixed issue #1016: AQL editor bug * fixed issue #1014: WITHIN function returns wrong distance * fixed AQL shortest path calculation in function `GRAPH_SHORTEST_PATH` to return complete vertex objects instead of just vertex ids * allow changing of attributes of documents stored in server-side JavaScript variables Previously, the following did not work: var doc = db.collection.document(key); doc._key = "abc"; // overwriting internal attributes not supported doc.value = 123; // overwriting existing attributes not supported Now, modifying documents stored in server-side variables (e.g. `doc` in the above case) is supported. Modifying the variables will not update the documents in the database, but will modify the JavaScript object (which can be written back to the database using `db.collection.update` or `db.collection.replace`) * fixed issue #997: arangoimp apparently doesn't support files >2gig on Windows large file support (requires using `_stat64` instead of `stat`) is now supported on Windows v2.2.3 (2014-09-02) ------------------- * added `around` for Foxx controller * added `type` option for HTTP API `GET /_api/document?collection=...` This allows controlling the type of results to be returned. By default, paths to documents will be returned, e.g. [ `/_api/document/test/mykey1`, `/_api/document/test/mykey2`, ... ] To return a list of document ids instead of paths, the `type` URL parameter can be set to `id`: [ `test/mykey1`, `test/mykey2`, ... ] To return a list of document keys only, the `type` URL parameter can be set to `key`: [ `mykey1`, `mykey2`, ... ] * properly capitalize HTTP response header field names in case the `x-arango-async` HTTP header was used in a request. * fixed several documentation issues * speedup for several general-graph functions, AQL functions starting with `GRAPH_` and traversals v2.2.2 (2014-08-08) ------------------- * allow storing non-reserved attribute names starting with an underscore Previous versions of ArangoDB parsed away all attribute names that started with an underscore (e.g. `_test', '_foo', `_bar`) on all levels of a document (root level and sub-attribute levels). While this behavior was documented, it was unintuitive and prevented storing documents inside other documents, e.g.: { "_key" : "foo", "_type" : "mydoc", "references" : [ { "_key" : "something", "_rev" : "...", "value" : 1 }, { "_key" : "something else", "_rev" : "...", "value" : 2 } ] } In the above example, previous versions of ArangoDB removed all attributes and sub-attributes that started with underscores, meaning the embedded documents would lose some of their attributes. 2.2.2 should preserve such attributes, and will also allow storing user-defined attribute names on the top-level even if they start with underscores (such as `_type` in the above example). * fix conversion of JavaScript String, Number and Boolean objects to JSON. Objects created in JavaScript using `new Number(...)`, `new String(...)`, or `new Boolean(...)` were not converted to JSON correctly. * fixed a race condition on task registration (i.e. `require("org/arangodb/tasks").register()`) this race condition led to undefined behavior when a just-created task with no offset and no period was instantly executed and deleted by the task scheduler, before the `register` function returned to the caller. * changed run-tests.sh to execute all suitable tests. * switch to new version of gyp * fixed upgrade button v2.2.1 (2014-07-24) ------------------- * fixed hanging write-ahead log recovery for certain cases that involved dropping databases * fixed issue with --check-version: when creating a new database the check failed * issue #947 Foxx applicationContext missing some properties * fixed issue with --check-version: when creating a new database the check failed * added startup option `--wal.suppress-shape-information` Setting this option to `true` will reduce memory and disk space usage and require less CPU time when modifying documents or edges. It should therefore be turned on for standalone ArangoDB servers. However, for servers that are used as replication masters, setting this option to `true` will effectively disable the usage of the write-ahead log for replication, so it should be set to `false` for any replication master servers. The default value for this option is `false`. * added optional `ttl` attribute to specify result cursor expiration for HTTP API method `POST /_api/cursor` The `ttl` attribute can be used to prevent cursor results from timing out too early. * issue #947: Foxx applicationContext missing some properties * (reported by Christian Neubauer): The problem was that in Google's V8, signed and unsigned chars are not always declared cleanly. so we need to force v8 to compile with forced signed chars which is done by the Flag: -fsigned-char at least it is enough to follow the instructions of compiling arango on rasperry and add "CFLAGS='-fsigned-char'" to the make command of V8 and remove the armv7=0 * Fixed a bug with the replication client. In the case of single document transactions the collection was not write locked. v2.2.0 (2014-07-10) ------------------- * The replication methods `logger.start`, `logger.stop` and `logger.properties` are no-ops in ArangoDB 2.2 as there is no separate replication logger anymore. Data changes are logged into the write-ahead log in ArangoDB 2.2, and not separately by the replication logger. The replication logger object is still there in ArangoDB 2.2 to ensure backwards-compatibility, however, logging cannot be started, stopped or configured anymore. Using any of these methods will do nothing. This also affects the following HTTP API methods: - `PUT /_api/replication/logger-start` - `PUT /_api/replication/logger-stop` - `GET /_api/replication/logger-config` - `PUT /_api/replication/logger-config` Using any of these methods is discouraged from now on as they will be removed in future versions of ArangoDB. * INCOMPATIBLE CHANGE: replication of transactions has changed. Previously, transactions were logged on a master in one big block and shipped to a slave in one block, too. Now transactions will be logged and replicated as separate entries, allowing transactions to be bigger and also ensure replication progress. This change also affects the behavior of the `stop` method of the replication applier. If the replication applier is now stopped manually using the `stop` method and later restarted using the `start` method, any transactions that were unfinished at the point of stopping will be aborted on a slave, even if they later commit on the master. In ArangoDB 2.2, stopping the replication applier manually should be avoided unless the goal is to stop replication permanently or to do a full resync with the master anyway. If the replication applier still must be stopped, it should be made sure that the slave has fetched and applied all pending operations from a master, and that no extra transactions are started on the master before the `stop` command on the slave is executed. Replication of transactions in ArangoDB 2.2 might also lock the involved collections on the slave while a transaction is either committed or aborted on the master and the change has been replicated to the slave. This change in behavior may be important for slave servers that are used for read-scaling. In order to avoid long lasting collection locks on the slave, transactions should be kept small. The `_replication` system collection is not used anymore in ArangoDB 2.2 and its usage is discouraged. * INCOMPATIBLE CHANGE: the figures reported by the `collection.figures` method now only reflect documents and data contained in the journals and datafiles of collections. Documents or deletions contained only in the write-ahead log will not influence collection figures until the write-ahead log garbage collection kicks in. The figures for a collection might therefore underreport the total resource usage of a collection. Additionally, the attributes `lastTick` and `uncollectedLogfileEntries` have been added to the result of the `figures` operation and the HTTP API method `PUT /_api/collection/figures` * added `insert` method as an alias for `save`. Documents can now be inserted into a collection using either method: db.test.save({ foo: "bar" }); db.test.insert({ foo: "bar" }); * added support for data-modification AQL queries * added AQL keywords `INSERT`, `UPDATE`, `REPLACE` and `REMOVE` (and `WITH`) to support data-modification AQL queries. Unquoted usage of these keywords for attribute names in AQL queries will likely fail in ArangoDB 2.2. If any such attribute name needs to be used in a query, it should be enclosed in backticks to indicate the usage of a literal attribute name. For example, the following query will fail in ArangoDB 2.2 with a parse error: FOR i IN foo RETURN i.remove and needs to be rewritten like this: FOR i IN foo RETURN i.`remove` * disallow storing of JavaScript objects that contain JavaScript native objects of type `Date`, `Function`, `RegExp` or `External`, e.g. db.test.save({ foo: /bar/ }); db.test.save({ foo: new Date() }); will now print Error: cannot be converted into JSON shape: could not shape document Previously, objects of these types were silently converted into an empty object (i.e. `{ }`). To store such objects in a collection, explicitly convert them into strings like this: db.test.save({ foo: String(/bar/) }); db.test.save({ foo: String(new Date()) }); * The replication methods `logger.start`, `logger.stop` and `logger.properties` are no-ops in ArangoDB 2.2 as there is no separate replication logger anymore. Data changes are logged into the write-ahead log in ArangoDB 2.2, and not separately by the replication logger. The replication logger object is still there in ArangoDB 2.2 to ensure backwards-compatibility, however, logging cannot be started, stopped or configured anymore. Using any of these methods will do nothing. This also affects the following HTTP API methods: - `PUT /_api/replication/logger-start` - `PUT /_api/replication/logger-stop` - `GET /_api/replication/logger-config` - `PUT /_api/replication/logger-config` Using any of these methods is discouraged from now on as they will be removed in future versions of ArangoDB. * INCOMPATIBLE CHANGE: replication of transactions has changed. Previously, transactions were logged on a master in one big block and shipped to a slave in one block, too. Now transactions will be logged and replicated as separate entries, allowing transactions to be bigger and also ensure replication progress. This change also affects the behavior of the `stop` method of the replication applier. If the replication applier is now stopped manually using the `stop` method and later restarted using the `start` method, any transactions that were unfinished at the point of stopping will be aborted on a slave, even if they later commit on the master. In ArangoDB 2.2, stopping the replication applier manually should be avoided unless the goal is to stop replication permanently or to do a full resync with the master anyway. If the replication applier still must be stopped, it should be made sure that the slave has fetched and applied all pending operations from a master, and that no extra transactions are started on the master before the `stop` command on the slave is executed. Replication of transactions in ArangoDB 2.2 might also lock the involved collections on the slave while a transaction is either committed or aborted on the master and the change has been replicated to the slave. This change in behavior may be important for slave servers that are used for read-scaling. In order to avoid long lasting collection locks on the slave, transactions should be kept small. The `_replication` system collection is not used anymore in ArangoDB 2.2 and its usage is discouraged. * INCOMPATIBLE CHANGE: the figures reported by the `collection.figures` method now only reflect documents and data contained in the journals and datafiles of collections. Documents or deletions contained only in the write-ahead log will not influence collection figures until the write-ahead log garbage collection kicks in. The figures for a collection might therefore underreport the total resource usage of a collection. Additionally, the attributes `lastTick` and `uncollectedLogfileEntries` have been added to the result of the `figures` operation and the HTTP API method `PUT /_api/collection/figures` * added `insert` method as an alias for `save`. Documents can now be inserted into a collection using either method: db.test.save({ foo: "bar" }); db.test.insert({ foo: "bar" }); * added support for data-modification AQL queries * added AQL keywords `INSERT`, `UPDATE`, `REPLACE` and `REMOVE` (and `WITH`) to support data-modification AQL queries. Unquoted usage of these keywords for attribute names in AQL queries will likely fail in ArangoDB 2.2. If any such attribute name needs to be used in a query, it should be enclosed in backticks to indicate the usage of a literal attribute name. For example, the following query will fail in ArangoDB 2.2 with a parse error: FOR i IN foo RETURN i.remove and needs to be rewritten like this: FOR i IN foo RETURN i.`remove` * disallow storing of JavaScript objects that contain JavaScript native objects of type `Date`, `Function`, `RegExp` or `External`, e.g. db.test.save({ foo: /bar/ }); db.test.save({ foo: new Date() }); will now print Error: cannot be converted into JSON shape: could not shape document Previously, objects of these types were silently converted into an empty object (i.e. `{ }`). To store such objects in a collection, explicitly convert them into strings like this: db.test.save({ foo: String(/bar/) }); db.test.save({ foo: String(new Date()) }); * honor startup option `--server.disable-statistics` when deciding whether or not to start periodic statistics collection jobs Previously, the statistics collection jobs were started even if the server was started with the `--server.disable-statistics` flag being set to `true` * removed startup option `--random.no-seed` This option had no effect in previous versions of ArangoDB and was thus removed. * removed startup option `--database.remove-on-drop` This option was used for debugging only. * removed startup option `--database.force-sync-properties` This option is now superfluous as collection properties are now stored in the write-ahead log. * introduced write-ahead log All write operations in an ArangoDB server instance are automatically logged to the server's write-ahead log. The write-ahead log is a set of append-only logfiles, and it is used in case of a crash recovery and for replication. Data from the write-ahead log will eventually be moved into the journals or datafiles of collections, allowing the server to remove older write-ahead log logfiles. Figures of collections will be updated when data are moved from the write-ahead log into the journals or datafiles of collections. Cross-collection transactions in ArangoDB should benefit considerably by this change, as less writes than in previous versions are required to ensure the data of multiple collections are atomically and durably committed. All data-modifying operations inside transactions (insert, update, remove) will write their operations into the write-ahead log directly, making transactions with multiple operations also require less physical memory than in previous versions of ArangoDB, that required all transaction data to fit into RAM. The `_trx` system collection is not used anymore in ArangoDB 2.2 and its usage is discouraged. The data in the write-ahead log can also be used in the replication context. The `_replication` collection that was used in previous versions of ArangoDB to store all changes on the server is not used anymore in ArangoDB 2.2. Instead, slaves can read from a master's write-ahead log to get informed about most recent changes. This removes the need to store data-modifying operations in both the actual place and the `_replication` collection. * removed startup option `--server.disable-replication-logger` This option is superfluous in ArangoDB 2.2. There is no dedicated replication logger in ArangoDB 2.2. There is now always the write-ahead log, and it is also used as the server's replication log. Specifying the startup option `--server.disable-replication-logger` will do nothing in ArangoDB 2.2, but the option should not be used anymore as it might be removed in a future version. * changed behavior of replication logger There is no dedicated replication logger in ArangoDB 2.2 as there is the write-ahead log now. The existing APIs for starting and stopping the replication logger still exist in ArangoDB 2.2 for downwards-compatibility, but calling the start or stop operations are no-ops in ArangoDB 2.2. When querying the replication logger status via the API, the server will always report that the replication logger is running. Configuring the replication logger is a no-op in ArangoDB 2.2, too. Changing the replication logger configuration has no effect. Instead, the write-ahead log configuration can be changed. * removed MRuby integration for arangod ArangoDB had an experimental MRuby integration in some of the publish builds. This wasn't continuously developed, and so it has been removed in ArangoDB 2.2. This change has led to the following startup options being superfluous: - `--ruby.gc-interval` - `--ruby.action-directory` - `--ruby.modules-path` - `--ruby.startup-directory` Specifying these startup options will do nothing in ArangoDB 2.2, but the options should be avoided from now on as they might be removed in future versions. * reclaim index memory when last document in collection is deleted Previously, deleting documents from a collection did not lead to index sizes being reduced. Instead, the already allocated index memory was re-used when a collection was refilled. Now, index memory for primary indexes and hash indexes is reclaimed instantly when the last document from a collection is removed. * inlined and optimized functions in hash indexes * added AQL TRANSLATE function This function can be used to perform lookups from static lists, e.g. LET countryNames = { US: "United States", UK: "United Kingdom", FR: "France" } RETURN TRANSLATE("FR", countryNames) * fixed datafile debugger * fixed check-version for empty directory * moved try/catch block to the top of routing chain * added mountedApp function for foxx-manager * fixed issue #883: arango 2.1 - when starting multi-machine cluster, UI web does not change to cluster overview * fixed dfdb: should not start any other V8 threads * cleanup of version-check, added module org/arangodb/database-version, added --check-version option * fixed issue #881: [2.1.0] Bombarded (every 10 sec or so) with "WARNING format string is corrupt" when in non-system DB Dashboard * specialized primary index implementation to allow faster hash table rebuilding and reduce lookups in datafiles for the actual value of `_key`. * issue #862: added `--overwrite` option to arangoimp * removed number of property lookups for documents during AQL queries that access documents * prevent buffering of long print results in arangosh's and arangod's print command this change will emit buffered intermediate print results and discard the output buffer to quickly deliver print results to the user, and to prevent constructing very large buffers for large results * removed sorting of attribute names for use in a collection's shaper sorting attribute names was done on document insert to keep attributes of a collection in sorted order for faster comparisons. The sort order of attributes was only used in one particular and unlikely case, so it was removed. Collections with many different attribute names should benefit from this change by faster inserts and slightly less memory usage. * fixed a bug in arangodump which got the collection name in _from and _to attributes of edges wrong (all were "_unknown") * fixed a bug in arangorestore which did not recognize wrong _from and _to attributes of edges * improved error detection and reporting in arangorestore v2.1.1 (2014-06-06) ------------------- * fixed dfdb: should not start any other V8 threads * signature for collection functions was modified The basic change was the substitution of the input parameter of the function by an generic options object which can contain multiple option parameter of the function. Following functions were modified remove removeBySample replace replaceBySample update updateBySample Old signature is yet supported but it will be removed in future versions v2.1.0 (2014-05-29) ------------------- * implemented upgrade procedure for clusters * fixed communication issue with agency which prevented reconnect after an agent failure * fixed cluster dashboard in the case that one but not all servers in the cluster are down * fixed a bug with coordinators creating local database objects in the wrong order (_system needs to be done first) * improved cluster dashboard v2.1.0-rc2 (2014-05-25) ----------------------- * fixed issue #864: Inconsistent behavior of AQL REVERSE(list) function v2.1.0-rc1 (XXXX-XX-XX) ----------------------- * added server-side periodic task management functions: - require("org/arangodb/tasks").register(): registers a periodic task - require("org/arangodb/tasks").unregister(): unregisters and removes a periodic task - require("org/arangodb/tasks").get(): retrieves a specific tasks or all existing tasks the previous undocumented function `internal.definePeriodic` is now deprecated and will be removed in a future release. * decrease the size of some seldom used system collections on creation. This will make these collections use less disk space and mapped memory. * added AQL date functions * added AQL FLATTEN() list function * added index memory statistics to `db..figures()` function The `figures` function will now return a sub-document `indexes`, which lists the number of indexes in the `count` sub-attribute, and the total memory usage of the indexes in bytes in the `size` sub-attribute. * added AQL CURRENT_DATABASE() function This function returns the current database's name. * added AQL CURRENT_USER() function This function returns the current user from an AQL query. The current user is the username that was specified in the `Authorization` HTTP header of the request. If authentication is turned off or the query was executed outside a request context, the function will return `null`. * fixed issue #796: Searching with newline chars broken? fixed slightly different handling of backslash escape characters in a few AQL functions. Now handling of escape sequences should be consistent, and searching for newline characters should work the same everywhere * added OpenSSL version check for configure It will report all OpenSSL versions < 1.0.1g as being too old. `configure` will only complain about an outdated OpenSSL version but not stop. * require C++ compiler support (requires g++ 4.8, clang++ 3.4 or Visual Studio 13) * less string copying returning JSONified documents from ArangoDB, e.g. via HTTP GET `/_api/document//` * issue #798: Lower case http headers from arango This change allows returning capitalized HTTP headers, e.g. `Content-Length` instead of `content-length`. The HTTP spec says that headers are case-insensitive, but in fact several clients rely on a specific case in response headers. This change will capitalize HTTP headers if the `X-Arango-Version` request header is sent by the client and contains a value of at least `20100` (for version 2.1). The default value for the compatibility can also be set at server start, using the `--server.default-api-compatibility` option. * simplified usage of `db._createStatement()` Previously, the function could not be called with a query string parameter as follows: db._createStatement(queryString); Calling it as above resulted in an error because the function expected an object as its parameter. From now on, it's possible to call the function with just the query string. * make ArangoDB not send back a `WWW-Authenticate` header to a client in case the client sends the `X-Omit-WWW-Authenticate` HTTP header. This is done to prevent browsers from showing their built-in HTTP authentication dialog for AJAX requests that require authentication. ArangoDB will still return an HTTP 401 (Unauthorized) if the request doesn't contain valid credentials, but it will omit the `WWW-Authenticate` header, allowing clients to bypass the browser's authentication dialog. * added REST API method HTTP GET `/_api/job/job-id` to query the status of an async job without potentially fetching it from the list of done jobs * fixed non-intuitive behavior in jobs API: previously, querying the status of an async job via the API HTTP PUT `/_api/job/job-id` removed a currently executing async job from the list of queryable jobs on the server. Now, when querying the result of an async job that is still executing, the job is kept in the list of queryable jobs so its result can be fetched by a subsequent request. * use a new data structure for the edge index of an edge collection. This improves the performance for the creation of the edge index and in particular speeds up removal of edges in graphs. Note however that this change might change the order in which edges starting at or ending in a vertex are returned. However, this order was never guaranteed anyway and it is not sensible to guarantee any particular order. * provide a size hint to edge and hash indexes when initially filling them this will lead to less re-allocations when populating these indexes this may speed up building indexes when opening an existing collection * don't requeue identical context methods in V8 threads in case a method is already registered * removed arangod command line option `--database.remove-on-compacted` * export the sort attribute for graph traversals to the HTTP interface * add support for arangodump/arangorestore for clusters v2.0.8 (XXXX-XX-XX) ------------------- * fixed too-busy iteration over skiplists Even when a skiplist query was restricted by a limit clause, the skiplist index was queried without the limit. this led to slower-than-necessary execution times. * fixed timeout overflows on 32 bit systems this bug has led to problems when select was called with a high timeout value (2000+ seconds) on 32bit systems that don't have a forgiving select implementation. when the call was made on these systems, select failed so no data would be read or sent over the connection this might have affected some cluster-internal operations. * fixed ETCD issues on 32 bit systems ETCD was non-functional on 32 bit systems at all. The first call to the watch API crashed it. This was because atomic operations worked on data structures that were not properly aligned on 32 bit systems. * fixed issue #848: db.someEdgeCollection.inEdge does not return correct value when called the 2nd time after a .save to the edge collection v2.0.7 (2014-05-05) ------------------- * issue #839: Foxx Manager missing "unfetch" * fixed a race condition at startup this fixes undefined behavior in case the logger was involved directly at startup, before the logger initialization code was called. This should have occurred only for code that was executed before the invocation of main(), e.g. during ctor calls of statically defined objects. v2.0.6 (2014-04-22) ------------------- * fixed issue #835: arangosh doesn't show correct database name v2.0.5 (2014-04-21) ------------------- * Fixed a caching problem in IE JS Shell * added cancelation for async jobs * upgraded to new gyp for V8 * new Windows installer v2.0.4 (2014-04-14) ------------------- * fixed cluster authentication front-end issues for Firefox and IE, there are still problems with Chrome v2.0.3 (2014-04-14) ------------------- * fixed AQL optimizer bug * fixed front-end issues * added password change dialog v2.0.2 (2014-04-06) ------------------- * during cluster startup, do not log (somewhat expected) connection errors with log level error, but with log level info * fixed dashboard modals * fixed connection check for cluster planning front end: firefox does not support async:false * document how to persist a cluster plan in order to relaunch an existing cluster later v2.0.1 (2014-03-31) ------------------- * make ArangoDB not send back a `WWW-Authenticate` header to a client in case the client sends the `X-Omit-WWW-Authenticate` HTTP header. This is done to prevent browsers from showing their built-in HTTP authentication dialog for AJAX requests that require authentication. ArangoDB will still return an HTTP 401 (Unauthorized) if the request doesn't contain valid credentials, but it will omit the `WWW-Authenticate` header, allowing clients to bypass the browser's authentication dialog. * fixed isses in arango-dfdb: the dfdb was not able to unload certain system collections, so these couldn't be inspected with the dfdb sometimes. Additionally, it did not truncate corrupt markers from datafiles under some circumstances * added `changePassword` attribute for users * fixed non-working "save" button in collection edit view of web interface clicking the save button did nothing. one had to press enter in one of the input fields to send modified form data * fixed V8 compile error on MacOS X * prevent `body length: -9223372036854775808` being logged in development mode for some Foxx HTTP responses * fixed several bugs in web interface dashboard * fixed issue #783: coffee script not working in manifest file * fixed issue #783: coffee script not working in manifest file * fixed issue #781: Cant save current query from AQL editor ui * bumped version in `X-Arango-Version` compatibility header sent by arangosh and other client tools from `1.5` to `2.0`. * fixed startup options for arango-dfdb, added details option for arango-dfdb * fixed display of missing error messages and codes in arangosh * when creating a collection via the web interface, the collection type was always "document", regardless of the user's choice v2.0.0 (2014-03-10) ------------------- * first 2.0 release v2.0.0-rc2 (2014-03-07) ----------------------- * fixed cluster authorization v2.0.0-rc1 (2014-02-28) ----------------------- * added sharding :-) * added collection._dbName attribute to query the name of the database from a collection more detailed documentation on the sharding and cluster features can be found in the user manual, section **Sharding** * INCOMPATIBLE CHANGE: using complex values in AQL filter conditions with operators other than equality (e.g. >=, >, <=, <) will disable usage of skiplist indexes for filter evaluation. For example, the following queries will be affected by change: FOR doc IN docs FILTER doc.value < { foo: "bar" } RETURN doc FOR doc IN docs FILTER doc.value >= [ 1, 2, 3 ] RETURN doc The following queries will not be affected by the change: FOR doc IN docs FILTER doc.value == 1 RETURN doc FOR doc IN docs FILTER doc.value == "foo" RETURN doc FOR doc IN docs FILTER doc.value == [ 1, 2, 3 ] RETURN doc FOR doc IN docs FILTER doc.value == { foo: "bar" } RETURN doc * INCOMPATIBLE CHANGE: removed undocumented method `collection.saveOrReplace` this feature was never advertised nor documented nor tested. * INCOMPATIBLE CHANGE: removed undocumented REST API method `/_api/simple/BY-EXAMPLE-HASH` this feature was never advertised nor documented nor tested. * added explicit startup parameter `--server.reuse-address` This flag can be used to control whether sockets should be acquired with the SO_REUSEADDR flag. Regardless of this setting, sockets on Windows are always acquired using the SO_EXCLUSIVEADDRUSE flag. * removed undocumented REST API method GET `/_admin/database-name` * added user validation API at POST `/_api/user/` * slightly improved users management API in `/_api/user`: Previously, when creating a new user via HTTP POST, the username needed to be passed in an attribute `username`. When users were returned via this API, the usernames were returned in an attribute named `user`. This was slightly confusing and was changed in 2.0 as follows: - when adding a user via HTTP POST, the username can be specified in an attribute `user`. If this attribute is not used, the API will look into the attribute `username` as before and use that value. - when users are returned via HTTP GET, the usernames are still returned in an attribute `user`. This change should be fully downwards-compatible with the previous version of the API. * added AQL SLICE function to extract slices from lists * made module loader more node compatible * the startup option `--javascript.package-path` for arangosh is now deprecated and does nothing. Using it will not cause an error, but the option is ignored. * added coffee script support * Several UI improvements. * Exchanged icons in the graphviewer toolbar * always start networking and HTTP listeners when starting the server (even in console mode) * allow vertex and edge filtering with user-defined functions in TRAVERSAL, TRAVERSAL_TREE and SHORTEST_PATH AQL functions: // using user-defined AQL functions for edge and vertex filtering RETURN TRAVERSAL(friends, friendrelations, "friends/john", "outbound", { followEdges: "myfunctions::checkedge", filterVertices: "myfunctions::checkvertex" }) // using the following custom filter functions var aqlfunctions = require("org/arangodb/aql/functions"); aqlfunctions.register("myfunctions::checkedge", function (config, vertex, edge, path) { return (edge.type !== 'dislikes'); // don't follow these edges }, false); aqlfunctions.register("myfunctions::checkvertex", function (config, vertex, path) { if (vertex.isDeleted || ! vertex.isActive) { return [ "prune", "exclude" ]; // exclude these and don't follow them } return [ ]; // include everything else }, false); * fail if invalid `strategy`, `order` or `itemOrder` attribute values are passed to the AQL TRAVERSAL function. Omitting these attributes is not considered an error, but specifying an invalid value for any of these attributes will make an AQL query fail. * issue #751: Create database through API should return HTTP status code 201 By default, the server now returns HTTP 201 (created) when creating a new database successfully. To keep compatibility with older ArangoDB versions, the startup parameter `--server.default-api-compatibility` can be set to a value of `10400` to indicate API compatibility with ArangoDB 1.4. The compatibility can also be enforced by setting the `X-Arango-Version` HTTP header in a client request to this API on a per-request basis. * allow direct access from the `db` object to collections whose names start with an underscore (e.g. db._users). Previously, access to such collections via the `db` object was possible from arangosh, but not from arangod (and thus Foxx and actions). The only way to access such collections from these places was via the `db._collection()` workaround. * allow `\n` (as well as `\r\n`) as line terminator in batch requests sent to `/_api/batch` HTTP API. * use `--data-binary` instead of `--data` parameter in generated cURL examples * issue #703: Also show path of logfile for fm.config() * issue #675: Dropping a collection used in "graph" module breaks the graph * added "static" Graph.drop() method for graphs API * fixed issue #695: arangosh server.password error * use pretty-printing in `--console` mode by default * simplified ArangoDB startup options Some startup options are now superfluous or their usage is simplified. The following options have been changed: * `--javascript.modules-path`: this option has been removed. The modules paths are determined by arangod and arangosh automatically based on the value of `--javascript.startup-directory`. If the option is set on startup, it is ignored so startup will not abort with an error `unrecognized option`. * `--javascript.action-directory`: this option has been removed. The actions directory is determined by arangod automatically based on the value of `--javascript.startup-directory`. If the option is set on startup, it is ignored so startup will not abort with an error `unrecognized option`. * `--javascript.package-path`: this option is still available but it is not required anymore to set the standard package paths (e.g. `js/npm`). arangod will automatically use this standard package path regardless of whether it was specified via the options. It is possible to use this option to add additional package paths to the standard value. Configuration files included with arangod are adjusted accordingly. * layout of the graphs tab adapted to better fit with the other tabs * database selection is moved to the bottom right corner of the web interface * removed priority queue index type this feature was never advertised nor documented nor tested. * display internal attributes in document source view of web interface * removed separate shape collections When upgrading to ArangoDB 2.0, existing collections will be converted to include shapes and attribute markers in the datafiles instead of using separate files for shapes. When a collection is converted, existing shapes from the SHAPES directory will be written to a new datafile in the collection directory, and the SHAPES directory will be removed afterwards. This saves up to 2 MB of memory and disk space for each collection (savings are higher, the less different shapes there are in a collection). Additionally, one less file descriptor per opened collection will be used. When creating a new collection, the amount of sync calls may be reduced. The same may be true for documents with yet-unknown shapes. This may help performance in these cases. * added AQL functions `NTH` and `POSITION` * added signal handler for arangosh to save last command in more cases * added extra prompt placeholders for arangosh: - `%e`: current endpoint - `%u`: current user * added arangosh option `--javascript.gc-interval` to control amount of garbage collection performed by arangosh * fixed issue #651: Allow addEdge() to take vertex ids in the JS library * removed command-line option `--log.format` In previous versions, this option did not have an effect for most log messages, so it got removed. * removed C++ logger implementation Logging inside ArangoDB is now done using the LOG_XXX() macros. The LOGGER_XXX() macros are gone. * added collection status "loading" v1.4.16 (XXXX-XX-XX) -------------------- * fixed too eager datafile deletion this issue could have caused a crash when the compaction had marked datafiles as obsolete and they were removed while "old" temporary query results still pointed to the old datafile positions * fixed issue #826: Replication fails when a collection's configuration changes v1.4.15 (2014-04-19) -------------------- * bugfix for AQL query optimizer the following type of query was too eagerly optimized, leading to errors in code-generation: LET a = (FOR i IN [] RETURN i) LET b = (FOR i IN [] RETURN i) RETURN 1 the problem occurred when both lists in the subqueries were empty. In this case invalid code was generated and the query couldn't be executed. v1.4.14 (2014-04-05) -------------------- * fixed race conditions during shape / attribute insertion A race condition could have led to spurious `cannot find attribute #xx` or `cannot find shape #xx` (where xx is a number) warning messages being logged by the server. This happened when a new attribute was inserted and at the same time was queried by another thread. Also fixed a race condition that may have occurred when a thread tried to access the shapes / attributes hash tables while they were resized. In this cases, the shape / attribute may have been hashed to a wrong slot. * fixed a memory barrier / cpu synchronization problem with libev, affecting Windows with Visual Studio 2013 (probably earlier versions are affected, too) The issue is described in detail here: http://lists.schmorp.de/pipermail/libev/2014q1/002318.html v1.4.13 (2014-03-14) -------------------- * added diagnostic output for Foxx application upload * allow dump & restore from ArangoDB 1.4 with an ArangoDB 2.0 server * allow startup options `temp-path` and `default-language` to be specified from the arangod configuration file and not only from the command line * fixed too eager compaction The compaction will now wait for several seconds before trying to re-compact the same collection. Additionally, some other limits have been introduced for the compaction. v1.4.12 (2014-03-05) -------------------- * fixed display bug in web interface which caused the following problems: - documents were displayed in web interface as being empty - document attributes view displayed many attributes with content "undefined" - document source view displayed many attributes with name "TYPEOF" and value "undefined" - an alert popping up in the browser with message "Datatables warning..." * re-introduced old-style read-write locks to supports Windows versions older than Windows 2008R2 and Windows 7. This should re-enable support for Windows Vista and Windows 2008. v1.4.11 (2014-02-27) -------------------- * added SHORTEST_PATH AQL function this calculates the shortest paths between two vertices, using the Dijkstra algorithm, employing a min-heap By default, ArangoDB does not know the distance between any two vertices and will use a default distance of 1. A custom distance function can be registered as an AQL user function to make the distance calculation use any document attributes or custom logic: RETURN SHORTEST_PATH(cities, motorways, "cities/CGN", "cities/MUC", "outbound", { paths: true, distance: "myfunctions::citydistance" }) // using the following custom distance function var aqlfunctions = require("org/arangodb/aql/functions"); aqlfunctions.register("myfunctions::distance", function (config, vertex1, vertex2, edge) { return Math.sqrt(Math.pow(vertex1.x - vertex2.x) + Math.pow(vertex1.y - vertex2.y)); }, false); * fixed bug in Graph.pathTo function * fixed small memleak in AQL optimizer * fixed access to potentially uninitialized variable when collection had a cap constraint v1.4.10 (2014-02-21) -------------------- * fixed graph constructor to allow graph with some parameter to be used * added node.js "events" and "stream" * updated npm packages * added loading of .json file * Fixed http return code in graph api with waitForSync parameter. * Fixed documentation in graph, simple and index api. * removed 2 tests due to change in ruby library. * issue #756: set access-control-expose-headers on CORS response the following headers are now whitelisted by ArangoDB in CORS responses: - etag - content-encoding - content-length - location - server - x-arango-errors - x-arango-async-id v1.4.9 (2014-02-07) ------------------- * return a document's current etag in response header for HTTP HEAD requests on documents that return an HTTP 412 (precondition failed) error. This allows retrieving the document's current revision easily. * added AQL function `SKIPLIST` to directly access skiplist indexes from AQL This is a shortcut method to use a skiplist index for retrieving specific documents in indexed order. The function capability is rather limited, but it may be used for several cases to speed up queries. The documents are returned in index order if only one condition is used. /* return all documents with mycollection.created > 12345678 */ FOR doc IN SKIPLIST(mycollection, { created: [[ '>', 12345678 ]] }) RETURN doc /* return first document with mycollection.created > 12345678 */ FOR doc IN SKIPLIST(mycollection, { created: [[ '>', 12345678 ]] }, 0, 1) RETURN doc /* return all documents with mycollection.created between 12345678 and 123456790 */ FOR doc IN SKIPLIST(mycollection, { created: [[ '>', 12345678 ], [ '<=', 123456790 ]] }) RETURN doc /* return all documents with mycollection.a equal 1 and .b equal 2 */ FOR doc IN SKIPLIST(mycollection, { a: [[ '==', 1 ]], b: [[ '==', 2 ]] }) RETURN doc The function requires a skiplist index with the exact same attributes to be present on the specified collection. All attributes present in the skiplist index must be specified in the conditions specified for the `SKIPLIST` function. Attribute declaration order is important, too: attributes must be specified in the same order in the condition as they have been declared in the skiplist index. * added command-line option `--server.disable-authentication-unix-sockets` with this option, authentication can be disabled for all requests coming in via UNIX domain sockets, enabling clients located on the same host as the ArangoDB server to connect without authentication. Other connections (e.g. TCP/IP) are not affected by this option. The default value for this option is `false`. Note: this option is only supported on platforms that support Unix domain sockets. * call global arangod instance destructor on shutdown * issue #755: TRAVERSAL does not use strategy, order and itemOrder options these options were not honored when configuring a traversal via the AQL TRAVERSAL function. Now, these options are used if specified. * allow vertex and edge filtering with user-defined functions in TRAVERSAL, TRAVERSAL_TREE and SHORTEST_PATH AQL functions: // using user-defined AQL functions for edge and vertex filtering RETURN TRAVERSAL(friends, friendrelations, "friends/john", "outbound", { followEdges: "myfunctions::checkedge", filterVertices: "myfunctions::checkvertex" }) // using the following custom filter functions var aqlfunctions = require("org/arangodb/aql/functions"); aqlfunctions.register("myfunctions::checkedge", function (config, vertex, edge, path) { return (edge.type !== 'dislikes'); // don't follow these edges }, false); aqlfunctions.register("myfunctions::checkvertex", function (config, vertex, path) { if (vertex.isDeleted || ! vertex.isActive) { return [ "prune", "exclude" ]; // exclude these and don't follow them } return [ ]; // include everything else }, false); * issue #748: add vertex filtering to AQL's TRAVERSAL[_TREE]() function v1.4.8 (2014-01-31) ------------------- * install foxx apps in the web interface * fixed a segfault in the import API v1.4.7 (2014-01-23) ------------------- * issue #744: Add usage example arangoimp from Command line * issue #738: added __dirname, __filename pseudo-globals. Fixes #733. (@by pluma) * mount all Foxx applications in system apps directory on startup v1.4.6 (2014-01-20) ------------------- * issue #736: AQL function to parse collection and key from document handle * added fm.rescan() method for Foxx-Manager * fixed issue #734: foxx cookie and route problem * added method `fm.configJson` for arangosh * include `startupPath` in result of API `/_api/foxx/config` v1.4.5 (2014-01-15) ------------------- * fixed issue #726: Alternate Windows Install Method * fixed issue #716: dpkg -P doesn't remove everything * fixed bugs in description of HTTP API `_api/index` * fixed issue #732: Rest API GET revision number * added missing documentation for several methods in HTTP API `/_api/edge/...` * fixed typos in description of HTTP API `_api/document` * defer evaluation of AQL subqueries and logical operators (lazy evaluation) * Updated font in WebFrontend, it now contains a version that renders properly on Windows * generally allow function return values as call parameters to AQL functions * fixed potential deadlock in global context method execution * added override file "arangod.conf.local" (and co) v1.4.4 (2013-12-24) ------------------- * uid and gid are now set in the scripts, there is no longer a separate config file for arangod when started from a script * foxx-manager is now an alias for arangosh * arango-dfdb is now an alias for arangod, moved from bin to sbin * changed from readline to linenoise for Windows * added --install-service and --uninstall-service for Windows * removed --daemon and --supervisor for Windows * arangosh and arangod now uses the config-file which maps the binary name, i. e. if you rename arangosh to foxx-manager it will use the config file foxx-manager.conf * fixed lock file for Windows * fixed issue #711, #687: foxx-manager throws internal errors * added `--server.ssl-protocol` option for client tools this allows connecting from arangosh, arangoimp, arangoimp etc. to an ArangoDB server that uses a non-default value for `--server.ssl-protocol`. The default value for the SSL protocol is 4 (TLSv1). If the server is configured to use a different protocol, it was not possible to connect to it with the client tools. * added more detailed request statistics This adds the number of async-executed HTTP requests plus the number of HTTP requests per individual HTTP method type. * added `--force` option for arangorestore this option allows continuing a restore operation even if the server reports errors in the middle of the restore operation * better error reporting for arangorestore in case the server returned an HTTP error, arangorestore previously reported this error as `internal error` without any details only. Now server-side errors are reported by arangorestore with the server's error message * include more system collections in dumps produced by arangodump previously some system collections were intentionally excluded from dumps, even if the dump was run with `--include-system-collections`. for example, the collections `_aal`, `_modules`, `_routing`, and `_users` were excluded. This makes sense in a replication context but not always in a dump context. When specifying `--include-system-collections`, arangodump will now include the above- mentioned collections in the dump, too. Some other system collections are still excluded even when the dump is run with `--include-system-collections`, for example `_replication` and `_trx`. * fixed issue #701: ArangoStatement undefined in arangosh * fixed typos in configuration files v1.4.3 (2013-11-25) ------------------- * fixed a segfault in the AQL optimizer, occurring when a constant non-list value was used on the right-hand side of an IN operator that had a collection attribute on the left-hand side * issue #662: Fixed access violation errors (crashes) in the Windows version, occurring under some circumstances when accessing databases with multiple clients in parallel * fixed issue #681: Problem with ArchLinux PKGBUILD configuration v1.4.2 (2013-11-20) ------------------- * fixed issue #669: Tiny documentation update * ported Windows version to use native Windows API SRWLocks (slim read-write locks) and condition variables instead of homemade versions MSDN states the following about the compatibility of SRWLocks and Condition Variables: Minimum supported client: Windows Server 2008 [desktop apps | Windows Store apps] Minimum supported server: Windows Vista [desktop apps | Windows Store apps] * fixed issue #662: ArangoDB on Windows hanging This fixes a deadlock issue that occurred on Windows when documents were written to a collection at the same time when some other thread tried to drop the collection. * fixed file-based logging in Windows the logger complained on startup if the specified log file already existed * fixed startup of server in daemon mode (`--daemon` startup option) * fixed a segfault in the AQL optimizer * issue #671: Method graph.measurement does not exist * changed Windows condition variable implementation to use Windows native condition variables This is an attempt to fix spurious Windows hangs as described in issue #662. * added documentation for JavaScript traversals * added --code-page command-line option for Windows version of arangosh * fixed a problem when creating edges via the web interface. The problem only occurred if a collection was created with type "document collection" via the web interface, and afterwards was dropped and re-created with type "edge collection". If the web interface page was not reloaded, the old collection type (document) was cached, making the subsequent creation of edges into the (seeming-to-be-document) collection fail. The fix is to not cache the collection type in the web interface. Users of an older version of the web interface can reload the collections page if they are affected. * fixed a caching problem in arangosh: if a collection was created using the web interface, and then removed via arangosh, arangosh did not actually drop the collection due to caching. Because the `drop` operation was not carried out, this caused misleading error messages when trying to re-create the collection (e.g. `cannot create collection: duplicate name`). * fixed ALT-introduced characters for arangosh console input on Windows The Windows readline port was not able to handle characters that are built using CTRL or ALT keys. Regular characters entered using the CTRL or ALT keys were silently swallowed and not passed to the terminal input handler. This did not seem to cause problems for the US keyboard layout, but was a severe issue for keyboard layouts that require the ALT (or ALT-GR) key to construct characters. For example, entering the character `{` with a German keyboard layout requires pressing ALT-GR + 9. * fixed issue #665: Hash/skiplist combo madness bit my ass this fixes a problem with missing/non-deterministic rollbacks of inserts in case of a unique constraint violation into a collection with multiple secondary indexes (with at least one of them unique) * fixed issue #664: ArangoDB installer on Windows requires drive c: * partly fixed issue #662: ArangoDB on Windows hanging This fixes dropping databases on Windows. In previous 1.4 versions on Windows, one shape collection file was not unloaded and removed when dropping a database, leaving one directory and one shape collection file in the otherwise-dropped database directory. * fixed issue #660: updated documentation on indexes v1.4.1 (2013-11-08) ------------------- * performance improvements for skip-list deletes v1.4.1-rc1 (2013-11-07) ----------------------- * fixed issue #635: Web-Interface should have a "Databases" Menu for Management * fixed issue #624: Web-Interface is missing a Database selector * fixed segfault in bitarray query * fixed issue #656: Cannot create unique index through web interface * fixed issue #654: bitarray index makes server down * fixed issue #653: Slow query * fixed issue #650: Randomness of any() should be improved * made AQL `DOCUMENT()` function polymorphic and work with just one parameter. This allows using the `DOCUMENT` function like this: DOCUMENT('users/john') DOCUMENT([ 'users/john', 'users/amy' ]) in addition to the existing use cases: DOCUMENT(users, 'users/john') DOCUMENT(users, 'john') DOCUMENT(users, [ 'users/john' ]) DOCUMENT(users, [ 'users/john', 'users/amy' ]) DOCUMENT(users, [ 'john', 'amy' ]) * simplified usage of ArangoDB batch API It is not necessary anymore to send the batch boundary in the HTTP `Content-Type` header. Previously, the batch API expected the client to send a Content-Type header of`multipart/form-data; boundary=`. This is still supported in ArangoDB 2.0, but clients can now also omit this header. If the header is not present in a client request, ArangoDB will ignore the request content type and read the MIME boundary from the beginning of the request body. This also allows using the batch API with the Swagger "Try it out" feature (which is not too good at sending a different or even dynamic content-type request header). * added API method GET `/_api/database/user` This returns the list of databases a specific user can see without changing the username/passwd. * issue #424: Documentation about IDs needs to be upgraded v1.4.0 (2013-10-29) ------------------- * fixed issue #648: /batch API is missing from Web Interface API Documentation (Swagger) * fixed issue #647: Icon tooltips missing * fixed issue #646: index creation in web interface * fixed issue #645: Allow jumping from edge to linked vertices * merged PR for issue #643: Some minor corrections and a link to "Downloads" * fixed issue #642: Completion of error handling * fixed issue #639: compiling v1.4 on maverick produces warnings on -Wstrict-null-sentinel * fixed issue #634: Web interface bug: Escape does not always propagate * fixed issue #620: added startup option `--server.default-api-compatibility` This adds the following changes to the ArangoDB server and clients: - the server provides a new startup option `--server.default-api-compatibility`. This option can be used to determine the compatibility of (some) server API return values. The value for this parameter is a server version number, calculated as follows: `10000 * major + 100 * minor` (e.g. `10400` for ArangoDB 1.3). The default value is `10400` (1.4), the minimum allowed value is `10300` (1.3). When setting this option to a value lower than the current server version, the server might respond with old-style results to "old" clients, increasing compatibility with "old" (non-up-to-date) clients. - the server will on each incoming request check for an HTTP header `x-arango-version`. Clients can optionally set this header to the API version number they support. For example, if a client sends the HTTP header `x-arango-version: 10300`, the server will pick this up and might send ArangoDB 1.3-style responses in some situations. Setting either the startup parameter or using the HTTP header (or both) allows running "old" clients with newer versions of ArangoDB, without having to adjust the clients too much. - the `location` headers returned by the server for the APIs `/_api/document/...` and `/_api/collection/...` will have different values depending on the used API version. If the API compatibility is `10300`, the `location` headers returned will look like this: location: /_api/document/.... whereas when an API compatibility of `10400` or higher is used, the `location` headers will look like this: location: /_db//_api/document/... Please note that even in the presence of this, old API versions still may not be supported forever by the server. * fixed issue #643: Some minor corrections and a link to "Downloads" by @frankmayer * started issue #642: Completion of error handling * fixed issue #639: compiling v1.4 on maverick produces warnings on -Wstrict-null-sentinel * fixed issue #621: Standard Config needs to be fixed * added function to manage indexes (web interface) * improved server shutdown time by signaling shutdown to applicationserver, logging, cleanup and compactor threads * added foxx-manager `replace` command * added foxx-manager `installed` command (a more intuitive alias for `list`) * fixed issue #617: Swagger API is missing '/_api/version' * fixed issue #615: Swagger API: Some commands have no parameter entry forms * fixed issue #614: API : Typo in : Request URL /_api/database/current * fixed issue #609: Graph viz tool - different background color * fixed issue #608: arangosh config files - eventually missing in the manual * fixed issue #607: Admin interface: no core documentation * fixed issue #603: Aardvark Foxx App Manager * fixed a bug in type-mapping between AQL user functions and the AQL layer The bug caused errors like the following when working with collection documents in an AQL user function: TypeError: Cannot assign to read only property '_id' of # * create less system collections when creating a new database This is achieved by deferring collection creation until the collections are actually needed by ArangoDB. The following collections are affected by the change: - `_fishbowl` - `_structures` v1.4.0-beta2 (2013-10-14) ------------------------- * fixed compaction on Windows The compaction on Windows did not ftruncate the cleaned datafiles to a smaller size. This has been fixed so not only the content of the files is cleaned but also files are re-created with potentially smaller sizes. * only the following system collections will be excluded from replication from now on: - `_replication` - `_trx` - `_users` - `_aal` - `_fishbowl` - `_modules` - `_routing` Especially the following system collections will now be included in replication: - `_aqlfunctions` - `_graphs` In previous versions of ArangoDB, all system collections were excluded from the replication. The change also caused a change in the replication logger and applier: in previous versions of ArangoDB, only a collection's id was logged for an operation. This has not caused problems for non-system collections but for system collections there ids might differ. In addition to a collection id ArangoDB will now also log the name of a collection for each replication event. The replication applier will now look for the collection name attribute in logged events preferably. * added database selection to arango-dfdb * provide foxx-manager, arangodump, and arangorestore in Windows build * ArangoDB 1.4 will refuse to start if option `--javascript.app-path` is not set. * added startup option `--server.allow-method-override` This option can be set to allow overriding the HTTP request method in a request using one of the following custom headers: - x-http-method-override - x-http-method - x-method-override This allows bypassing proxies and tools that would otherwise just let certain types of requests pass. Enabling this option may impose a security risk, so it should only be used in very controlled environments. The default value for this option is `false` (no method overriding allowed). * added "details" URL parameter for bulk import API Setting the `details` URL parameter to `true` in a call to POST `/_api/import` will make the import return details about non-imported documents in the `details` attribute. If `details` is `false` or omitted, no `details` attribute will be present in the response. This is the same behavior that previous ArangoDB versions exposed. * added "complete" option for bulk import API Setting the `complete` URL parameter to `true` in a call to POST `/_api/import` will make the import completely fail if at least one of documents cannot be imported successfully. It defaults to `false`, which will make ArangoDB continue importing the other documents from the import even if some documents cannot be imported. This is the same behavior that previous ArangoDB versions exposed. * added missing swagger documentation for `/_api/log` * calling `/_api/logs` (or `/_admin/logs`) is only permitted from the `_system` database now. Calling this API method for/from other database will result in an HTTP 400. ' ported fix from https://github.com/novus/nvd3/commit/0894152def263b8dee60192f75f66700cea532cc This prevents JavaScript errors from occurring in Chrome when in the admin interface, section "Dashboard". * show current database name in web interface (bottom right corner) * added missing documentation for /_api/import in swagger API docs * allow specification of database name for replication sync command replication applier This allows syncing from a master database with a different name than the slave database. * issue #601: Show DB in prompt arangosh now displays the database name as part of the prompt by default. Can change the prompt by using the `--prompt` option, e.g. > arangosh --prompt "my db is named \"%d\"> " v1.4.0-beta1 (2013-10-01) ------------------------- * make the Foxx manager use per-database app directories Each database now has its own subdirectory for Foxx applications. Each database can thus use different Foxx applications if required. A Foxx app for a specific database resides in `/databases//`. System apps are shared between all databases. They reside in `/system/`. * only trigger an engine reset in development mode for URLs starting with `/dev/` This prevents ArangoDB from reloading all Foxx applications when it is not actually necessary. * changed error code from 10 (bad parameter) to 1232 (invalid key generator) for errors that are due to an invalid key generator specification when creating a new collection * automatic detection of content-type / mime-type for Foxx assets based on filenames, added possibility to override auto detection * added endpoint management API at `/_api/endpoint` * changed HTTP return code of PUT `/_api/cursor` from 400 to 404 in case a non-existing cursor is referred to * issue #360: added support for asynchronous requests Incoming HTTP requests with the headers `x-arango-async: true` or `x-arango-async: store` will be answered by the server instantly with a generic HTTP 202 (Accepted) response. The actual requests will be queued and processed by the server asynchronously, allowing the client to continue sending other requests without waiting for the server to process the actually requested operation. The exact point in time when a queued request is executed is undefined. If an error occurs during execution of an asynchronous request, the client will not be notified by the server. The maximum size of the asynchronous task queue can be controlled using the new option `--scheduler.maximal-queue-size`. If the queue contains this many number of tasks and a new asynchronous request comes in, the server will reject it with an HTTP 500 (internal server error) response. Results of incoming requests marked with header `x-arango-async: true` will be discarded by the server immediately. Clients have no way of accessing the result of such asynchronously executed request. This is just _fire and forget_. To later retrieve the result of an asynchronously executed request, clients can mark a request with the header `x-arango-async: keep`. This makes the server store the result of the request in memory until explicitly fetched by a client via the `/_api/job` API. The `/_api/job` API also provides methods for basic inspection of which pending or already finished requests there are on the server, plus ways for garbage collecting unneeded results. * Added new option `--scheduler.maximal-queue-size`. * issue #590: Manifest Lint * added data dump and restore tools, arangodump and arangorestore. arangodump can be used to create a logical dump of an ArangoDB database, or just dedicated collections. It can be used to dump both a collection's structure (properties and indexes) and data (documents). arangorestore can be used to restore data from a dump created with arangodump. arangorestore currently does not re-create any indexes, and doesn't yet handle referenced documents in edges properly when doing just partial restores. This will be fixed until 1.4 stable. * introduced `--server.database` option for arangosh, arangoimp, and arangob. The option allows these client tools to use a certain database for their actions. In arangosh, the current database can be switched at any time using the command db._useDatabase(); When no database is specified, all client tools will assume they should use the default database `_system`. This is done for downwards-compatibility reasons. * added basic multi database support (alpha) New databases can be created using the REST API POST `/_api/database` and the shell command `db._createDatabase()`. The default database in ArangoDB is called `_system`. This database is always present and cannot be deleted by the user. When an older version of ArangoDB is upgraded to 1.4, the previously only database will automatically become the `_system` database. New databases can be created with the above commands, and can be deleted with the REST API DELETE `/_api/database/` or the shell command `db._dropDatabase();`. Deleting databases is still unstable in ArangoDB 1.4 alpha and might crash the server. This will be fixed until 1.4 stable. To access a specific database via the HTTP REST API, the `/_db//` prefix can be used in all URLs. ArangoDB will check if an incoming request starts with this prefix, and will automatically pick the database name from it. If the prefix is not there, ArangoDB will assume the request is made for the default database (`_system`). This is done for downwards-compatibility reasons. That means, the following URL pathnames are logically identical: /_api/document/mycollection/1234 /_db/_system/document/mycollection/1234 To access a different database (e.g. `test`), the URL pathname would look like this: /_db/test/document/mycollection/1234 New databases can also be created and existing databases can only be dropped from within the default database (`_system`). It is not possible to drop the `_system` database itself. Cross-database operations are unintended and unsupported. The intention of the multi-database feature is to have the possibility to have a few databases managed by ArangoDB in parallel, but to only access one database at a time from a connection or a request. When accessing the web interface via the URL pathname `/_admin/html/` or `/_admin/aardvark`, the web interface for the default database (`_system`) will be displayed. To access the web interface for a different database, the database name can be put into the URLs as a prefix, e.g. `/_db/test/_admin/html` or `/_db/test/_admin/aardvark`. All internal request handlers and also all user-defined request handlers and actions (including Foxx) will only get to see the unprefixed URL pathnames (i.e. excluding any database name prefix). This is to ensure downwards-compatibility. To access the name of the requested database from any action (including Foxx), use use `req.database`. For example, when calling the URL `/myapp/myaction`, the content of `req.database` will be `_system` (the default database because no database got specified) and the content of `req.url` will be `/myapp/myaction`. When calling the URL `/_db/test/myapp/myaction`, the content of `req.database` will be `test`, and the content of `req.url` will still be `/myapp/myaction`. * Foxx now excludes files starting with . (dot) when bundling assets This mitigates problems with editor swap files etc. * made the web interface a Foxx application This change caused the files for the web interface to be moved from `html/admin` to `js/apps/aardvark` in the file system. The base URL for the admin interface changed from `_admin/html/index.html` to `_admin/aardvark/index.html`. The "old" redirection to `_admin/html/index.html` will now produce a 404 error. When starting ArangoDB with the `--upgrade` option, this will automatically be remedied by putting in a redirection from `/` to `/_admin/aardvark/index.html`, and from `/_admin/html/index.html` to `/_admin/aardvark/index.html`. This also obsoletes the following configuration (command-line) options: - `--server.admin-directory` - `--server.disable-admin-interface` when using these now obsolete options when the server is started, no error is produced for downwards-compatibility. * changed User-Agent value sent by arangoimp, arangosh, and arangod from "VOC-Agent" to "ArangoDB" * changed journal file creation behavior as follows: Previously, a journal file for a collection was always created when a collection was created. When a journal filled up and became full, the current journal was made a datafile, and a new (empty) journal was created automatically. There weren't many intended situations when a collection did not have at least one journal. This is changed now as follows: - when a collection is created, no journal file will be created automatically - when there is a write into a collection without a journal, the journal will be created lazily - when there is a write into a collection with a full journal, a new journal will be created automatically From the end user perspective, nothing should have changed, except that there is now less disk usage for empty collections. Disk usage of infrequently updated collections might also be reduced significantly by running the `rotate()` method of a collection, and not writing into a collection subsequently. * added method `collection.rotate()` This allows premature rotation of a collection's current journal file into a (read-only) datafile. The purpose of using `rotate()` is to prematurely allow compaction (which is performed on datafiles only) on data, even if the journal was not filled up completely. Using `rotate()` may make sense in the following scenario: c = db._create("test"); for (i = 0; i < 1000; ++i) { c.save(...); // insert lots of data here } ... c.truncate(); // collection is now empty // only data in datafiles will be compacted by following compaction runs // all data in the current journal would not be compacted // calling rotate will make the current journal a datafile, and thus make it // eligible for compaction c.rotate(); Using `rotate()` may also be useful when data in a collection is known to not change in the immediate future. After having completed all write operations on a collection, performing a `rotate()` will reduce the size of the current journal to the actually required size (remember that journals are pre-allocated with a specific size) before making the journal a datafile. Thus `rotate()` may cause disk space savings, even if the datafiles does not qualify for compaction after rotation. Note: rotating the journal is asynchronous, so that the actual rotation may be executed after `rotate()` returns to the caller. * changed compaction to merge small datafiles together (up to 3 datafiles are merged in a compaction run) In the regular case, this should leave less small datafiles stay around on disk and allow using less file descriptors in total. * added AQL MINUS function * added AQL UNION_DISTINCT function (more efficient than combination of `UNIQUE(UNION())`) * updated mruby to 2013-08-22 * issue #587: Add db._create() in help for startup arangosh * issue #586: Share a link on installation instructions in the User Manual * issue #585: Bison 2.4 missing on Mac for custom build * issue #584: Web interface images broken in devel * issue #583: Small documentation update * issue #581: Parameter binding for attributes * issue #580: Small improvements (by @guidoreina) * issue #577: Missing documentation for collection figures in implementor manual * issue #576: Get disk usage for collections and graphs This extends the result of the REST API for /_api/collection/figures with the attributes `compactors.count`, `compactors.fileSize`, `shapefiles.count`, and `shapefiles.fileSize`. * issue #575: installing devel version on mac (low prio) * issue #574: Documentation (POST /_admin/routing/reload) * issue #558: HTTP cursors, allow count to ignore LIMIT v1.4.0-alpha1 (2013-08-02) -------------------------- * added replication. check online manual for details. * added server startup options `--server.disable-replication-logger` and `--server.disable-replication-applier` * removed action deployment tool, this now handled with Foxx and its manager or by kaerus node utility * fixed a server crash when using byExample / firstExample inside a transaction and the collection contained a usable hash/skiplist index for the example * defineHttp now only expects a single context * added collection detail dialog (web interface) Shows collection properties, figures (datafiles, journals, attributes, etc.) and indexes. * added documents filter (web interface) Allows searching for documents based on attribute values. One or many filter conditions can be defined, using comparison operators such as '==', '<=', etc. * improved AQL editor (web interface) Editor supports keyboard shortcuts (Submit, Undo, Redo, Select). Editor allows saving and reusing of user-defined queries. Added example queries to AQL editor. Added comment button. * added document import (web interface) Allows upload of JSON-data from files. Files must have an extension of .json. * added dashboard (web interface) Shows the status of replication and multiple system charts, e.g. Virtual Memory Size, Request Time, HTTP Connections etc. * added API method `/_api/graph` to query all graphs with all properties. * added example queries in web interface AQL editor * added arango.reconnect() method for arangosh to dynamically switch server or user name * added AQL range operator `..` The `..` operator can be used to easily iterate over a sequence of numeric values. It will produce a list of values in the defined range, with both bounding values included. Example: 2010..2013 will produce the following result: [ 2010, 2011, 2012, 2013 ] * added AQL RANGE function * added collection.first(count) and collection.last(count) document access functions These functions allow accessing the first or last n documents in a collection. The order is determined by document insertion/update time. * added AQL INTERSECTION function * INCOMPATIBLE CHANGE: changed AQL user function namespace resolution operator from `:` to `::` AQL user-defined functions were introduced in ArangoDB 1.3, and the namespace resolution operator for them was the single colon (`:`). A function call looked like this: RETURN mygroup:myfunc() The single colon caused an ambiguity in the AQL grammar, making it indistinguishable from named attributes or the ternary operator in some cases, e.g. { mygroup:myfunc ? mygroup:myfunc } The change of the namespace resolution operator from `:` to `::` fixes this ambiguity. Existing user functions in the database will be automatically fixed when starting ArangoDB 1.4 with the `--upgrade` option. However, queries using user-defined functions need to be adjusted on the client side to use the new operator. * allow multiple AQL LET declarations separated by comma, e.g. LET a = 1, b = 2, c = 3 * more useful AQL error messages The error position (line/column) is more clearly indicated for parse errors. Additionally, if a query references a collection that cannot be found, the error message will give a hint on the collection name * changed return value for AQL `DOCUMENT` function in case document is not found Previously, when the AQL `DOCUMENT` function was called with the id of a document and the document could not be found, it returned `undefined`. This value is not part of the JSON type system and this has caused some problems. Starting with ArangoDB 1.4, the `DOCUMENT` function will return `null` if the document looked for cannot be found. In case the function is called with a list of documents, it will continue to return all found documents, and will not return `null` for non-found documents. This has not changed. * added single line comments for AQL Single line comments can be started with a double forward slash: `//`. They end at the end of the line, or the end of the query string, whichever is first. * fixed documentation issues #567, #568, #571. * added collection.checksum() method to calculate CRC checksums for collections This can be used to - check if data in a collection has changed - compare the contents of two collections on different ArangoDB instances * issue #565: add description line to aal.listAvailable() * fixed several out-of-memory situations when double freeing or invalid memory accesses could happen * less msyncing during the creation of collections This is achieved by not syncing the initial (standard) markers in shapes collections. After all standard markers are written, the shapes collection will get synced. * renamed command-line option `--log.filter` to `--log.source-filter` to avoid misunderstandings * introduced new command-line option `--log.content-filter` to optionally restrict logging to just specific log messages (containing the filter string, case-sensitive). For example, to filter on just log entries which contain `ArangoDB`, use: --log.content-filter "ArangoDB" * added optional command-line option `--log.requests-file` to log incoming HTTP requests to a file. When used, all HTTP requests will be logged to the specified file, containing the client IP address, HTTP method, requests URL, HTTP response code, and size of the response body. * added a signal handler for SIGUSR1 signal: when ArangoDB receives this signal, it will respond all further incoming requests with an HTTP 503 (Service Unavailable) error. This will be the case until another SIGUSR1 signal is caught. This will make ArangoDB start serving requests regularly again. Note: this is not implemented on Windows. * limited maximum request URI length to 16384 bytes: Incoming requests with longer request URIs will be responded to with an HTTP 414 (Request-URI Too Long) error. * require version 1.0 or 1.1 in HTTP version signature of requests sent by clients: Clients sending requests with a non-HTTP 1.0 or non-HTTP 1.1 version number will be served with an HTTP 505 (HTTP Version Not Supported) error. * updated manual on indexes: using system attributes such as `_id`, `_key`, `_from`, `_to`, `_rev` in indexes is disallowed and will be rejected by the server. This was the case since ArangoDB 1.3, but was not properly documented. * issue #563: can aal become a default object? aal is now a prefab object in arangosh * prevent certain system collections from being renamed, dropped, or even unloaded. Which restrictions there are for which system collections may vary from release to release, but users should in general not try to modify system collections directly anyway. Note: there are no such restrictions for user-created collections. * issue #559: added Foxx documentation to user manual * added server startup option `--server.authenticate-system-only`. This option can be used to restrict the need for HTTP authentication to internal functionality and APIs, such as `/_api/*` and `/_admin/*`. Setting this option to `true` will thus force authentication for the ArangoDB APIs and the web interface, but allow unauthenticated requests for other URLs (including user defined actions and Foxx applications). The default value of this option is `false`, meaning that if authentication is turned on, authentication is still required for *all* incoming requests. Only by setting the option to `true` this restriction is lifted and authentication becomes required for URLs starting with `/_` only. Please note that authentication still needs to be enabled regularly by setting the `--server.disable-authentication` parameter to `false`. Otherwise no authentication will be required for any URLs as before. * protect collections against unloading when there are still document barriers around. * extended cap constraints to optionally limit the active data size in a collection to a specific number of bytes. The arguments for creating a cap constraint are now: `collection.ensureCapConstraint(, );` It is supported to specify just a count as in ArangoDB 1.3 and before, to specify just a fileSize, or both. The first met constraint will trigger the automated document removal. * added `db._exists(doc)` and `collection.exists(doc)` for easy document existence checks * added API `/_api/current-database` to retrieve information about the database the client is currently connected to (note: the API `/_api/current-database` has been removed in the meantime. The functionality is accessible via `/_api/database/current` now). * ensure a proper order of tick values in datafiles/journals/compactors. any new files written will have the _tick values of their markers in order. for older files, there are edge cases at the beginning and end of the datafiles when _tick values are not properly in order. * prevent caching of static pages in PathHandler. whenever a static page is requested that is served by the general PathHandler, the server will respond to HTTP GET requests with a "Cache-Control: max-age=86400" header. * added "doCompact" attribute when creating collections and to collection.properties(). The attribute controls whether collection datafiles are compacted. * changed the HTTP return code from 400 to 404 for some cases when there is a referral to a non-existing collection or document. * introduced error code 1909 `too many iterations` that is thrown when graph traversals hit the `maxIterations` threshold. * optionally limit traversals to a certain number of iterations the limitation can be achieved via the traversal API by setting the `maxIterations` attribute, and also via the AQL `TRAVERSAL` and `TRAVERSAL_TREE` functions by setting the same attribute. If traversals are not limited by the end user, a server-defined limit for `maxIterations` may be used to prevent server-side traversals from running endlessly. * added graph traversal API at `/_api/traversal` * added "API" link in web interface, pointing to REST API generated with Swagger * moved "About" link in web interface into "links" menu * allow incremental access to the documents in a collection from out of AQL this allows reading documents from a collection chunks when a full collection scan is required. memory usage might be must lower in this case and queries might finish earlier if there is an additional LIMIT statement * changed AQL COLLECT to use a stable sort, so any previous SORT order is preserved * issue #547: Javascript error in the web interface * issue #550: Make AQL graph functions support key in addition to id * issue #526: Unable to escape when an errorneous command is entered into the js shell * issue #523: Graph and vertex methods for the javascript api * issue #517: Foxx: Route parameters with capital letters fail * issue #512: Binded Parameters for LIMIT v1.3.3 (2013-08-01) ------------------- * issue #570: updateFishbowl() fails once * updated and fixed generated examples * issue #559: added Foxx documentation to user manual * added missing error reporting for errors that happened during import of edges v1.3.2 (2013-06-21) ------------------- * fixed memleak in internal.download() * made the shape-collection journal size adaptive: if too big shapes come in, a shape journal will be created with a big-enough size automatically. the maximum size of a shape journal is still restricted, but to a very big value that should never be reached in practice. * fixed a segfault that occurred when inserting documents with a shape size bigger than the default shape journal size (2MB) * fixed a locking issue in collection.truncate() * fixed value overflow in accumulated filesizes reported by collection.figures() * issue #545: AQL FILTER unnecessary (?) loop * issue #549: wrong return code with --daemon v1.3.1 (2013-05-24) ------------------- * removed currently unused _ids collection * fixed usage of --temp-path in aranogd and arangosh * issue #540: suppress return of temporary internal variables in AQL * issue #530: ReferenceError: ArangoError is not a constructor * issue #535: Problem with AQL user functions javascript API * set --javascript.app-path for test execution to prevent startup error * issue #532: Graph _edgesCache returns invalid data? * issue #531: Arangod errors * issue #529: Really weird transaction issue * fixed usage of --temp-path in aranogd and arangosh v1.3.0 (2013-05-10) ------------------- * fixed problem on restart ("datafile-xxx is not sealed") when server was killed during a compaction run * fixed leak when using cursors with very small batchSize * issue #508: `unregistergroup` function not mentioned in http interface docs * issue #507: GET /_api/aqlfunction returns code inside parentheses * fixed issue #489: Bug in aal.install * fixed issue 505: statistics not populated on MacOS v1.3.0-rc1 (2013-04-24) ----------------------- * updated documentation for 1.3.0 * added node modules and npm packages * changed compaction to only compact datafiles with more at least 10% of dead documents (byte size-wise) * issue #498: fixed reload of authentication info when using `require("org/arangodb/users").reload()` * issue #495: Passing an empty array to create a document results in a "phantom" document * added more precision for requests statistics figures * added "sum" attribute for individual statistics results in statistics API at /_admin/statistics * made "limit" an optional parameter in AQL function NEAR(). limit can now be either omitted completely, or set to 0. If so, an internal default value (currently 100) will be applied for the limit. * issue #481 * added "attributes.count" to output of `collection.figures()` this also affects the REST API /_api/collection//figures * added IndexedPropertyGetter for ShapedJson objects * added API for user-defined AQL functions * issue #475: A better error message for deleting a non-existent graph * issue #474: Web interface problems with the JS Shell * added missing documentation for AQL UNION function * added transaction support. This provides ACID transactions for ArangoDB. Transactions can be invoked using the `db._executeTransaction()` function, or the `/_api/transaction` REST API. * switched to semantic versioning (at least for alpha & alpha naming) * added saveOrReplace() for server-side JS v1.3.alpha1 (2013-04-05) ------------------------ * cleanup of Module, Package, ArangoApp and modules "internal", "fs", "console" * use Error instead of string in throw to allow stack-trace * issue #454: error while creation of Collection * make `collection.count()` not recalculate the number of documents on the fly, but use some internal document counters. * issue #457: invalid string value in web interface * make datafile id (datafile->_fid) identical to the numeric part of the filename. E.g. the datafile `journal-123456.db` will now have a datafile marker with the same fid (i.e. `123456`) instead of a different value. This change will only affect datafiles that are created with 1.3 and not any older files. The intention behind this change is to make datafile debugging easier. * consistently discard document attributes with reserved names (system attributes) but without any known meaning, for example `_test`, `_foo`, ... Previously, these attributes were saved with the document regularly in some cases, but were discarded in other cases. Now these attributes are discarded consistently. "Real" system attributes such as `_key`, `_from`, `_to` are not affected and will work as before. Additionally, attributes with an empty name (``) are discarded when documents are saved. Though using reserved or empty attribute names in documents was not really and consistently supported in previous versions of ArangoDB, this change might cause an incompatibility for clients that rely on this feature. * added server startup flag `--database.force-sync-properties` to force syncing of collection properties on collection creation, deletion and on property update. The default value is true to mimic the behavior of previous versions of ArangoDB. If set to false, collection properties are written to disk but no call to sync() is made. * added detailed output of server version and components for REST APIs `/_admin/version` and `/_api/version`. To retrieve this extended information, call the REST APIs with URL parameter `details=true`. * issue #443: For git-based builds include commit hash in version * adjust startup log output to be more compact, less verbose * set the required minimum number of file descriptors to 256. On server start, this number is enforced on systems that have rlimit. If the limit cannot be enforced, starting the server will fail. Note: 256 is considered to be the absolute minimum value. Depending on the use case for ArangoDB, a much higher number of file descriptors should be used. To avoid checking & potentially changing the number of maximum open files, use the startup option `--server.descriptors-minimum 0` * fixed shapedjson to json conversion for special numeric values (NaN, +inf, -inf). Before, "NaN", "inf", or "-inf" were written into the JSONified output, but these values are not allowed in JSON. Now, "null" is written to the JSONified output as required. * added AQL functions VARIANCE_POPULATION(), VARIANCE_SAMPLE(), STDDEV_POPULATION(), STDDEV_SAMPLE(), AVERAGE(), MEDIAN() to calculate statistical values for lists * added AQL SQRT() function * added AQL TRIM(), LEFT() and RIGHT() string functions * fixed issue #436: GET /_api/document on edge * make AQL REVERSE() and LENGTH() functions work on strings, too * disabled DOT generation in `make doxygen`. this speeds up docs generation * renamed startup option `--dispatcher.report-intervall` to `--dispatcher.report-interval` * renamed startup option `--scheduler.report-intervall` to `--scheduler.report-interval` * slightly changed output of REST API method /_admin/log. Previously, the log messages returned also contained the date and log level, now they will only contain the log message, and no date and log level information. This information can be re-created by API users from the `timestamp` and `level` attributes of the result. * removed configure option `--enable-zone-debug` memory zone debugging is now automatically turned on when compiling with ArangoDB `--enable-maintainer-mode` * removed configure option `--enable-arangob` arangob is now always included in the build v1.2.3 (XXXX-XX-XX) ------------------- * added optional parameter `edgexamples` for AQL function EDGES() and NEIGHBORS() * added AQL function NEIGHBORS() * added freebsd support * fixed firstExample() query with `_id` and `_key` attributes * issue triAGENS/ArangoDB-PHP#55: AQL optimizer may have mis-optimized duplicate filter statements with limit v1.2.2 (2013-03-26) ------------------- * fixed save of objects with common sub-objects * issue #459: fulltext internal memory allocation didn't scale well This fix improves loading times for collections with fulltext indexes that have lots of equal words indexed. * issue #212: auto-increment support The feature can be used by creating a collection with the extra `keyOptions` attribute as follows: db._create("mycollection", { keyOptions: { type: "autoincrement", offset: 1, increment: 10, allowUserKeys: true } }); The `type` attribute will make sure the keys will be auto-generated if no `_key` attribute is specified for a document. The `allowUserKeys` attribute determines whether users might still supply own `_key` values with documents or if this is considered an error. The `increment` value determines the actual increment value, whereas the `offset` value can be used to seed to value sequence with a specific starting value. This will be useful later in a multi-master setup, when multiple servers can use different auto-increment seed values and thus generate non-conflicting auto-increment values. The default values currently are: - `allowUserKeys`: `true` - `offset`: `0` - `increment`: `1` The only other available key generator type currently is `traditional`. The `traditional` key generator will auto-generate keys in a fashion as ArangoDB always did (some increasing integer value, with a more or less unpredictable increment value). Note that for the `traditional` key generator there is only the option to disallow user-supplied keys and give the server the sole responsibility for key generation. This can be achieved by setting the `allowUserKeys` property to `false`. This change also introduces the following errors that API implementors may want to check the return values for: - 1222: `document key unexpected`: will be raised when a document is created with a `_key` attribute, but the underlying collection was set up with the `keyOptions` attribute `allowUserKeys: false`. - 1225: `out of keys`: will be raised when the auto-increment key generator runs out of keys. This may happen when the next key to be generated is 2^64 or higher. In practice, this will only happen if the values for `increment` or `offset` are not set appropriately, or if users are allowed to supply own keys, those keys are near the 2^64 threshold, and later the auto-increment feature kicks in and generates keys that cross that threshold. In practice it should not occur with proper configuration and proper usage of the collections. This change may also affect the following REST APIs: - POST `/_api/collection`: the server does now accept the optional `keyOptions` attribute in the second parameter - GET `/_api/collection/properties`: will return the `keyOptions` attribute as part of the collection's properties. The previous optional attribute `createOptions` is now gone. * fixed `ArangoStatement.explain()` method with bind variables * fixed misleading "cursor not found" error message in arangosh that occurred when `count()` was called for client-side cursors * fixed handling of empty attribute names, which may have crashed the server under certain circumstances before * fixed usage of invalid pointer in error message output when index description could not be opened v1.2.1 (2013-03-14) ------------------- * issue #444: please darken light color in arangosh * issue #442: pls update post install info on osx * fixed conversion of special double values (NaN, -inf, +inf) when converting from shapedjson to JSON * fixed compaction of markers (location of _key was not updated correctly in memory, leading to _keys pointing to undefined memory after datafile rotation) * fixed edge index key pointers to use document master pointer plus offset instead of direct _key address * fixed case when server could not create any more journal or compactor files. Previously a wrong status code may have been returned, and not being able to create a new compactor file may have led to an infinite loop with error message "could not create compactor". * fixed value truncation for numeric filename parts when renaming datafiles/journals v1.2.0 (2013-03-01) ------------------- * by default statistics are now switch off; in order to enable comment out the "disable-statistics = yes" line in "arangod.conf" * fixed issue #435: csv parser skips data at buffer border * added server startup option `--server.disable-statistics` to turn off statistics gathering without recompilation of ArangoDB. This partly addresses issue #432. * fixed dropping of indexes without collection name, e.g. `db.xxx.dropIndex("123456");` Dropping an index like this failed with an assertion error. * fixed issue #426: arangoimp should be able to import edges into edge collections * fixed issue #425: In case of conflict ArangoDB returns HTTP 400 Bad request (with 1207 Error) instead of HTTP 409 Conflict * fixed too greedy token consumption in AQL for negative values: e.g. in the statement `RETURN { a: 1 -2 }` the minus token was consumed as part of the value `-2`, and not interpreted as the binary arithmetic operator v1.2.beta3 (2013-02-22) ----------------------- * issue #427: ArangoDB Importer Manual has no navigation links (previous|home|next) * issue #319: Documentation missing for Emergency console and incomplete for datafile debugger. * issue #370: add documentation for reloadRouting and flushServerModules * issue #393: added REST API for user management at /_api/user * issue #393, #128: added simple cryptographic functions for user actions in module "crypto": * require("org/arangodb/crypto").md5() * require("org/arangodb/crypto").sha256() * require("org/arangodb/crypto").rand() * added replaceByExample() Javascript and REST API method * added updateByExample() Javascript and REST API method * added optional "limit" parameter for removeByExample() Javascript and REST API method * fixed issue #413 * updated bundled V8 version from 3.9.4 to 3.16.14.1 Note: the Windows version used a more recent version (3.14.0.1) and was not updated. * fixed issue #404: keep original request url in request object v1.2.beta2 (2013-02-15) ----------------------- * fixed issue #405: 1.2 compile warnings * fixed issue #333: [debian] Group "arangodb" is not used when starting vie init.d script * added optional parameter 'excludeSystem' to GET /_api/collection This parameter can be used to disable returning system collections in the list of all collections. * added AQL functions KEEP() and UNSET() * fixed issue #348: "HTTP Interface for Administration and Monitoring" documentation errors. * fix stringification of specific positive int64 values. Stringification of int64 values with the upper 32 bits cleared and the 33rd bit set were broken. * issue #395: Collection properties() function should return 'isSystem' for Javascript and REST API * make server stop after upgrade procedure when invoked with `--upgrade option`. When started with the `--upgrade` option, the server will perfom the upgrade, and then exit with a status code indicating the result of the upgrade (0 = success, 1 = failure). To start the server regularly in either daemon or console mode, the `--upgrade` option must not be specified. This change was introduced to allow init.d scripts check the result of the upgrade procedure, even in case an upgrade was successful. this was introduced as part of issue #391. * added AQL function EDGES() * added more crash-protection when reading corrupted collections at startup * added documentation for AQL function CONTAINS() * added AQL function LIKE() * replaced redundant error return code 1520 (Unable to open collection) with error code 1203 (Collection not found). These error codes have the same meanings, but one of them was returned from AQL queries only, the other got thrown by other parts of ArangoDB. Now, error 1203 (Collection not found) is used in AQL too in case a non-existing collection is used. v1.2.beta1 (2013-02-01) ----------------------- * fixed issue #382: [Documentation error] Maschine... should be Machine... * unified history file locations for arangod, arangosh, and arangoirb. - The readline history for arangod (emergency console) is now stored in file $HOME/.arangod. It was stored in $HOME/.arango before. - The readline history for arangosh is still stored in $HOME/.arangosh. - The readline history for arangoirb is now stored in $HOME/.arangoirb. It was stored in $HOME/.arango-mrb before. * fixed issue #381: _users user should have a unique constraint * allow negative list indexes in AQL to access elements from the end of a list, e.g. ```RETURN values[-1]``` will return the last element of the `values` list. * collection ids, index ids, cursor ids, and document revision ids created and returned by ArangoDB are now returned as strings with numeric content inside. This is done to prevent some value overrun/truncation in any part of the complete client/server workflow. In ArangoDB 1.1 and before, these values were previously returned as (potentially very big) integer values. This may cause problems (clipping, overrun, precision loss) for clients that do not support big integers natively and store such values in IEEE754 doubles internally. This type loses precision after about 52 bits and is thus not safe to hold an id. Javascript and 32 bit-PHP are examples for clients that may cause such problems. Therefore, ids are now returned by ArangoDB as strings, with the string content being the integer value as before. Example for documents ("_rev" attribute): - Document returned by ArangoDB 1.1: { "_rev": 1234, ... } - Document returned by ArangoDB 1.2: { "_rev": "1234", ... } Example for collections ("id" attribute / "_id" property): - Collection returned by ArangoDB 1.1: { "id": 9327643, "name": "test", ... } - Collection returned by ArangoDB 1.2: { "id": "9327643", "name": "test", ... } Example for cursors ("id" attribute): - Collection returned by ArangoDB 1.1: { "id": 11734292, "hasMore": true, ... } - Collection returned by ArangoDB 1.2: { "id": "11734292", "hasMore": true, ... } * global variables are not automatically available anymore when starting the arangod Javascript emergency console (i.e. ```arangod --console```). Especially, the variables `db`, `edges`, and `internal` are not available anymore. `db` and `internal` can be made available in 1.2 by ```var db = require("org/arangodb").db;``` and ```var internal = require("internal");```, respectively. The reason for this change is to get rid of global variables in the server because this will allow more specific inclusion of functionality. For convenience, the global variable `db` is still available by default in arangosh. The global variable `edges`, which since ArangoDB 1.1 was kind of a redundant wrapper of `db`, has been removed in 1.2 completely. Please use `db` instead, and if creating an edge collection, use the explicit ```db._createEdgeCollection()``` command. * issue #374: prevent endless redirects when calling admin interface with unexpected URLs * issue #373: TRAVERSAL() `trackPaths` option does not work. Instead `paths` does work * issue #358: added support for CORS * honor optional waitForSync property for document removal, replace, update, and save operations in arangosh. The waitForSync parameter for these operations was previously honored by the REST API and on the server-side, but not when the waitForSync parameter was specified for a document operation in arangosh. * calls to db.collection.figures() and /_api/collection//figures now additionally return the number of shapes used in the collection in the extra attribute "shapes.count" * added AQL TRAVERSAL_TREE() function to return a hierarchical result from a traversal * added AQL TRAVERSAL() function to return the results from a traversal * added AQL function ATTRIBUTES() to return the attribute names of a document * removed internal server-side AQL functions from global scope. Now the AQL internal functions can only be accessed via the exports of the ahuacatl module, which can be included via ```require("org/arangodb/ahuacatl")```. It shouldn't be necessary for clients to access this module at all, but internal code may use this module. The previously global AQL-related server-side functions were moved to the internal namespace. This produced the following function name changes on the server: old name new name ------------------------------------------------------ AHUACATL_RUN => require("internal").AQL_QUERY AHUACATL_EXPLAIN => require("internal").AQL_EXPLAIN AHUACATL_PARSE => require("internal").AQL_PARSE Again, clients shouldn't have used these functions at all as there is the ArangoStatement object to execute AQL queries. * fixed issue #366: Edges index returns strange description * added AQL function MATCHES() to check a document against a list of examples * added documentation and tests for db.collection.removeByExample * added --progress option for arangoimp. This will show the percentage of the input file that has been processed by arangoimp while the import is still running. It can be used as a rough indicator of progress for the entire import. * make the server log documents that cannot be imported via /_api/import into the logfile using the warning log level. This may help finding illegal documents in big import runs. * check on server startup whether the database directory and all collection directories are writable. if not, the server startup will be aborted. this prevents serious problems with collections being non-writable and this being detected at some pointer after the server has been started * allow the following AQL constructs: FUNC(...)[...], FUNC(...).attribute * fixed issue #361: Bug in Admin Interface. Header disappears when clicking new collection * Added in-memory only collections Added collection creation parameter "isVolatile": if set to true, the collection is created as an in-memory only collection, meaning that all document data of that collection will reside in memory only, and will not be stored permanently to disk. This means that all collection data will be lost when the collection is unloaded or the server is shut down. As this collection type does not have datafile disk overhead for the regular document operations, it may be faster than normal disk-backed collections. The actual performance gains strongly depend on the underlying OS, filesystem, and settings though. This collection type should be used for caches only and not for any sensible data that cannot be re-created otherwise. Some platforms, namely Windows, currently do not support this collection type. When creating an in-memory collection on such platform, an error message will be returned by ArangoDB telling the user the platform does not support it. Note: in-memory collections are an experimental feature. The feature might change drastically or even be removed altogether in a future version of ArangoDB. * fixed issue #353: Please include "pretty print" in Emergency Console * fixed issue #352: "pretty print" console.log This was achieved by adding the dump() function for the "internal" object * reduced insertion time for edges index Inserting into the edges index now avoids costly comparisons in case of a hash collision, reducing the prefilling/loading timer for bigger edge collections * added fulltext queries to AQL via FULLTEXT() function. This allows search fulltext indexes from an AQL query to find matching documents * added fulltext index type. This index type allows indexing words and prefixes of words from a specific document attribute. The index can be queries using a SimpleQueryFull object, the HTTP REST API at /_api/simple/fulltext, or via AQL * added collection.revision() method to determine whether a collection has changed. The revision method returns a revision string that can be used by client programs for equality/inequality comparisons. The value returned by the revision method should be treated by clients as an opaque string and clients should not try to figure out the sense of the revision id. This is still useful enough to check whether data in a collection has changed. * issue #346: adaptively determine NUMBER_HEADERS_PER_BLOCK * issue #338: arangosh cursor positioning problems * issue #326: use limit optimization with filters * issue #325: use index to avoid sorting * issue #324: add limit optimization to AQL * removed arango-password script and added Javascript functionality to add/delete users instead. The functionality is contained in module `users` and can be invoked as follows from arangosh and arangod: * require("users").save("name", "passwd"); * require("users").replace("name", "newPasswd"); * require("users").remove("name"); * require("users").reload(); These functions are intentionally not offered via the web interface. This also addresses issue #313 * changed print output in arangosh and the web interface for JSON objects. Previously, printing a JSON object in arangosh resulted in the attribute values being printed as proper JSON, but attribute names were printed unquoted and unescaped. This was fine for the purpose of arangosh, but lead to invalid JSON being produced. Now, arangosh will produce valid JSON that can be used to send it back to ArangoDB or use it with arangoimp etc. * fixed issue #300: allow importing documents via the REST /_api/import API from a JSON list, too. So far, the API only supported importing from a format that had one JSON object on each line. This is sometimes inconvenient, e.g. when the result of an AQL query or any other list is to be imported. This list is a JSON list and does not necessary have a document per line if pretty-printed. arangoimp now supports the JSON list format, too. However, the format requires arangoimp and the server to read the entire dataset at once. If the dataset is too big (bigger than --max-upload-size) then the import will be rejected. Even if increased, the entire list must fit in memory on both the client and the server, and this may be more resource-intensive than importing individual lines in chunks. * removed unused parameter --reuse-ids for arangoimp. This parameter did not have any effect in 1.2, was never publicly announced and did evil (TM) things. * fixed issue #297 (partly): added whitespace between command line and command result in arangosh, added shell colors for better usability * fixed issue #296: system collections not usable from AQL * fixed issue #295: deadlock on shutdown * fixed issue #293: AQL queries should exploit edges index * fixed issue #292: use index when filtering on _key in AQL * allow user-definable document keys users can now define their own document keys by using the _key attribute when creating new documents or edges. Once specified, the value of _key is immutable. The restrictions for user-defined key values are: * the key must be at most 254 bytes long * it must consist of the letters a-z (lower or upper case), the digits 0-9, the underscore (_) or dash (-) characters only * any other characters, especially multi-byte sequences, whitespace or punctuation characters cannot be used inside key values Specifying a document key is optional when creating new documents. If no document key is specified, ArangoDB will create a document key itself. There are no guarantees about the format and pattern of auto-generated document keys other than the above restrictions. Clients should therefore treat auto-generated document keys as opaque values. Keys can be used to look up and reference documents, e.g.: * saving a document: `db.users.save({ "_key": "fred", ... })` * looking up a document: `db.users.document("fred")` * referencing other documents: `edges.relations.save("users/fred", "users/john", ...)` This change is downwards-compatible to ArangoDB 1.1 because in ArangoDB 1.1 users were not able to define their own keys. If the user does not supply a _key attribute when creating a document, ArangoDB 1.2 will still generate a key of its own as ArangoDB 1.1 did. However, all documents returned by ArangoDB 1.2 will include a _key attribute and clients should be able to handle that (e.g. by ignoring it if not needed). Documents returned will still include the _id attribute as in ArangoDB 1.1. * require collection names everywhere where a collection id was allowed in ArangoDB 1.1 & 1.0 This change requires clients to use a collection name in place of a collection id at all places the client deals with collections. Examples: * creating edges: the _from and _to attributes must now contain collection names instead of collection ids: `edges.relations.save("test/my-key1", "test/my-key2", ...)` * retrieving edges: the returned _from and _to attributes now will contain collection names instead of ids, too: _from: `test/fred` instead of `1234/3455` * looking up documents: db.users.document("fred") or db._document("users/fred") Collection names must be used in REST API calls instead of collection ids, too. This change is thus not completely downwards-compatible to ArangoDB 1.1. ArangoDB 1.1 required users to use collection ids in many places instead of collection names. This was unintuitive and caused overhead in cases when just the collection name was known on client-side but not its id. This overhead can now be avoided so clients can work with the collection names directly. There is no need to work with collection ids on the client side anymore. This change will likely require adjustments to API calls issued by clients, and also requires a change in how clients handle the _id value of returned documents. Previously, the _id value of returned documents contained the collection id, a slash separator and the document number. Since 1.2, _id will contain the collection name, a slash separator and the document key. The same applies to the _from and _to attribute values of edges that are returned by ArangoDB. Also removed (now unnecessary) location header in responses of the collections REST API. The location header was previously returned because it was necessary for clients. When clients created a collection, they specified the collection name. The collection id was generated on the server, but the client needed to use the server-generated collection id for further API calls, e.g. when creating edges etc. Therefore, the full collection URL, also containing the collection id, was returned by the server in responses to the collection API, in the HTTP location header. Returning the location header has become unnecessary in ArangoDB 1.2 because users can access collections by name and do not need to care about collection ids. v1.1.3 (2013-XX-XX) ------------------- * fix case when an error message was looked up for an error code but no error message was found. In this case a NULL ptr was returned and not checked everywhere. The place this error popped up was when inserting into a non-unique hash index failed with a specific, invalid error code. * fixed issue #381: db._collection("_users").getIndexes(); * fixed issue #379: arango-password fatal issue javscript.startup-directory * fixed issue #372: Command-Line Options for the Authentication and Authorization v1.1.2 (2013-01-20) ------------------- * upgraded to mruby 2013-01-20 583983385b81c21f82704b116eab52d606a609f4 * fixed issue #357: Some spelling and grammar errors * fixed issue #355: fix quotes in pdf manual * fixed issue #351: Strange arangosh error message for long running query * fixed randomly hanging connections in arangosh on MacOS * added "any" query method: this returns a random document from a collection. It is also available via REST HTTP at /_api/simple/any. * added deployment tool * added getPeerVertex * small fix for logging of long messages: the last character of log messages longer than 256 bytes was not logged. * fixed truncation of human-readable log messages for web interface: the trailing \0 byte was not appended for messages longer than 256 bytes * fixed issue #341: ArangoDB crashes when stressed with Batch jobs Contrary to the issue title, this did not have anything to do with batch jobs but with too high memory usage. The memory usage of ArangoDB is now reduced for cases when there are lots of small collections with few documents each * started with issue #317: Feature Request (from Google Groups): DATE handling * backported issue #300: Extend arangoImp to Allow importing result set-like (list of documents) formatted files * fixed issue #337: "WaitForSync" on new collection does not work on Win/X64 * fixed issue #336: Collections REST API docs * fixed issue #335: mmap errors due to wrong memory address calculation * fixed issue #332: arangoimp --use-ids parameter seems to have no impact * added option '--server.disable-authentication' for arangosh as well. No more passwd prompts if not needed * fixed issue #330: session logging for arangosh * fixed issue #329: Allow passing script file(s) as parameters for arangosh to run * fixed issue #328: 1.1 compile warnings * fixed issue #327: Javascript parse errors in front end v1.1.1 (2012-12-18) ------------------- * fixed issue #339: DELETE /_api/cursor/cursor-identifier return incollect errorNum The fix for this has led to a signature change of the function actions.resultNotFound(). The meaning of parameter #3 for This function has changed from the error message string to the error code. The error message string is now parameter #4. Any client code that uses this function in custom actions must be adjusted. * fixed issue #321: Problem upgrading arangodb 1.0.4 to 1.1.0 with Homebrew (OSX 10.8.2) * fixed issue #230: add navigation and search for online documentation * fixed issue #315: Strange result in PATH * fixed issue #323: Wrong function returned in error message of AQL CHAR_LENGTH() * fixed some log errors on startup / shutdown due to pid file handling and changing of directories v1.1.0 (2012-12-05) ------------------- * WARNING: arangod now performs a database version check at startup. It will look for a file named "VERSION" in its database directory. If the file is not present, arangod will perform an automatic upgrade of the database directory. This should be the normal case when upgrading from ArangoDB 1.0 to ArangoDB 1.1. If the VERSION file is present but is from an older version of ArangoDB, arangod will refuse to start and ask the user to run a manual upgrade first. A manual upgrade can be performed by starting arangod with the option `--upgrade`. This upgrade procedure shall ensure that users have full control over when they perform any updates/upgrades of their data, and can plan backups accordingly. The procedure also guarantees that the server is not run without any required system collections or with in incompatible data state. * added AQL function DOCUMENT() to retrieve a document by its _id value * fixed issue #311: fixed segfault on unload * fixed issue #309: renamed stub "import" button from web interface * fixed issue #307: added WaitForSync column in collections list in in web interface * fixed issue #306: naming in web interface * fixed issue #304: do not clear AQL query text input when switching tabs in web interface * fixed issue #303: added documentation about usage of var keyword in web interface * fixed issue #301: PATCH does not work in web interface # fixed issue #269: fix make distclean & clean * fixed issue #296: system collections not usable from AQL * fixed issue #295: deadlock on shutdown * added collection type label to web interface * fixed issue #290: the web interface now disallows creating non-edges in edge collections when creating collections via the web interface, the collection type must also be specified (default is document collection) * fixed issue #289: tab-completion does not insert any spaces * fixed issue #282: fix escaping in web interface * made AQL function NOT_NULL take any number of arguments. Will now return its first argument that is not null, or null if all arguments are null. This is downwards compatible. * changed misleading AQL function name NOT_LIST() to FIRST_LIST() and slightly changed the behavior. The function will now return its first argument that is a list, or null if none of the arguments are lists. This is mostly downwards-compatible. The only change to the previous implementation in 1.1-beta will happen if two arguments were passed and the 1st and 2nd arguments were both no lists. In previous 1.1, the 2nd argument was returned as is, but now null will be returned. * add AQL function FIRST_DOCUMENT(), with same behavior as FIRST_LIST(), but working with documents instead of lists. * added UPGRADING help text * fixed issue #284: fixed Javascript errors when adding edges/vertices without own attributes * fixed issue #283: AQL LENGTH() now works on documents, too * fixed issue #281: documentation for skip lists shows wrong example * fixed AQL optimizer bug, related to OR-combined conditions that filtered on the same attribute but with different conditions * fixed issue #277: allow usage of collection names when creating edges the fix of this issue also implies validation of collection names / ids passed to the REST edge create method. edges with invalid collection ids or names in the "from" or "to" values will be rejected and not saved v1.1.beta2 (2012-11-13) ----------------------- * fixed arangoirb compilation * fixed doxygen v1.1.beta1 (2012-10-24) ----------------------- * fixed AQL optimizer bug * WARNING: - the user has changed from "arango" to "arangodb", the start script has changed from "arangod" to "arangodb", the database directory has changed from "/var/arangodb" to "/var/lib/arangodb" to be compliant with various Linux policies - In 1.1, we have introduced types for collections: regular documents go into document collections, and edges go into edge collections. The prefixing (db.xxx vs. edges.xxx) works slightly different in 1.1: edges.xxx can still be used to access collections, however, it will not determine the type of existing collections anymore. To create an edge collection 1.1, you can use db._createEdgeCollection() or edges._create(). And there's of course also db._createDocumentCollection(). db._create() is also still there and will create a document collection by default, whereas edges._create() will create an edge collection. - the admin web interface that was previously available via the simple URL suffix / is now available via a dedicated URL suffix only: /_admin/html The reason for this is that routing and URLs are now subject to changes by the end user, and only URLs parts prefixed with underscores (e.g. /_admin or /_api) are reserved for ArangoDB's internal usage. * the server now handles requests with invalid Content-Length header values as follows: - if Content-Length is negative, the server will respond instantly with HTTP 411 (length required) - if Content-Length is positive but shorter than the supplied body, the server will respond with HTTP 400 (bad request) - if Content-Length is positive but longer than the supplied body, the server will wait for the client to send the missing bytes. The server allows 90 seconds for this and will close the connection if the client does not send the remaining data - if Content-Length is bigger than the maximum allowed size (512 MB), the server will fail with HTTP 413 (request entity too large). - if the length of the HTTP headers is greater than the maximum allowed size (1 MB), the server will fail with HTTP 431 (request header fields too large) * issue #265: allow optional base64 encoding/decoding of action response data * issue #252: create _modules collection using arango-upgrade (note: arango-upgrade was finally replaced by the `--upgrade` option for arangod) * issue #251: allow passing arbitrary options to V8 engine using new command line option: --javascript.v8-options. Using this option, the Harmony features or other settings in v8 can be enabled if the end user requires them * issue #248: allow AQL optimizer to pull out completely uncorrelated subqueries to the top level, resulting in less repeated evaluation of the subquery * upgraded to Doxygen 1.8.0 * issue #247: added AQL function MERGE_RECURSIVE * issue #246: added clear() function in arangosh * issue #245: Documentation: Central place for naming rules/limits inside ArangoDB * reduced size of hash index elements by 50 %, allowing more index elements to fit in memory * issue #235: GUI Shell throws Error:ReferenceError: db is not defined * issue #229: methods marked as "under construction" * issue #228: remove unfinished APIs (/_admin/config/*) * having the OpenSSL library installed is now a prerequisite to compiling ArangoDB Also removed the --enable-ssl configure option because ssl is always required. * added AQL functions TO_LIST, NOT_LIST * issue #224: add optional Content-Id for batch requests * issue #221: more documentation on AQL explain functionality. Also added ArangoStatement.explain() client method * added db._createStatement() method on server as well (was previously available on the client only) * issue #219: continue in case of "document not found" error in PATHS() function * issue #213: make waitForSync overridable on specific actions * changed AQL optimizer to use indexes in more cases. Previously, indexes might not have been used when in a reference expression the inner collection was specified last. Example: FOR u1 IN users FOR u2 IN users FILTER u1._id == u2._id Previously, this only checked whether an index could be used for u2._id (not possible). It was not checked whether an index on u1._id could be used (possible). Now, for expressions that have references/attribute names on both sides of the above as above, indexes are checked for both sides. * issue #204: extend the CSV import by TSV and by user configurable separator character(s) * issue #180: added support for batch operations * added startup option --server.backlog-size this allows setting the value of the backlog for the listen() system call. the default value is 10, the maximum value is platform-dependent * introduced new configure option "--enable-maintainer-mode" for ArangoDB maintainers. this option replaces the previous compile switches --with-boost-test, --enable-bison, --enable-flex and --enable-errors-dependency the individual configure options have been removed. --enable-maintainer-mode turns them all on. * removed potentially unused configure option --enable-memfail * fixed issue #197: HTML web interface calls /_admin/user-manager/session * fixed issue #195: VERSION file in database directory * fixed issue #193: REST API HEAD request returns a message body on 404 * fixed issue #188: intermittent issues with 1.0.0 (server-side cursors not cleaned up in all cases, pthreads deadlock issue) * issue #189: key store should use ISO datetime format bug * issue #187: run arango-upgrade on server start (note: arango-upgrade was finally replaced by the `--upgrade` option for arangod)n * fixed issue #183: strange unittest error * fixed issue #182: manual pages * fixed issue #181: use getaddrinfo * moved default database directory to "/var/lib/arangodb" in accordance with http://www.pathname.com/fhs/pub/fhs-2.3.html * fixed issue #179: strange text in import manual * fixed issue #178: test for aragoimp is missing * fixed issue #177: a misleading error message was returned if unknown variables were used in certain positions in an AQL query. * fixed issue #176: explain how to use AQL from the arangosh * issue #175: re-added hidden (and deprecated) option --server.http-port. This option is only there to be downwards-compatible to Arango 1.0. * fixed issue #174: missing Documentation for `within` * fixed issue #170: add db..all().toArray() to arangosh help screen * fixed issue #169: missing argument in Simple Queries * added program arango-upgrade. This program must be run after installing ArangoDB and after upgrading from a previous version of ArangoDB. The arango-upgrade script will ensure all system collections are created and present in the correct state. It will also perform any necessary data updates. Note: arango-upgrade was finally replaced by the `--upgrade` option for arangod. * issue #153: edge collection should be a flag for a collection collections now have a type so that the distinction between document and edge collections can now be done at runtime using a collection's type value. A collection's type can be queried in Javascript using the .type() method. When new collections are created using db._create(), they will be document collections by default. When edge._create() is called, an edge collection will be created. To explicitly create a collection of a specific/different type, use the methods _createDocumentCollection() or _createEdgeCollection(), which are available for both the db and the edges object. The Javascript objects ArangoEdges and ArangoEdgesCollection have been removed completely. All internal and test code has been adjusted for this, and client code that uses edges.* should also still work because edges is still there and creates edge collections when _create() is called. INCOMPATIBLE CHANGE: Client code might still need to be changed in the following aspect: Previously, collections did not have a type so documents and edges could be inserted in the same collection. This is now disallowed. Edges can only be inserted into edge collections now. As there were no collection types in 1.0, ArangoDB will perform an automatic upgrade when migrating from 1.0 to 1.1. The automatic upgrade will check every collection and determine its type as follows: - if among the first 50 documents in the collection there are documents with attributes "_from" and "_to", the collection is typed as an edge collection - if among the first 50 documents in the collection there are no documents with attributes "_from" and "_to", the collection is made as a document collection * issue #150: call V8 garbage collection on server periodically * issue #110: added support for partial updates The REST API for documents now offers an HTTP PATCH method to partially update documents. Overwriting/replacing documents is still available via the HTTP PUT method as before. The Javascript API in the shell also offers a new update() method in extension to the previously existing replace() method. v1.0.4 (2012-11-12) ------------------- * issue #275: strange error message in arangosh 1.0.3 at startup v1.0.3 (2012-11-08) ------------------- * fixed AQL optimizer bug * issue #273: fixed segfault in arangosh on HTTP 40x * issue #265: allow optional base64 encoding/decoding of action response data * issue #252: _modules collection not created automatically v1.0.2 (2012-10-22) ------------------- * repository CentOS-X.Y moved to CentOS-X, same for Debian * bugfix for rollback from edges * bugfix for hash indexes * bugfix for StringBuffer::erase_front * added autoload for modules * added AQL function TO_LIST v1.0.1 (2012-09-30) ------------------- * draft for issue #165: front-end application howto * updated mruby to cf8fdea4a6598aa470e698e8cbc9b9b492319d * fix for issue #190: install doesn't create log directory * fix for issue #194: potential race condition between creating and dropping collections * fix for issue #193: REST API HEAD request returns a message body on 404 * fix for issue #188: intermittent issues with 1.0.0 * fix for issue #163: server cannot create collection because of abandoned files * fix for issue #150: call V8 garbage collection on server periodically v1.0.0 (2012-08-17) ------------------- * fix for issue #157: check for readline and ncurses headers, not only libraries v1.0.beta4 (2012-08-15) ----------------------- * fix for issue #152: fix memleak for barriers v1.0.beta3 (2012-08-10) ----------------------- * fix for issue #151: Memleak, collection data not removed * fix for issue #149: Inconsistent port for admin interface * fix for issue #163: server cannot create collection because of abandoned files * fix for issue #157: check for readline and ncurses headers, not only libraries * fix for issue #108: db..truncate() inefficient * fix for issue #109: added startup note about cached collection names and how to refresh them * fix for issue #156: fixed memleaks in /_api/import * fix for issue #59: added tests for /_api/import * modified return value for calls to /_api/import: now, the attribute "empty" is returned as well, stating the number of empty lines in the input. Also changed the return value of the error code attribute ("errorNum") from 1100 ("corrupted datafile") to 400 ("bad request") in case invalid/unexpected JSON data was sent to the server. This error code is more appropriate as no datafile is broken but just input data is incorrect. * fix for issue #152: Memleak for barriers * fix for issue #151: Memleak, collection data not removed * value of --database.maximal-journal-size parameter is now validated on startup. If value is smaller than the minimum value (currently 1048576), an error is thrown and the server will not start. Before this change, the global value of maximal journal size was not validated at server start, but only on collection level * increased sleep value in statistics creation loop from 10 to 500 microseconds. This reduces accuracy of statistics values somewhere after the decimal points but saves CPU time. * avoid additional sync() calls when writing partial shape data (attribute name data) to disk. sync() will still be called when the shape marker (will be written after the attributes) is written to disk * issue #147: added flag --database.force-sync-shapes to force synching of shape data to disk. The default value is true so it is the same behavior as in version 1.0. if set to false, shape data is synched to disk if waitForSync for the collection is set to true, otherwise, shape data is not synched. * fix for issue #145: strange issue on Travis: added epsilon for numeric comparison in geo index * fix for issue #136: adjusted message during indexing * issue #131: added timeout for HTTP keep-alive connections. The default value is 300 seconds. There is a startup parameter server.keep-alive-timeout to configure the value. Setting it to 0 will disable keep-alive entirely on the server. * fix for issue #137: AQL optimizer should use indexes for ref accesses with 2 named attributes v1.0.beta2 (2012-08-03) ----------------------- * fix for issue #134: improvements for centos RPM * fixed problem with disable-admin-interface in config file v1.0.beta1 (2012-07-29) ----------------------- * fixed issue #118: We need a collection "debugger" * fixed issue #126: Access-Shaper must be cached * INCOMPATIBLE CHANGE: renamed parameters "connect-timeout" and "request-timeout" for arangosh and arangoimp to "--server.connect-timeout" and "--server.request-timeout" * INCOMPATIBLE CHANGE: authorization is now required on the server side Clients sending requests without HTTP authorization will be rejected with HTTP 401 To allow backwards compatibility, the server can be started with the option "--server.disable-authentication" * added options "--server.username" and "--server.password" for arangosh and arangoimp These parameters must be used to specify the user and password to be used when connecting to the server. If no password is given on the command line, arangosh/ arangoimp will interactively prompt for a password. If no user name is specified on the command line, the default user "root" will be used. * added startup option "--server.ssl-cipher-list" to determine which ciphers to use in SSL context. also added SSL_OP_CIPHER_SERVER_PREFERENCE to SSL default options so ciphers are tried in server and not in client order * changed default SSL protocol to TLSv1 instead of SSLv2 * changed log-level of SSL-related messages * added SSL connections if server is compiled with OpenSSL support. Use --help-ssl * INCOMPATIBLE CHANGE: removed startup option "--server.admin-port". The new endpoints feature (see --server.endpoint) allows opening multiple endpoints anyway, and the distinction between admin and "other" endpoints can be emulated later using privileges. * INCOMPATIBLE CHANGE: removed startup options "--port", "--server.port", and "--server.http-port" for arangod. These options have been replaced by the new "--server.endpoint" parameter * INCOMPATIBLE CHANGE: removed startup option "--server" for arangosh and arangoimp. These options have been replaced by the new "--server.endpoint" parameter * Added "--server.endpoint" option to arangod, arangosh, and arangoimp. For arangod, this option allows specifying the bind endpoints for the server The server can be bound to one or multiple endpoints at once. For arangosh and arangoimp, the option specifies the server endpoint to connect to. The following endpoint syntax is currently supported: - tcp://host:port or http@tcp://host:port (HTTP over IPv4) - tcp://[host]:port or http@tcp://[host]:port (HTTP over IPv6) - ssl://host:port or http@tcp://host:port (HTTP over SSL-encrypted IPv4) - ssl://[host]:port or http@tcp://[host]:port (HTTP over SSL-encrypted IPv6) - unix:///path/to/socket or http@unix:///path/to/socket (HTTP over UNIX socket) If no port is specified, the default port of 8529 will be used. * INCOMPATIBLE CHANGE: removed startup options "--server.require-keep-alive" and "--server.secure-require-keep-alive". The server will now behave as follows which should be more conforming to the HTTP standard: * if a client sends a "Connection: close" header, the server will close the connection * if a client sends a "Connection: keep-alive" header, the server will not close the connection * if a client does not send any "Connection" header, the server will assume "keep-alive" if the request was an HTTP/1.1 request, and "close" if the request was an HTTP/1.0 request * (minimal) internal optimizations for HTTP request parsing and response header handling * fixed Unicode unescaping bugs for \f and surrogate pairs in BasicsC/strings.c * changed implementation of TRI_BlockCrc32 algorithm to use 8 bytes at a time * fixed issue #122: arangod doesn't start if cannot be created * fixed issue #121: wrong collection size reported * fixed issue #98: Unable to change journalSize * fixed issue #88: fds not closed * fixed escaping of document data in HTML admin front end * added HTTP basic authentication, this is always turned on * added server startup option --server.disable-admin-interface to turn off the HTML admin interface * honor server startup option --database.maximal-journal-size when creating new collections without specific journalsize setting. Previously, these collections were always created with journal file sizes of 32 MB and the --database.maximal-journal-size setting was ignored * added server startup option --database.wait-for-sync to control the default behavior * renamed "--unit-tests" to "--javascript.unit-tests" v1.0.alpha3 (2012-06-30) ------------------------ * fixed issue #116: createCollection=create option doesn't work * fixed issue #115: Compilation issue under OSX 10.7 Lion & 10.8 Mountain Lion (homebrew) * fixed issue #114: image not found * fixed issue #111: crash during "make unittests" * fixed issue #104: client.js -> ARANGO_QUIET is not defined v1.0.alpha2 (2012-06-24) ------------------------ * fixed issue #112: do not accept document with duplicate attribute names * fixed issue #103: Should we cleanup the directory structure * fixed issue #100: "count" attribute exists in cursor response with "count: false" * fixed issue #84 explain command * added new MRuby version (2012-06-02) * added --log.filter * cleanup of command line options: ** --startup.directory => --javascript.startup-directory ** --quite => --quiet ** --gc.interval => --javascript.gc-interval ** --startup.modules-path => --javascript.modules-path ** --action.system-directory => --javascript.action-directory ** --javascript.action-threads => removed (is now the same pool as --server.threads) * various bug-fixes * support for import * added option SKIP_RANGES=1 for make unittests * fixed several range-related assertion failures in the AQL query optimizer * fixed AQL query optimizations for some edge cases (e.g. nested subqueries with invalid constant filter expressions) v1.0.alpha1 (2012-05-28) ------------------------ Alpha Release of ArangoDB 1.0